Furore over journal impact factors
William Burns
Tuesday, 03 February 2009 02:16 UTC
For any of you who don’t live in the small world of history and philosophy of science, you might have missed this furore about impact factors. I think it raises a number of questions about our own reading habits in science. And because reading is the yang to writing’s yin, it raises questions about our writing habits too.
The background to this story is that the European Science Foundation want to develop impact factor measurements for the quality of various history and philosophy of science journals.
This has not gone down well. In fact, the editors of all the major journals in the field have said a resounding ‘no’ to the whole idea.
One argument against the ESF’s impact factor scheme is that such calculations don’t take account of excellent work in minor publications – or mediocre work that gets into the more widely read journals. Another argument being made is that the calculations are biased towards English-language journals (which get more widely cited not neccesarily because their content is the best but because more people have the language skills to read them).
In essence, the argument is that impact factors encourage us to assess arguments not by their quality but by what journal they’re in. We stop reading articles and instead put our blind faith in the metrics.
Well, we seem to have gone a long way along this road in science. How many fellowship/grant/recruitment committees read all the publications listed on CVs – or at least the most recent ones? Even if they read them, do ‘the powers that be’ really engage with the arguments and data in these papers, and judge candidates based on them? I doubt it, although you may tell me otherwise. We simply look at the impact factor and make a judgement based on that.
Isn’t this a bad thing? Have we got to a stage where the writing is important only in as much as it gets the paper past the foibles of a few referees (and therefore into a top journal)? Does it matter after that if no one reads it any more? Does it matter if the paper’s central argument is even true?
-
Replies
This topic has been locked by the forum moderators.
Jump to resultsResults
-
William,
thanks for your detailed post. We have had and will continue to have many discussions about impact factors here on Nature Network, e.g. in the Citation in Science forum.
For the discussion here in the Good Paper Journal Club I would like to ask a related question. Does the (over)use of impact factors change the readability of papers? Do authors cut a research project into smaller pieces that may increase your science metrics but make the project more difficult to understand? In what other ways could the reliance on impact factors change the readability for the worse (or the better)? My first thought is that (over)using science metrics has many undesired effects (you mentioned some of them), but that it shouldn’t make papers more difficult to read.
-
Thanks for your reply Martin. Yes, I guess these issues are pretty familiar, and I hope I’m not repeating some discussion that appeared somewhere else in the depths of Nature Network, but in answer to the question you pose: I was wondering if the metric approach – and the way we read that comes from it – was influencing our writing. I think its rather a hard idea to test, because if you go back before impact factors – well, sure, scientific writing might look slightly different. But it would be pretty hard to pin that down to metrics alone I think.
Anecdotaly, however, I feel metrics must have changed the way we write. Writers, surely by definition, always try to respond to their audience, or percieved audience? For instance, do we work as hard on a paper for, say, the Hawaiian Journal of Surfing Studies as we would for Nature? I guess the answer is no. The reasoning is that the career reward from publication in the Hawaiian Journal is much less than the same publication in Nature. Not only career reward, but also the human desire to have our writing read. We believe that no one will read our efforts, no matter how good, if we publish in the lower IF levels. Perhaps it’s also easier to pass off silly or poorly argued conclusions further down the chain.
I suspect there is a lot of awful, unreadable, stuff circulating in the depths which was just published to bulk up CVs. So could it be that the quality of writing/exposition has dropped in the lesser (lower impact factor) journals, given that the writers and editors think no one is watching them? And if so, could we relate that to the rise of metrics in general?
-
William, as you say it is difficult to test whether the use of science metrics such as impact factors has an influence on the readability of a paper. It could for example also be possible that we work harder on the quality of the writing because acceptance of that paper is so critical for our CV.
And I don’t know if there is a correlation between the impact factor of a journal and the readability of the papers in that journal.
-
Bob O’Hara seems to have posted about this same proposal “over in Citation in Science”: http://network.nature.com/groups/citation-science/forum/topics/3103 a while ago. I’ve crossreferenced this discussion over there, also.
-
Sorry, forgot to preview. Here is the link, in active form.
-
I thought the idea of using impact factors for journals to rate the qualiry of papers was dead and buried years ago. But bone-headed bean-counters with nothing better to do seem to keep the discussion going. It’s a bit like homeopathy. Demonstrable rubbish but it won’t go away.
-
David, welcome to the Good Paper Journal Club. We have mentioned one of your papers here before (On the nature of partial agonism in the nicotinic receptor superfamily), and I especially liked the introduction.
I’m hoping that this Journal Club stays focussed on good science writing and leaves the Impact Factor discussion to other people. But I guess you can’t avoid to talk about it.
-
I fear that David Colquhoun’s opinion about impact factors is not quite neutral. If it is no matter where to publish good paper, why did his scientific team publish their paper in “Nature” (impact factor is more than 28.7)? They could send it to Neuroscience or J. Neurophysiol. (impact factors are about 3.3 and 3.5), for example, or to Biochemistry (Moscow) (impact factor is about 1.4 even in best times).
Surely, I understand that members of his lab will not do it. It is clear, that they don’t share his opinion about impact factor. But what about himself? Could he publish his best paper in low-impact-factored journal? -
Svetlana – I think there’s a difference between the quality of a journal and its impact factor. DC rails against the impact factor because it’s not necessarily a good measure the quality of a paper or journal, and hence the bean counters do a poor job of judging quality. But his is not the same thing as acknowledging that there differences in the qualities of journals.
-
Bob,
Then – at least several challenges arise.
1) Is it possible to create good method of quality estimation of journals?2) Main thing in Science is quality of scientist himself/herself, but not quality of journals, where he/she publishes the papers. Some researchers have a lot of papers in high-impact-factored journals (or better – in high-quality journals, using your terminology ;) ). But they with their papers are known only for narrow circle of specialists. Other scientists almost have not papers in world scientific journals with high impact factor. Nevertheless they are great scientists and all world knows them. For example, I have found none of paper by Russian physicist Z.I. Alferov in “Nature”. However, he is Nobel Laureate and all world uses his discovery. It is possible to create an adequate estimation method of quality of scientist?
3) Another main thing is a quality (and significance for mankind) of research itself. Even “Nature” and “Cell” publish a lot of works, which will be forgotten tomorrow. But there is really great discoveries, which were never published in high-quality journals. How to estimate real significance of scientific work?
4) At last another important problem concerns a discrepancy between words and real actions. Well. We know that such discrepancy is main trait of bureaucrats. Unfortunately now we see that this unworthy process starts to capture scientists themselves. How to fight against it?
David,
Sapienti sat.
Results
-
