Mathematicians report on misuse of citation statistics
Maxine Clarke
Friday, 13 June 2008 14:34 UTC
The International Mathematical Union has released a report on the use of citations in assessing research quality. The report, Citation Statistics, is written from a mathematical perspective and strongly cautions against the over-reliance on citation statistics such as impact factor and h-index. The belief that these parameters are accurate, objective and simple, is unfounded, states the report.
It states that the objectivity of citations is illusory because the meaning of citations is not well-understood. Its meaning can be very far from ‘impact’. While having a single number to judge quality is indeed simple, it can lead to a shallow understanding of something as complicated as research. Numbers are not inherently superior to sound judgments.
The report promotes the sensible use of citation statistics in evaluating research and points out several common misuses. It is written by mathematical scientists about a widespread application of mathematics. While the authors of the report recognise that assessment must be practical and that easily-derived citation statistics will be part of the process, they caution that citations provide only a limited and incomplete view of research quality. Research is too important, they say, for its value to be measured with only a single coarse tool.
(This is a precis of the press release accompanying publication of the report, see links above.)
Updated 13 June 2008 14:35 UTC
-
Replies
-
I read the report and I agree with it basically. The whole “impact factor” notion is really not good. It turns science over to the hands of the publishers, who for the most part are not doing the science.
The question really is, can a small journal, say “The Journal of Southwestern Dirt” (I made that up, obviously) ever catch up to a Nature or a Science based on the damage impact factor has already done to the economics of publishing? So, in other words-even if our metrics get better, is there not a subconscious gravitation to these journals because of the historical (and economic) resources they have amassed? Even if studies in The Journal of Southwestern Dirt are scientifically more sound then those in Science or Nature?
-
These are indeed issues, as you rightly say, Michael. We also know that in all journals, from Nature or Science to the Journal of Southwestern Dirt, contain papers that are not cited at all. Only a relatively small proportion of papers publised are highly cited, even in the “high impact” journals.
And of course, one should not forget that in a journal like Nature or Science that is interdisciplinary, there are some disciplines where citations are not high, and others where it is. Cell biology, cancer, immunology and so on are disciplines with a highly citing, high-frequency publication culture, whereas paleontology, Earth Sciences and so on are disciplines where publication (and citing) is not so frequent. -
To visualize what Maxine is correctly stating with regards to impact factor being heavily-influenced by a small fraction of highly-cited papers, please see this old Nature Neuroscience editorial.
-
