ESEP forum on use and misuse of bibliometric indicators
Maxine Clarke
Monday, 02 June 2008 09:22 UTC
A journal called Ethics in Science and Environmental Politics "is currently running a set of perspective articles ":http://www.int-res.com/abstracts/esep/v8/n1 on ‘The use and misuse of bibliometric indices in evaluating scholarly performance’. The articles are open access. From the summary:
Quantifying the relative performance of individual scholars, groups of scholars, departments, institutions, provinces/states/regions and countries has become an integral part of decision-making over research policy, funding allocations, awarding of grants, faculty hirings, and claims for promotion and tenure. Bibliometric indices (based mainly upon citation counts), such as the h-index and the journal impact factor, are heavily relied upon in such assessments. There is a growing consensus, and a deep concern, that these indices — more-and-more often used as a replacement for the informed judgement of peers — are misunderstood and are, therefore, often misinterpreted and misused. The articles in this ESEP Theme Section present a range of perspectives on these issues. Alternative approaches, tools and metrics that will hopefully lead to a more balanced role for these instruments are presented.
One of the contributors is Philip Campbell, Editor of Nature and Editor in Chief of Nature publications. This is the abstract of his contribution, ‘Escape from the Impact Factor’:
As Editor-in-Chief of the journal Nature, I am concerned by the tendency within academic administrations to focus on a journal’s impact factor when judging the worth of scientific contributions by researchers, affecting promotions, recruitment and, in some countries, financial bonuses for each paper. Our own internal research demonstrates how a high journal impact factor can be the skewed result of many citations of a few papers rather than the average level of the majority, reducing its value as an objective measure of an individual paper. Proposed alternative indices have their own drawbacks. Many researchers say that their important work has been published in low-impact journals. Focusing on the citations of individual papers is a more reliable indicator of an individual’s impact. A positive development is the increasing ability to track the contributions of individuals by means of author-contribution statements and perhaps, in the future, citability of components of papers rather than the whole. There are attempts to escape the hierarchy of high-impact-factor journals by means of undifferentiated databases of peer-reviewed papers such as PLoS One. It remains to be seen whether that model will help outstanding work to rise to due recognition regardless of editorial selectivity. Although the current system may be effective at measuring merit on national and institutional scales, the most effective and fair analysis of a person’s contribution derives from a direct assessment of individual papers, regardless of where they were published.
Please do read the full article (available in PDF) and the other contributions; table of contents here. They are highly relevant to this Nature Network forum as they range from as ‘How measurement harms science’ [by Peter Lawrence, not David Colquhoun ;-) ], to codes of practice for bibliometric indicators, to ‘Validation of research performance metrics against peer science’ to an evaluation of Google Scholar, among others.
Updated 09 June 2008 15:59 UTC
-
Replies
-
This should be mandatory reading material for [Ph.D./post-doc] project proposal referees.
-
Yes, this is definitely required reading (and it’s all open access). I’ve collected some of what I thought were the most interesting talking points.
-
Your link appears not to work, Bjorn.
-
Maxine, these are indeed all very relevant papers. Many of the arguments against the uncritical use of bibliometric indicators have already been discussed in this forum and elsewhere on Nature Network.
I would like to emphasize one important argument by Peter Lawrence (and also used by David Colquhoun): The use of numerical measures rewards aggressive, acquisitive and exploitative behaviour, their use will select for scientists with these characteristics. More reflective modest scientists are at a disadvantage in this system, but could still produce great scientific work (exemplified by Nobel Prize winner Ed Lewis).
-
Strange, the link works for me…
-
My big issue with a bibliometrics approach is that the system of valuation and assessment inevitably has an impact on the science that is carried out. To give an illustration; as part of my PhD I conducted a field experiment. Over 2 years I took thousands of samples from the field (ah, fond memories of being blown in to hedges in lashing rain), took them into the lab and conducted several analyses on each sample. This amounted to one paper. My point is that if one is judged on output of papers, this will invariably affect the science that one is willing to conduct. If a couple of weeks in the lab will yield sufficient results for a paper, versus a study of a couple of years, then, for a rational person who is aware of the system, it’s a no brainer. Does that mean these are the experiments that should be done? Surely this choice should be made on a scientific basis, rather than based on the assessment procedures.
We are swamped by the volume of publications as we need to publish every small piece of work; gone are the days when scientists published when they really thought they were ready with something worth communicating. At the recent Talk Science event, I think it was Ian who proposed that a scientist should have an annual publication page limit. Although obviously unrealistic, it would mean that papers were well considered, had substance and were really worth reading.
-
