Better metrics for an individual's "value"
Maxine Clarke
Wednesday, 11 June 2008 11:53 UTC
The misuse of Impact Factors is mainly in two areas. First, different disciplines have different publication rates and citation behaviour, leading to a host of errors and misunderstandings; and second, a journal’s Impact Factor is sometimes (in some countries) applied to individuals to assign a quantitative score for purposes such as awarding positions, tenure or grants, which is incorrect.
In addition, while seemingly simple in its definition, the exact parameters used to calculate the Impact Factor are opaque.
Scientists are only interested in the Impact Factor because it is used as a surrogate measure for their own work. A set of agreed metrics that assesses the value of one’s contribution to the field would result in much less emphasis and attention on the Impact Factor.
(1) What factors should be included in “value to the field” in addition to a person’s publications (papers not journals)? Conference talks, posters, committees, peer-review activities, awards, mentoring, grants, fellowships…. (2) What metrics can be used or developed to measure these factors?
Updated 11 June 2008 11:55 UTC
-
Replies
-
Dear Maxine,
I totally agree with your analysis of the problem, which is particularly valid for the field I currently work in. As a Physicist working in a Medicine environment I was astonished by the discovery that the CV’s of my colleagues are evaluated by using the Impact Factor which is often a misleading indicator of the quality of a Journal. For sure it cannot be considered as a less misleading indicator of the quality of someone’s research work.
Even if one focuses only on the publications it would be desirable to find better descriptors for the productivity of a scientist. In Physics the use of other indicators is becoming a standard such as the h-index which is based on an “integral” measure of the citations received by the papers of the scientist under evaluation. Of course as any other indicator even the h-index has limitations. For example the h-index is virtually nonsense for a young scientist because it does not account for the increasing probability of getting cited as time passes and his h-index will be unavoidably lower than the one of a middle-aged colleague.
In my opinion the assessment of the productivity of scientists should mandatory pass through a better evaluation of the impact of THEIR papers on the community. All the other activities that you mention in your post are of course important but in my opinion they are somehow secondary. Nevertheless they must be accounted for when evaluating somebody’s CV and this is what regularly happens. When you browse a CV you do pay attention to invited talks, awards, mentoring, etc. etc. but as you point out no shared metrics are used for their quantitative evaluation.
One interesting factor for an experiment in this sense is the peer reviewing activity. Some Journals already publish at the end of the year the list of names of Reviewers. Perhaps they could add some quantitative factor such as the number of papers reviewed or an evaluation index of the Reviewer’s work? This would be a small but useful step to assess the contributions of individuals to their fields of activity.
-
One telling example of the limitations of the h-index is described by Bob O’Hara at his Nature Network blog in a post ‘Outdone by mis-prints’. To quote:
….the result of all this endeavour was to discover that the mis-citations of the Cox PH paper on their own (i.e. ignoring the correct citation) had gained themselves an h of 12: a level that Hirsch had concluded ”…might be a typical value for advancement to tenure…”.
My own h index at the moment is a meagre 9. Which means my contribution to science is being exceeded by a set of misprints. -
“Perhaps they could add some quantitative factor such as the number of papers reviewed or an evaluation index of the Reviewer’s work?”
I am not sure this is a good idea. Researchers may tend to accept as much review requests as possible to increase their r-factor and no longer only review those manuscripts that are really in their field of expertise. For instance, I could start accepting all review requests on nutrient cycling that are sent to me by accident.
-
You are both right. It is not easy at all. I think that maybe the use of many of these indicators can attenuate the problems you are pointing out. If the h-index is coupled to the impact factor and to a multitude of other indicators, appropriately weighted this might attenuate the effect of a “fake” indicator on the evaluation of someone’s activity.
Quoting Raf: “Researchers may tend to accept as much review requests as possible to increase their r-factor and no longer only review those manuscripts that are really in their field of expertise”
In principle you are right. In practice I’m sure that if you review a paper for which you do not have adequate competences this would be soon very evident!!! I believe that editors are very good in evaluating not only the papers but also their reviewers.
-
I think you are both correct. Journals do value their peer-reviewers highly. Journals do provide some benefits for reviewers, including the annual lists provided by some journals, but they could do other things too, I think.
However, it is also correct, as Raf says, that an “r” factor would be quite hard to devise! For example a fast reviewer is not necessarily a good one; what is the relative value of a 5-page detailed report vs a couple of paragraphs – both may result in the same decision by the journal, but which is most “valuable”?, etc.
-
Well, editors usually evaluate the quality of reviewers (fast/slow, detailed/superficial, critical/lenient, etc…) and tend to avoid asking poor referees to review again. There is some ‘natural selection’ in the process. A simple ranking of referees could be based on combining frequency of review with speed: referees who review *f*requently and *f*ast (let’s call this the FF-metric…) for a given journal are likely to be good referees, at least within the field covered by the journal. Not perfect, but would be a start…
-
_In addition, while seemingly simple in its definition, the exact parameters used to calculate the Impact Factor are opaque. _
It is true that it is difficult to reproduce the absolute numbers of Thomson (see debate between Rossner et al 2007 and reply by Thomson). I tried to use the citation information accessible from the “Web of Science” (using the “Cited Reference Search” function) to predict the “official” Impact Factor (“Journal Citation Reports”, JCR). Here is an estimation for 20 different journals:
The estimation is systematically underestimating the IF by a factor of 1.3 suggesting that the two Thomson products, “Web of Science” and “Journal of Citation Report” may rely on somewhat different datasets. Or perhaps my procedure was too quick and dirty? Not sure…
But the good news is that the difference between the two appears to be a systematical one, allowing reasonable empirical estimations.
Did anyone achieve perfect reproduction of the published IF for a broad range of journals? How did you do?
-
I just came accross two posts on Reputation Systems which are exceedingly pertinent to this discussion (as much as Thomson’s IF is indeed totally irrelevant as Maxine has pointed out so eloquently).
IMHO, this is the kind of system that needs to be developed and as many metrics as possible should be included. Why as many as possible? For one, it makes gaming the system more difficult. Moreover, “reputation” is not one-dimensional and a multifactorial assessment will show the strengths and weaknesses of the individual scientist. Last but not least, different people looking at a scientist’s reputation are probably looking for different things (e.g., media, colleagues, administrators).
However, it should be always kept in mind that these measures, no matter how sophisticated, can never substitute the good old reading of the actual contributions themselves! -
A commentary has been published on topic Is citation a good Crietria
-
