Impact factors again, folks. Plenty of discussion about those and other indices of our personal scientific worth, just search for the impact factor tag on NN for e-sheaths and screeds of disgruntlement, or see here for a more gladiatorial point of view.
I shall merely take the opportunity today to present some simple stats about those simple stats.
I took the top 60 journals in Ecology and Evolutionary Biology (ranked by 2-year Impact Factor) from the ISI Journal Citation Reports for 2008. I looked at 6 of the statistics published by ISI based on different :
- Impact Factor (2-years)
- 5-Year Impact Factor
- Immediacy Index
- Cited Half-Life (adjusted so anything ≥ 10 was set to 10)
- EigenfactorTM Score
- Article InfluenceTM Score
I calculated Pearson’s linear correlation coefficient for those scores across 57 of the top 60 journals (3 were excluded, lacking 5-year IF and Article Influence scores0).
And here are the results:
Table 1. Correlation between different indices on ISI Journal Citation Reports_1
!http://farm4.static.flickr.com/3403/3671403588470c0ed892.jpg?v=0!
So, what can we take from this? Well, I was interested in how similar these metrics were, in terms of describing, urrrr, anything about the journals and/or their content. Basically, there are a few different options to choose from; do they actually tell us different things, or do they all say pretty much the same?
Well, it seems that 2- and 5-year Impact Factor tell us a pretty similar story (r0 = 0.712). A journal with a high (or low) 2-year IF will generally also have a high (low) 5-year IF. This is probably to be expected as the same information is used for both metrics (2 years is a subset of 5 years).
EigenfactorTM seems to give us slightly different information2, and Half-life is generally unrelated to any other index across these journals. These last two indices hopefully give us novel information about journals and their content.
One result sprang out for me. The 5-year IF and Article InfluenceTM appear to tell us almost exactly the same thing (r0 = 0.983!).
Figure ∆: Spot the difference.
So, do we really need all these different metrics? The above analysis is quick and dirty, to say the least. I haven’t even gone into what the different metrics actually mean here. But with concerns about how IF is used to judge individual scientists, and new metrics springing up reasonably often, it’s worth asking how useful the new metrics really are. How much extra information can be gleaned while having to differentiate between what the new indices really mean and their method of calculation?
Article influence can potentially be dumped at the journal level – it just duplicates 5-year IF information. But if it tells us something about individual papers, then it retains utility.
In fact, Article Influence is a measure of the per-article influence of a given journal, which is the Eigenfactor scaled by something called the “normalised article vector”. As far as I can tell, this vector relates to the number of articles a journal publishes, so AI is the Eigenvalue scaled to be comparable to Impact Factor.
This tells us that the normalised article vector does something important to the Eigenfactor, as EF and IF (2- or 5) don’t correlate too strongly. So, Article Influence really doesn’t add anything new to the story, and Eigenfactor is just a longer term impact factor, lacking information about the number of articles a journal actually publishes. There’s been a bit of hope about what these new measures can offer, but I’m not yet convinced these two actually add anything new or more intuitive to the story.
Now I just need to think of a snappy way to end this post. I have the same problem finishing off my Discussions in manuscripts.
Snap.
0 These were ranks 35 (Biogeosciences), 47 (Biol. Invas.) and 55 (J. Syst. Paleontol.)
1 How on earth do you make tables on NN? This ‘table’ is a screen shot from a word processing document.
2 Duncan, if you’re reading this, my analysis of Eigenfactor sort of goes against the view in your blog post. Different set of journals? Any comment?
If only you have rotated that matrix, you could have got it into PLoS One.
Good grief – and I was worried about making this post easy for people to understand.
Thank goodness there’s a measure called “Betweenness”.
Interestingly, the comment accompanying the PLoS One article Bob cites raises some points I tried to avoid in my analysis.
I deliberately tried not to give any value judgements about what the different measures say/mean. Ramy over at PLoS highlights that "Bollen et al ":http://dx.doi.org/10.1371/journal.pone.0006022 (2009) seem to want to criticize IF (Journal Impact Factor, JIF), by stating that it’s often used in the wrong way.
I would add that the act of using a statistic in the wrong way is not the fault of the statistic, rather the people misusing it. It all seems like a bit of a straw man.