Sorting pubmed articles on the impact factor
Pierre Lindenbaum
Wednesday, 18 June 2008 14:03 UTC
Hi all,
We recently had an interesting discussion on friendfeed about how it would be possible to order the articles in pubmed using the impact factor of their journals as a key of sorting
Lars Juhl Jensen and Deepak Singh suggested me to have a look at http://www.eigenfactor.org where the Eigenfactor is a measure of the journal’s total importance to the scientific community. I then wrote a tool sorting the articles in pubmed using this score.
See here for more information.
Pierre
-
Replies
Jump to resultsResults
-
It looks a really impressive piece of work, Pierre. I am afraid I am not able to work out how to customise it, but that’s my ignorance. Perhaps at the moment it is oriented to people who understand things like code, and not for people like me who aren’t so clever and need instructions!
-
Sorry Maxine, this is again another one of those geeky-command-line-tools loved by the bioinformaticians
:-)
-
Oh wow, I love those top two screenshots! Takes me back to those ms dos days!
-
All very interesting to journal editors I expect, but nothing whatsoever to do with science.
I had a quick look at the ranking of neuroscience journals on eigenfactor.org and only two of the top nine contain any original science at all, The rest are reviews.
From the point of view of the practising scientist, this is just a time-wasting game (but a game with the potential to harm science if anyone were to take it seriously).
-
What about the practicing scientist who is trying to decide where to submit a paper? Or the practicing scientist who has been asked to write a review article and wants to know about the journal, eg its readership?
-
Well, one of the best known things about bibliometrics is that there is no detectable correlation between the number of citations that a paper gets and the impact factor of the journal in which it is published. The rational scientist, therefore, will nor waste time agonising about which journal to choose. That being said, I have to agree that there are plenty of scientists who behave irrationally when it comes to this sort of choice.
The fact that hardly anybody scans paper journals these days makes this more true than at any tine in the past. You do a subject search and see what comes up. Where it is published has never been less important.
-
I think your points are correct, David.
But I do know that practising scientists who are deciding where to submit their research, and whether to write an invited review, want to know the IF of the journal — irrational as that may be. What do to about that dichotomy? -
What this is all moving towards, at least in my mind, is a per-article metric instead of a per-journal metric, which could vastly improve science if everyone were to take a proper implementation seriously and use it.
Everyone knows that journal editors and reviewers are overworked, especially the people at the top tier journals. This overworking is responsible for the, at times, shallow and unhelpful reviews of manuscripts, and it’s also a major reason that doctored or fraudulent data slips past the people who are supposed to catch it.
If we had a metric per-article, the incentive to submit to the high IF journals would decrease, and the burden of reviewing would be spread more evenly across journals. It might even be to your advantage to submit to a smaller circulation journal, if you knew you were more likely to get a higher number of citations or downloads per number of subscribers or dollar spent on subscription fees.
Reviewers could take more time per article, so the reviews would be better and more thoughtful, and that could only help science out overall.
-
Zen and the art of STM publishing (with apologies to Robert Pirsig)
If quality were a measurable parameter, it would not have a normal distribution (in the statistical sense). At best it would be nominal, i.e. categorical. In a recent PLoS ONE article by Liz Allen et al , the authors define a four point scale with the quality of a paper being assessed as “landmark”, “major addition to knowledge”, “useful step forward”, or just “for the record”. But this is, as Pirsig would describe, still a romantic view of reality. Allen et al are after all ranking the publications sourced by their own institute and all estimates of quality are very much in the eye of the beholder.
John Ioaniddis and colleagues have argued, most current published research findings are false. For a research finding to be true, as Liz Allen and her colleagues define it, two things must be true. First, the hypothesis underpinning the study must be correct, and secondly the experimental methodology must be powerful enough to provide a conclusive result. Allen et als paper only reviews cases where they believe both of these test are positive. But there are three other possibilities: that the hypothesis is correct but the design is flawed and yields a negative result (“false negatives”), that the hypothesis is wrong but the statistical result is positive (“false positives”), or that neither is true (“noise”). It is likely that in terms of published articles, the “noise” category is the most numerous, followed by the “false positives”. By contrast, “false negatives” will be quite rare, and articles detailing real progress will be fewer still.
Peter Binfield, Managing Editor at PLos argues that most journals make the peer review process unnecessarily complex and time-consuming by trying to assess whether a paper will have “substantial impact” or “significant advance”, rather than just focussing on methodological rigor., and allowing posterity to be the judge of significance. In other words, the classical STM peer-review process supports the view that quality is a continuous parameter, whereas, in reality, with the benefit of sufficient hindsight, the probabilities of either the design of the hypothesis being correct are either 0 or 1.
So article quality doesn’t end in a decimal point, and if it is deemed to be an important factor, then it should be measured from within the research program that funded the work. After all, as a tax payer, scientific progress means wealth, health and a better world for my grandchildren. Impact Factors, Eigenfactors, and Hirsch Indices aren’t really going to fire me up at the next Election…
-
David, yes the system is ridiculous, but it is far from a waste of time or irrational for practising scientists to pay very careful attention to the IF of journals they are submitting to. Our highly cynical irrationality is driven purely by the need to impress potential employers and funding bodies with the IF of our publications.
Results
-
