Are downloads an indicator of influence?
Maxine Clarke
Thursday, 29 May 2008 16:25 UTC
The June editorial in Nature Neuroscience discusses the relationship between web traffic and citations. The journal’s preliminary analysis indicates that the number of downloads a paper receives immediately following its appearance online correlates very well with its citation frequency years after publication. Noah Gray, one of the Nature Neuroscience editors, has written a post at Action Potential, the journal’s blog, to provide more of the details behind the data and analysis, and to initiate discussion. He writes (edited for length):
Everyone has their own pet problem with impact factors, but despite these concerns, these numbers are typically used to rate the importance or prominence of a particular journal, and thus by proxy, the importance of the individual papers published within. This is a seriously flawed use of association, leading scientists to often equate the total number of citations with scientific impact, which can be fraught with problems. Searching for an alternative measure of impact that is perhaps free of the “bias of authority” (citing a paper because it is from a famous lab) or the “lemming bias” (citing a paper just because everyone else seems to do so whenever broaching a particular subject) led us to explore readership….
The “number of downloads” measure potentially provides a piece of an alternative solution for deciphering the impact of an individual paper. In this current scientific climate where tenure and grant funding decisions are influenced by flawed metrics like impact factor, it is important to make good use of all available technology in an attempt to realize a better system of measuring the scientific impact of any particular paper. This analysis is obviously preliminary and flawed in its own ways, but perhaps metrics such as paper downloads can find a place in a compilation of aggregated stats, painting a more accurate and informative picture of manuscript influence.
The Nature Neuroscience editorial.
The Action Potential post.
Updated 29 May 2008 16:27 UTC
-
Replies
Jump to resultsResults
-
Noah, thanks for your response.
I realize that you only analyzed PDF downloads, but the notion that articles could be evaluated based on download counts implies that we would want to compare articles published in different journals often published by different publishers (remember that not everything is published by Macmillan ;-)
Citation counts (unlike download counts) are not sensitive in the same way to interface effects as article downloads. Individuals will (and often do) download the same article multiple times. If you cite someone’s paper in your article, it doesn’t matter how many times you’ve downloaded it — the citation count remains one.
-
I still don’t see why we should not have several metrics if they have even only a small amount of information regarding the use and impact of the published work. Unfortunately it is impossible to even look at different sorts of metrics because they are either locked way in private databases (citations) or not made available at all (page hits and number of downloads). The most easily accessible metrics (number of times stored in online reference managers and citations in blog posts) are still not mature enough to analyze on a large scale.
-
This is rather late in the day but one thing that strikes me about this whole debate is at the end of the day, does it matter very much? Promotion and hiring decisions are made by small groups of people based on highly subjective criteria generally by people who aren’t expert (or in many cases even competent – and I don’t mean that in pejorative way – they will be competent at other things) in the area of the candidates’ science. Giving the panel a slightly improved, but more difficult to calculate, parameter as an input to what is actually a rather small part of the selection process seems like a lot of hard work for very little benefit. Although its fun to argue about it :)
Or to put it another way – why are we so obsessed with measuring and quantifying things that we know are difficult to value. If we are in this just to play the game of ‘my paper is bigger than your paper’ then people will always choose a metric that suits them. As the saying goes ‘the wonderful thing about standards….’
In some other place I think we ended up with thinking that at the end of the day, the most effective way of judging someone’s contribution is by asking a group of their peers to assess it :)
-
In one sense you are right, but talking to many people who sit on on these kinds of panels over the years, they are all so busy, and being on search committees, awarding grants, promotions, hiring, etc is a huge time-sink, most people want an “at a glance” objective-ish scale for looking at a mass of applications (of whatever kind). Most people seem to say that if they had time, they’d do a lot of detailed work and reading, but they don’t have the time. And these “administration” duties of various kinds increase all the time for people, increasing their burden further and detracting from the science that they want to do (and are qualified to do!).
I don’t think I or anyone thinks that there “is” a quick fix, but I certainly hear that people would love one, because what they are forced to do now is to not be thorough enough, or they find that the available metrics aren’t appropriate for a particular circumstance, etc.
-
We seem to be heading towards the general topic of ‘how to judge research impact/quality’ – I assume most researchers would agree that it is as easy to delineate as ‘how long is a piece of string?’.
Given all that, my additional comment to Maxine’s comes from several years of running peer-review panels and boards for assessment of biomedical research grants. The bottom line when you have about 7 hours in a one-day meeting to make a funding decision on 50+ grants when you are often not familiar with the track record of the research team (granted that there will be some referees’ comments to help), you will look to any ‘credible’ quantifiable measures of past research performance. That’s the way of the world – at present. -
I think that adopting a variety of alternative measures for the community to access is important for exactly the reasons that Allan brought up. Again, most of the time, more senior investigators are somewhat insulated from these forces; they have the luxury of being “known”. However, it is critical for the young investigators to have more means to demonstrate past success and future potential.
-
It is fine to investigate a wide range of measures as long as you accept the possibility (inevitability would be my guess) that none of them will turn out to be at all satisfactory.
As Maxins says, there are people on promotion and grant committees who look for short cuts. There are also a lot of people who don’t. The answer to that is to have on such committees only people who are willing to do the necessary work. More particularly, people who don’t know the area should keep quiet rather than giving opinions based on unsatisfactory metrics. They should rely solely on the opinions of expert referees. Much depends, of course, on the choice of referees. My impression (no more than that) is that the MRC used to be very good at choosing appropriate referees, but with the fast rate of staff turnover they have become substantially worse at doing that critical job.
The applicant’s own choice of referees is part of the answer, but only part (after all, that is how people get away with degrees in homeopathy -they are accredited by other believers in magic medicine).
The refereeing of papers is certainly a problem. but in large part it is a problem inflicted on us by bean-counters (aided and abetted by bibliometricians). They encourage the publication of vast numbers of papers, often trivial and overlapping, and all of them have to be refereed by someone. If people wrote papers only when they had something valuable and complete to say, the refereeing problem would be greatly reduced. Bad measures of productivity are actively harmful to science, and that is something that their advocates should bear in mind. They are encouraging dishonesty.
Once again I urge that anyone who advocates any any sort of automated assessment of individuals should try out their measures on the early careers of people who subsequently turned out to be universally respected. Just see how many of them would have been fired before they got their honours.
Perhaps for each field, a short list of universally accepted top people could be assembled (though reaching agreement even on that could be hard), then no paper on bibliometrics could be accepted without an analysis of how those people would have fared between the ages of, say, 25 to 40.
Results
-
