Are downloads an indicator of influence?
Maxine Clarke
Thursday, 29 May 2008 16:25 UTC
The June editorial in Nature Neuroscience discusses the relationship between web traffic and citations. The journal’s preliminary analysis indicates that the number of downloads a paper receives immediately following its appearance online correlates very well with its citation frequency years after publication. Noah Gray, one of the Nature Neuroscience editors, has written a post at Action Potential, the journal’s blog, to provide more of the details behind the data and analysis, and to initiate discussion. He writes (edited for length):
Everyone has their own pet problem with impact factors, but despite these concerns, these numbers are typically used to rate the importance or prominence of a particular journal, and thus by proxy, the importance of the individual papers published within. This is a seriously flawed use of association, leading scientists to often equate the total number of citations with scientific impact, which can be fraught with problems. Searching for an alternative measure of impact that is perhaps free of the “bias of authority” (citing a paper because it is from a famous lab) or the “lemming bias” (citing a paper just because everyone else seems to do so whenever broaching a particular subject) led us to explore readership….
The “number of downloads” measure potentially provides a piece of an alternative solution for deciphering the impact of an individual paper. In this current scientific climate where tenure and grant funding decisions are influenced by flawed metrics like impact factor, it is important to make good use of all available technology in an attempt to realize a better system of measuring the scientific impact of any particular paper. This analysis is obviously preliminary and flawed in its own ways, but perhaps metrics such as paper downloads can find a place in a compilation of aggregated stats, painting a more accurate and informative picture of manuscript influence.
The Nature Neuroscience editorial.
The Action Potential post.
Updated 29 May 2008 16:27 UTC
-
Replies
Jump to resultsResults
-
All metrics are subject to manipulation and abuse. If this one were to be taken seriously it could very easily be manipulated through a multiple download call. Now I know the article is using it as a cross-reference but I would not like to use this one.
Publishers take the imperfect impact factor seriously. I am at the World Biomaterials Congress and the publishers are hawking their wares with posters of imapct factors. I have sat on editorial committes where people have in all seriousness talked about how to weed out papers that will receive no citations at the referee/editorial stage (of course there is no way to do this). They will talk about commissioning reviews from well known names to boost citations and add positive bias to the journal.
Yes we all prostitute ourselves.
-
The impact factor of the journal is such a poor metric for evaluation of individual papers that it is worth looking for new metrics. The number of downloads is easier to game than the impact factor of the journal but it can be made more robust. Analyzing the IP addresses and timing of requests for the PDFs for example could make it much harder to inflate the values. In any case having additional metrics is positive even if they are not perfect (none are).
-
Pedro, I’d like to ask you (or other reader) specifically about your point:
Analyzing the IP addresses and timing of requests for the PDFs for example could make it much harder to inflate the values.How exactly does this translate from a thought experiment into reality? The fact that something can be done (technically) in principle is different from it being practical, efficient, easy and economical in practice.
I am maybe showing my ignorance here, but although I can see in principle that one could find ways to counter web spiders, robots, spam, etc; and ways to stop an author and all his/her friends doing multiple downloads from different IP3 addresses, and so on, how would this be done in principle? Would you need a whole new organisation like COUNTER or ISI to “officially produce” the sanitised stats? Or other system?
How then would the system be quality-controlled and funded? -
The download issue is a minefield. What is not clear is how would anyone wishing to assess it take into account downloads from repositories? And what is the impact of paid-for versus open access articles?
Those of us assessing the use of repositories (the British Library is managing the UKPMC repository of biomedical publications) pours over our download stats every month to assess usage but we have to ask a lot of questions about what those figures mean. -
Actually the correlation is not at all impressive. Just look at Figure 1 (and remember that it is r-squared that matters, not r). Furthermore, as Aronson has pointed out, the writers of the editorial have distorted the data by using log scales for a linear measure, and they have not stated which of the two possible lines they have drawn through the data.
In any case all this is irrelevant until such time as somebody produces a convincing argument that we should be using citations to assess people. All you have to do is count citations of papers by someone whom you respect. Just look at the real-life examples here
It really is nonsense.
-
Maxine, I don’t know the amount of resources that would be necessary to make this a reliable metric. It also depends on the effort individuals would be willing to put into this to game the system. Since there are alternative and legit ways to promote something that has been published among our peers I suspect the majority of people won’t go through the trouble of trying complex ways of inflating the number of downloads.
The most obvious analogy would be the online advertisement and click fraud. Ideally it would best to collaborate with companies/groups that work on these problems already but I don’t know what would be in it for them.
The second possibility would be just to make downloads per paper available in some way and let others try to find ways (correlation with other information) that would spot inflated values.In any case there are other reasons why the usage information should be made available:
1 – authors want to know; One of the first things a blogger does for a new blog is find a way to track the readers. It is just nice to know who is reading your work so why should I as an authors of a scientific publication not have access to the same information about my articles as I have for my blog?
2- creative use of data; If this information is not available it would never be used. If it is, there will be someone playing with it and coming up with ideas that are not obvious right now. One example would be to include usage information in paper recommendation engines.Lee-Ann Coleman – Regarding the repositories and the issue with open access, each paper has an universal identifier so the number of downloads can be aggregated across different (trusted) sources. Ideally publishers should agree on a common way to communicate information about a DOI (number of downloads/comments/ratings/etc).
David Colquhoun – I am obviously too young an naive to know how decisions are made regarding grant applications and PI positions but my current impression is that there is a lot emphasis on the impact factor of the journals for the evaluation of individuals. I think usage is a far better metric than the impact factor of journals, also because it would make it easier to introduce the evaluation of other activities that are not so easily publishable. For example, people that create materials and reagents that are useful for many different applications don’t usually receive the credit that they deserve. Also, some scientific databases struggle to maintain funding, even if they are amazingly useful, because the funding serves for research (create the database) not for maintaining a useful resource.
-
Please re-read the editorial for the original motivation behind the analysis. The fact that any relationship existed at all between these two variables was striking enough to initiate a separate discussion regarding whether downloads could be used as one part of a greater metric evaluating scientific impact. It was clearly stated in both the editorial and the blog post that there are significant pitfalls and concerns with using downloads, as with any statistical evaluation of “impact”, but perhaps in conjunction with a variety of different measurements, we could begin to formulate a better method of manuscript analysis to replace the flawed and inaccurate practice of equating impact factor or citation counts with scientific quality.
It is naive to think that we can easily switch to a Utopian system where individual papers are not dissected under the microscope in an attempt to determine who is doing the best science. Although we may disagree with this practice, the truth of the matter is that faculty search committees and granting institutions regularly use impact factors and citations in their assessment of candidates or grant applications. Search committees are combining the track record (i.e. publication record) of the individual with their potential forward-looking visions to find the best fit for their department. Increasingly conservative grant study sections are trying to fund projects that will actually be successful, and make these decisions based on how well and often the PI has published in the past, a loose predictor of future success, at best. Since such decisions change the lives of scientists, and with these decisions influenced by impact factor and citation counts, it seems imperative to find a metric or equation that could more accurately reflect scientific importance. Whether other scientists actually read or seek out your paper is one variable in that large, complicated equation. The system or process of offering jobs or grant money in science is not going to change anytime soon, but it may be feasible to change the guiding principles (i.e. the measures of candidate or application “quality”) sooner.
David, regarding your strong criticism of the analysis, please remember that the correlation measures how well two variables influence one another, or whether they co-vary. Our data suggest a strong positive correlation between download statistics within the initial 90 days of article online publication and the eventual citation counts more than two years later. R-squared evaluates the goodness-of-fit, in this case assessing our notion that the relationship between our two variables is in fact linear. The R-squared value for the Nature Neuroscience data is 0.41, while the value for the Nature Genetics data is 0.5. These are weak to moderate R-squared values, suggesting that a linear equation does not fully represent the data, but still, almost half of the variability in the data can be explained by the proposed relationship in the editorial. Not bad, considering the type of data we used.
It is possible that the R-squared value will continue to increase as the citation data becomes more mature. Unfortunately, due to data constraints, we could only use papers from 2005, but examining a much earlier cohort of papers could more accurately portray any potential relationship between downloads and citations.
Regarding the plots, the log scales were used strictly for visualization purposes, making it easier to see all of the individual data points. When the data is plotted on linear scales, as expected, a portion of the data clusters towards the origin, still leaving the relationship intact. Plotting on the log scale does not change the correlations or the R-squared values (which are now all listed), hardly “distorting” the data, or grounds for claiming that the analysis is “disingenuous”. In fact, we didn’t even have to present the graph, listing only the correlations (and now the R-squared values), since the graph doesn’t provide the grounds for the statistical relationship. The linear scale plots will be published on the Action Potential blog for your benefit.
I take issue with your harsh tone both here and in your comments on the blog, as it does not seem conducive to initiating a civil and productive conversation about the big-picture issues raised in your posts, or hinted at in this forum. Ironically, I think that all of us are on the same page with regards to changing the way that people view citation counts. Therefore, it is good to have a lively discussion surrounding what can and cannot work to fix things, but there is no need to insult our attempts at stimulating conversation or dismissing our preliminary analysis as dishonest or nonsense.
That aside, I’d like to continue this conversation by debating alternative and practical strategies for breaking our impact factor/citation count habit, since we can only move forward as far as our creativity will take us.
-
Pedro — you write: I suspect the majority of people won’t go through the trouble of trying complex ways of inflating the number of downloads. We are hearing all the time of “citation misconduct”, which was one of the subjects discussed by Tim Birkhead and others at the citation meeting that kicked off this forum, but was hardly unknown before that. People cite their friends or those who agree with their own scientific view; it has been alleged (but not, to my knowledge, demonstrated) that some journals specifically ask authors to cite papers in that journal as a condition of publication, and so on.
I do agree with you, Pedro, of course, that authors want to know metrics and statistics such as number of downloads, who is citing their papers, and so on. However, this is not the same thing as suggesting that these stas are a reliable, robust and demonstrably better indicator than Impact Factors and other citation metrics, for formal purposes such as grants, tenure and new positions. For example, setting up a system to deal with the issues raised by Lee-Ann would be possible but non-trivial, I suggest – how confident can we be that it would be worth it, and that we would not end up with another system that everyone would criticise as they do “Impact Factors” (but for other reasons as well as for some of the same ones)?
Noah: you write alternative and practical strategies for breaking our impact factor/citation count habit; this good point is the intention of the BL citation science group, as I understand it, and one of the purposes of this post-meeting forum, so I for one look forward to some more ideas and proposals.
-
The connection between article downloads and citation is not novel, and yet the discussants so far have missed one important issue: Reader behaviour is heavily influenced by the user interface of the electronic journal.
In a study we published in 2006 (http://dx.doi.org/10.1002/asi.20405), we determined that different publisher interfaces had a significant (and sometimes huge) influence on the pattern of article downloads. Moreover, the same journal hosted on two different platforms (HighWire Press and Nature’s own interface) showed dramatically different download patterns.
In sum, if you want to move to a model where articles (and their authors) are evaluated based on downloads, you will need to take the online interface into consideration.
see:
Davis, Philip M, and Jason S Price. “Ejournal Interface Can Influence Usage Statistics: Implications for Libraries, Publishers, and Project Counter.” Journal of the American Society for Information Science and Technology 57, no. 9 (2006): 1243-48. http://dx.doi.org/10.1002/asi.20405 -
Hi Phil. Definitely the connection is not novel, however, it had not really been explored beyond physics preprints and in a couple of other smaller datasets of publications, to my knowledge.
Your study is interesting, but the HTML-PDF download ratio is not an issue here since we only focused on the PDF downloads for the comparisons.
I agree that there are many pitfalls to this sort of connection, and enormous problems and hurdles to overcome before making downloads a more legitimate metric. However, the purpose here was to get a discussion started as to how some or all of these hurdles can be minimized or eliminated.
Results
-
