Recent replies to "Who wants to be eating dirt? Musings on search, self-archiving, citations and "findability" "

Reply from William Gunn

William Gunn — Sat, 07 Jun 2008 16:40:34 -0000

I think the decline in impact factor of Cell Press has a lot to do with their findability, and the ease of findability of NPG articles certainly hasn’t hurt them.

Reply from Maxine Clarke

Maxine Clarke — Fri, 23 May 2008 10:48:42 -0000

I’ve discovered a new forum on Nature Network called Citation in Science. I think it is great to have a focus to discuss the issues in Allan’s topic list (at the link). Please join if you are interested in continuing the conversation there.

Reply from Cameron Neylon

Cameron Neylon — Sat, 10 May 2008 21:12:53 -0000

I think Brian’s point really goes to the heart of the argument. He says ‘the implications…[of being scooped]...for the student will appear much more serious, possibly the end of the world’

Yet our response to this is not to put make a clear claim ‘we have done this, we think this’ which would protect the student, but to delay; actually risking precisely the thing we are trying to avoid. The potential consequences are devastating yet we constantly play this game of brinksmanship. And this is despite the fact that the work would probably go faster and be more complete and comprehensive with other’s input.

My view is; if you make a clear, easily findable, and unambiguous claim to a piece of data or an idea, whether or not that claim is peer reviewed, for someone else to describe the data or idea in a peer reviewed publication without attribution is unethical behaviour of the sort that should get people tarred, feathered, and run out town.

The problem in my view is building a system that makes these claims easily and effectively findable. Currently it is possible for someone to publish a paper and say later ‘it wasn’t in PubMed/WOK/whatever so I didn’t see it’. If we can change that then I think we can make the whole endeavour more productive.

Science is not a zero sum game. We all win when more people make a contribution (paraphrased from Deepak Singh)

Reply from Brian Derby

Brian Derby — Fri, 09 May 2008 22:44:02 -0000

Hilary, Implications of others beating you to it are different depending on your position. If you are a tenured academic (such as me) you curse and move on. However, for the student it will appear a much more serious issue, possibly even the end of the world.

Of course things aren’t really that simple, it is highly unlikely that someone else will have exactly the same model as you are proposing. You have a good idea from the literature how others see the problem and you can be cofident that they won’t spring any surprises. Of course I may be wrong (in my work) and that is why no-one else is doing it!

I don’t really like preprint servers other than as a way of setting out something you are already submitting.

Reply from Hilary Spencer

Hilary Spencer — Fri, 09 May 2008 20:18:38 -0000

Hi Brian – Would you post your work on a preprint server when you feel it is complete? If you wouldn’t, why do you think your nemesis would? And finally, if you did find that your nemesis had posted something, what would that mean for your research? (I’m assuming here that you scan ArXiv… but if you use another preprint server, please let us know.)

Reply from Brian Derby

Brian Derby — Fri, 09 May 2008 20:11:37 -0000

I am coming to this lively discussion a little late. Possibly because all the examples are in biology and a dumb physical scientist like me gets put off by long words. What is interesting in my field (materials science), is that in the good old days when the internet was not even science fiction (and I hasten to add before my time) there were many examples of duplicate discovery. These now have double barrelled names. The Frank-Reid source, Nabarro-Herring creep, the Hall-Petch relation. Of course citation metrics had not been thought of either in those halcyon long gone days.

More seriously, you are all correct, priority is all these days. There is a real temptation to publish early when your work is not complete to make sure you get in there first. I am living in nervous horror at the moment. I am working with a student on a new model for nanomechanical deformation that no-one else has seen – even though it is blindingly obvious. I nervously scan preprint servers for my nemesis, a frisson opf horror when I see the title, a quick read of the abstract and we are safe. When can we publish? But we must be complete in our argument and model, or at least complete enough before we dare publish.

Reply from Maxine Clarke

Maxine Clarke — Thu, 08 May 2008 15:17:47 -0000

David—I accidentally discovered once that to make strike-throughs you type a dash before and after the words to be so struck. It works.

Reply from Cameron Neylon

Cameron Neylon — Sun, 04 May 2008 13:59:40 -0000

Hilary, I probably was a bit unclear. What I meant to suggest was that once the tools work reasonably well then the excuse ‘I didn’t see that example of prior work’ just doesn’t cut it any more so that strong attribution should be much easier to enforce.

But I am absolutely with you on the idea that there are much more important things than ‘being first’. Particularly when things get rushed out in half baked form.

David makes an interesting point though. I’d not really thought that one through but he is perfectly right to suggest that if you can’t cite the appropriate source in any specific medium then you shouldn’t use that medium. I don’t know how you balance that against the (perceived?) need for citations in journal articles to be stable over extended periods of time.

Reply from Santosh Patnaik

Santosh Patnaik — Sat, 03 May 2008 23:06:17 -0000

Regarding differences in citations of (almost) simultaneously-published articles reporting (almost) identical observations

As pointed out before, a difference could reflect other aspects of the work such as techniques that were developed or reagents that were generated during its course.

For the examples given in the posts here (the works on the auxin receptor and on ALS/astrocytes), ‘findability’ or access to full-text cannot be a reason for the differential citation as both papers of the pairs are published in the same journal.

Could it be that there actually is no significant difference when citations in original research articles are tallied? I have tried to do so using Google Scholar.

.... Auxin receptor work
........ Total citations: 267 vs 232
........ Citations in original research: 122 vs 117

.... ALS/astrocyte work
........ Total citations: 42 vs 35
........ Citations in original research: 14 vs 15

Reply from David Whitlock

David Whitlock — Sat, 03 May 2008 18:00:20 -0000

If I might add a few thoughts to the excellent comment that Cameron made. If you use someone else’s work that requires citation and you don’t cite it, you have committed scientific misconduct. If the journal you would like to publish in won’t allow you to cite what you need to cite to avoid scientific misconduct, then you can’t publish your work in that journal.

If you come across a brilliant idea in a blog comment (such as one of my comments;) and want to expand on it such that it requires citation, and a journal such as PreterNature (to use a fictitious name) won’t allow such a citation, then you cannot publish a derivative work expanding on the brilliant idea in PreterNature no matter how brilliant that derivative work is.

The reasoning behind why a journal would set up a system for scientists to communicate (such as in a blog/comment system), and then not allow citations to that communication system should sufficiently important and brilliant ideas be communicated is not something I understand. Is it their purpose to stifle scientific discussions? To limit discussions to mundane ideas? To enforce the editors’ monopoly power on what constitutes “science”?

Perhaps Maxine or Henry Gee could comment on that? I suspect it is simply like the many other petty inane stupid nonsensical arcane rules of all most some science editors, it seemed like a good idea at the time. Or perhaps the editors feel their stylistic whims should dictate where and how science is done as well as communicated?

(note, I had wanted to use strike through in some of the adjectives that I used to make my intent appear more humorous (instead of harsh) but couldn’t find an explanation of how to do so. The formatting explanatory link didn’t explain how to do so, and I am having great difficulty with my (dial-up) ISP connection (now at 28.8 kbs) and can’t find it elsewhere.)

Reply from Hilary Spencer

Hilary Spencer — Fri, 02 May 2008 17:00:02 -0000

Then the issue of findability goes away because everything is linked

I’m not actually sure that the issue of “findability” goes away… I think that the question of findability has to do with what tools we’re using to find things. Just because something is linked to doesn’t mean that it won’t still be buried. Journals in a library are easy to find if you’re using the card catalog, but Google won’t tell you which shelf to look on. Highly linked documents online are easy to find using Google because of their page rank algorithm. But if, in 10 years, everyone is using some other tool, then other articles might be more “findable”.

Most of the time, authors don’t need to worry about this—it’s why they publish in journals, and why journals have websites, issue press releases, and send copies to libraries. What I was hoping to suggest is that with the constant focus on being first to publish, other factors related to credit are perhaps being overlooked.

Reply from Cameron Neylon

Cameron Neylon — Fri, 02 May 2008 15:21:26 -0000

Great post and discussion. I would say that the central issues revolve around precedence and attribution (which in terms is influenced by citability). The issue is who did something ‘first’. And this is why most journals include submission date as well as publication date on papers. The submission date is the clear third party certified date on when this work was done.

Now my argument would be that to get the earliest possible recognition of what you’ve done you should release the raw data and your conclusions immediately to get yourself precedence. The problem with that is, as Hilary questioned, will anyone find it, and if so how do they cite it. We’re at a slightly awkward point in terms of the technology at the moment but in a few years there won’t be any excuse for not searching and finding these things. But we need a way of citing them.

To put it another way, I’m hardly going to leave my brilliant new scientific idea in this comment because a) I can’t cite it – it has no independent existence and b) I don’t know that this post will be here in two years time. We need to link up all the elements of the scientific process, proposals, data, papers, and discussion, and make it all citeable and linked up. Then the issue of findability goes away because everything is linked, the issue of citeability goes away, because each element has its own existence, and if there are third party or reliable datestamps there can be no argument over who did a specific thing first.

And with my ‘open science’ hat on. If in the example Noah gave these people had known that they were both doing the same thing, how much time and effort could have been saved? Maybe no-one had to eat dirt? Would the papers have been better had the two teams worked together?

Reply from Maxine Clarke

Maxine Clarke — Fri, 02 May 2008 15:13:51 -0000

Chris writes: if two papers report the same thing at the same time in the same place they should by rights get cited together and so get the same number of citations. One shouldn’t get cited more simply due to relatively arbitrary decision of a the journal’s editors, and yet it invariably does.

The subsequent discussion has focused on citation counts – and I am sure that Scopus, ISI and GS are accurate only to a certain degree (not the same in all cases, and not in a predictably systematic way in all cases).

But apart from that aspect, two papers published simultaneously will often be of different quality. I can think of two obvious examples from my own field. In the first of these, the first paper was far skimpier than the second. After a while, nobody cited it, everyone cited the second, solid study. In the second example, one journal published two papers and another journal published an inferior paper at the same time (all by the same group, as it happens). Again, nobody cited the inferior paper.

I think it would be very hard to perform a systematic study of this fascinating question because of qualitative factors such as the relative merits of the publications concerned, quite apart from errors within the citation databases (or number of downloads, or whatever metric is being used).

Reply from Hilary Spencer

Hilary Spencer — Fri, 02 May 2008 14:25:53 -0000

Whoops!—mistyped the Scopus count above. It’s 213 vs. 199.

Reply from Noah Gray

Noah Gray — Fri, 02 May 2008 14:19:56 -0000

Thanks for the example Chris. I guess I was not expecting a systematic study, but had hoped that perhaps you had already seen data out there from others who had completed such a study. FYI, Scopus results give 213 and 199 for those two citations respectively. And ISI gives 201 and 194, respectively.

I think that one has to be extremely careful with which numbers are used. Google’s counts typically include citations from any paper listed on any server anywhere, including multiple versions of the same paper referring back to the original. Therefore, there is a strong chance of counting the same citation twice at different locations. Scopus does state that it includes pre-print servers in their counts. I couldn’t find the exact resources for ISI, but I believe that they may only use post-print published materials from journals (online or otherwise) in their counts.

Therefore, taking all of that information into account, the relationship between TOC location and citations breaks down quite a bit for these two papers (201 vs. 194). Even the Scopus data is a lot closer together. Is this really a systematic bias towards citing the first paper? I don’t know, but I find it hard to believe.

The funny part is that when one does a PubMed search for “TIR1 auxin receptor”, you actually get the second paper listed first, since PubMed displays manuscript PMIDs in descending order. Since I tried to argue above that people would usually search PubMed to flesh out their bibliography with exactly the search I used, your positional premise combined with my bibliography-building premise would suggest that the second paper would actually be cited more since authors would come across that paper first in PubMed. Well, the citation numbers don’t bear this out, so the way the argument is structured is important.

Perhaps the (TOC) first paper used different techniques that were more widely adopted by the field, was more detailed in their analysis of the receptor, or provided insights not mentioned by the second paper. All of those reasons could cause an increase in citation numbers as authors referred back to a particular diagnostic or experimental protocol in that particular paper. And by a quirk of editorial fate, it just happened to be published with earlier page numbers.

A recent example off the top of my head from my journal:

“Non–cell autonomous effect of glia on motor neurons in an embryonic stem cell–based ALS model.”
Francesco Paolo Di Giorgio, Monica A Carrasco, Michelle C Siao, Tom Maniatis & Kevin Eggan
Nature Neuroscience 10, 608 – 614 (2007) 34 Citations

“Astrocytes expressing ALS-linked mutated SOD1 release factors selectively toxic to motor neurons.”
Makiko Nagai, Diane B Re, Tetsuya Nagata, Alcmène Chalazonitis, Thomas M Jessell, Hynek Wichterle & Serge Przedborski
Nature Neuroscience 10, 615 – 622 (2007) 41 Citations

(For the record, Google Scholar gave 37 and 44 citations, respectively, to these papers…)

Opposite result, but obviously, not enough time has passed to allow these manuscripts to have thoroughly sunk into the literature. In this case, I would argue that the Przedborski paper may continue to increase in citations over the Eggan paper simply because the former relies on a primary culture system that is more likely to be adopted and used by other researchers, as opposed to the embryonic stem cell-derived motor neurons used in the latter.

Anyway, these are all interesting topics to debate, but I am still curious as to both yours and Hilary’s thoughts on what I had suggested in an earlier post here. Namely, that although OA articles may on average receive more citations, with increased author diligence to archive their papers openly when they are able to, does a 6 month head start by the OA paper really make that big of a difference in the eventual overall citations? Has anyone seen research out there where the “closed-access” papers were parsed into bins representing those that were immediately and openly self-archived anywhere (everywhere) versus those where the author failed to complete this task? It would be interesting to see if self-archiving allows the non-OA papers to catch up in citation counts…

Do I get the award for the longest comment ever on a NN forum posting?

Reply from Hilary Spencer

Hilary Spencer — Fri, 02 May 2008 14:15:10 -0000

From Scopus, the count is 213 vs. 299.

Reply from Chris Surridge

Chris Surridge — Fri, 02 May 2008 08:24:55 -0000

To answer Noah I’m going to provide anecdote rather than evidence I’m afraid as I haven’t time right now to do a systematic study. Also I’ve only got access to Google Scholar for citation numbers.

Anyway, compare two reports of the identification of the auxin receptor from 2005:

Dharmasiri N., Dharmasiri S. & Estelle M. “The F-box protein TIR1 is an auxin receptor.” Nature 435, 441-445.

Kepinski S. & Leyser O. “The Arabidopsis F-box protein TIR1 is an auxin receptor.” Nature 435, 446-451.

Same issue of the same journal, back-to-back, practically the same title AND submitted on the same day. So why has the first got 267 citations and the second only 236?

Reply from Noah Gray

Noah Gray — Thu, 01 May 2008 18:56:32 -0000

Hi Chris,
I guess I find it hard to believe that there is a categorical pattern of papers being cited more because they are physically situated first in the publication order (TOC) of a journal. To only cite one paper when two clearly present the same advance from the same journal issue is borderline unethical. In addition, a good portion of most bibliographies are built from PubMed searches where TOCs and publication order become irrelevant.

The ArXiv paper doesn’t really look into this back-to-back issue, but it is almost equally as surprising. Basically, there was no good explanation for this effect since the authors could not disentangle “self-promotion” bias from the “visibility” bias (although these are somewhat related). With manuscripts representing different sub-disciplines semi-randomized within the list (since the pre-prints are ordered according to submission time), perhaps this effect arises from a few very competitive sub-disciplines that consistently grapple for the top position of the list every day, with the intent to increase visibility. If these same sub-disciplines were also higher producers of citations, then the mystery would be solved: a competitive subset of the community chronically fights for visibility (self-promotion), leading to a disproportionate clustering of their papers (from highly cited fields) near the top of the list. When comparing these studies to papers lower on the list (which could disproportionately arise from low-citation rate sub-disciplines that do not bias their submission time to make the top of the next day’s list), there will be an obvious effect.

In short, normalize for subject matter and see if the effect disappears.

Reply from Hilary Spencer

Hilary Spencer — Thu, 01 May 2008 17:27:12 -0000

I was also just reminded of a recent study on positioning on ArXiv:
The Importance of Being First: Position Dependent Citation Rates on arXiv:astro-ph

We find that e-prints appearing at or near the top of the astro-ph mailings receive significantly more citations than those further down the list.

Reply from Hilary Spencer

Hilary Spencer — Thu, 01 May 2008 17:17:55 -0000

Chris—so by “the one appearing first”, you mean the one appearing first in the TOC? (I initially thought you meant the one appearing first in time). That’s really interesting.