The Ioannidis Observation, and preprint repositories
Santosh Patnaik
Monday, 20 October 2008 03:22 UTC
Dr. John Ioannidis, and his colleagues, have brought forth a number of interesting observations on the nature of scientific publications.
A couple of papers describing their thoughts are listed below.
The latest issue of The Economist has an article covering these ideas.
I wonder how significant the “Ioannidis observation” is when one considers preprint publications in repositories like arXiv and Nature Precedings.
-
Replies
-
Not at all, given that the conclusions of these articles are based on the clinical-trials and clinical-research literature, which is not relevant to Nature Precedings or Archiv. There is not one shred of evidence that “most published research findings are false”.
The titles of the two articles you cite are highly misleading at best.
There has been a lot of discussion of these articles (and the Economist piece) on Nature Network and lots of other places online.
-
Certainly the second listed publication seems relevant for non-clinical science as well.
-
Most published research findings are false?
-
Quote: “Most published research findings are false?”
Let aside the title and consider the issues raised.
-
I don’t think these papers are really talking about forums like Precedings and arXiv. The argument is premised on publication space being limited, and goes on to claim that this can lead to a distorted view of science. Precedings accepts things like negative results and other ‘uninteresting’ findings (as long as they appear to be a genuine contribution from a real scientist), so if such an effect exists then I don’t think it would apply there. In fact, open forums such as Precedings and arXiv might help to negate any ‘Ioannidis effect’ that might exist elsewhere. (Maybe this was your point, Santosh.)
But I don’t really believe that this effect exists in practice. As a scientist who might have been an economist if given his time again, I found the Young et al. paper to be an interesting piece of theorising, but it doesn’t seem to me to bear much relation to the real world (outside areas like clinical trials).
To see why, consider the piece in the The Economist, which is uncharacteristically weak-headed. If you were to replace the following words and phrases in that article:
science → current affairs
findings → news
journals → newspapers
scientific papers → stories
scientists → business people and politicians
submissions → agency reports and press releases
negative results → good news
Nature → The Economist…then you’d end up with an article that explains why most of what you read in The Economist is wrong. It really is complete nonsense.
The reality is that in virtually all areas of society we rely to a large degree on editors of one type or another to separate out what’s interesting and important from everything else. If they do a good job then their publications will flourish, and if not then not.
True, the web allows other (I would argue, complementary) wisdom-of-crowds approaches. But we’re a long way from being able to do away with the judgement of good editors. In fact, automated and distributed ‘Web 2.0’ methods of filtering information are much more vulnerable to abuse and distortion of the kind that Young et al. worry about than a well-informed and honest human editor. There’s a whole industry that purports to be able to get your website onto the first page of a Google results set, or the top page of digg. I don’t know of any organisations that will offer to get your flaky results into the pages of a top journal. This isn’t to say that it never happens, but claims of a systematic bias are overblown to put it mildly.
-
Thanks, Timo.
I will just add my usual comment about the peer-review system: it works!
It is far too big a leap between some reports from some sections of the literature that reviewers miss some technical details, to concluding that the papers themselves are erroneous or that peer-review is not working. Unless you can compare the original submission with the final published version, these studies are speculations based on incomplete evidence. Journal editors do see the original submissions and this is one reason why we know that the peer-review system works.
Another reason that scientists know this is that the scientific record is reproducible and built-on. If there is an occasional error, it is corrected for the record and for future research in the area. This is far from the peer-review system not working and “most published research fundings are false”; the opposite in fact.
-
“Fundings” should read “findings” in my last sentence above.
-
