Time (and the ability to read to the end of this post) will tell whether this entry turns into a bitter rant, but today I’d like to bring up the ever popular topic of how we judge our colleagues’ work, bringing up a couple of frustrating and interesting paradoxes. The Peer Review Process: if it ain’t fixed, don’t broke it.
I’ll apologise now for the length of this post, but I haven’t posted for a few weeks, so consider it a punishment exercise. For you me. Let’s start with a concept that is often invoked to differentiate (in some people’s opinion, to elevate) science above other fields of intellectual pursuit; the application of dogma.
As scientists, we’re supposed to avoid being dogmatic about how we think about the world. Yet, in order to communicate effectively, to compare, contrast and or relate our work with others in the same or different fields, we tend to invoke scientific paradigms
So, when is a paradigm really different to a dogma? This might not be such an easy question to answer, but let’s suggest that paradigms are there to be shifted, dogmas aren’t.
Henry G posted a related idea which touched on published discussions of scientific concepts, where opinions differ about different approaches. Sometimes I love reading these sort of things, as they can provide a great summary of some important philosophical issues in a field. Other times it’s tedious semantic claptrap. Yet, the beauty of this is that the discussion is now in the “public” domain1, and all interested readers have access to different viewpoints and can judge from the published literature which is the most appealing paradigm. Although the proponents may be dogmatic in their views, by having a public discussion, one side can at least potentially gain more support than the other, and the discipline/paradigm can move on from there.
However, in the review process, novel ideas or interpretations can easily get blocked before getting aired in front of a wider audience. There are a few different reasons for this, the first I’ll discuss here is reviewer naïvité, based on personal experience.
I was first invited to review papers for ecological journals as a PhD student. I clearly wasn’t qualified as an “expert” in the field, but I rarely had to deal with author responses that showed me up for what I was: nought but a naïve academic babe (in the youthful, rather than sexy sense). This could be because handling editors swiftly realized I was a buffoon, incapable of sensibly evaluating the work of my “peers”, or because I luckily managed to highlight some salient errors with the work I was evaluating. I’m sure there were other PhD students who were capable of evaluating work in a more mature way than I was, but there are also likely to be others like me – holding up the publication process, affecting other individual’s access to grants and jobs…Yoinks! Just writing about it gives me a terrible sense of guilt.
As I’ve become more academically mature, I feel I’ve mellowed somewhat in certain aspects of my approach to my and others’ work – I look for different things now, including how effectively ideas are communicated – but I’m also finding that I am on the wrong end of some buffoonish reviews. Karma’s a cruel mistress. I am always tempted to write back to the Editor and describe in great detail why I know more about the subject than the narrow minded prig charitable reviewer who spewed forth uninformed drivel about graciously gave their time and reviewed my latest groundbreaking discoveries humble offering. Recently, I’ve actually started to write back, surprisingly with some success. But it’s tiring. It’s also highlighted the relative lack of understanding amongst supposed “expert” reviewers in my field. I know most Editors are extremely short of time, but I’ve felt that some reviewers’ comments should have been weeded out before getting back to me, rather than me having to work very hard at weeding them out for the Editor in a diplomatic way.
When submitting a new manuscript for review, a common review process might go something like this: Managing/Head Editor receives MS with cover letter, and makes an initial decision based on reading the abstract and cover letter. If deemed interesting enough and at an appropriate level for the journal, it will be passed onto a more specialised handling Editor, who should have a reasonably good understanding of the field of the paper. Handling Ed then selects a couple of “expert” reviewers, who should have an excellent grasp of the field. I guess many editors are likely to select people they already know as reviewers, so may be more likely to accept their opinions about the quality of the work over those of an unknown author. This introduces a potential bias in the process (I’ve probably benefited from this before), and says nothing of the amount of effort an author puts into researching and writing an article, compared to a referee reading briefly through it, although many reviews are extremely thorough and helpful.
A philosophical problem also arises at the level of the MS being given to the experts. If the work is ground-breaking, either in terms of novelty, or reversing some previously held view, it is extending the current boundaries of knowledge. How easy is it for people still working within those boundaries to accurately judge the quality of the work?
There are flaws in the current system (some financial2), but we currently accept it. Is it possible to come up with other ways of evaluating each others work in a rigorous fashion? Is there already a discussion about improving the status quo?
Some “alternatives” within the traditional peer review paradigm revolve around more or less blinding of authors or reviewers, and have been discussed extensively on NN3. These aren’t necessarily so different from the current system though – you still have one (supposedly expert) person evaluating the work of another. It doesn’t get around the fact that the person doing the original research could easily know more about a given system than the reviewer. Another possibility was brought up (and humanely put down) by Noah Gray e.g., sequential reviewing just the other day, and I agree that reading colleagues’ peer reviews is an excellent learning resource. I need to read up about the peer review consortium idea before I say anything about that though, but it looks very interesting. The idea of dropping confidential comments is pretty useful to me – if a reviewer can’t state a concern to an author, what chance does the author have to respond legitimately to that concern4?
The other model that I’ve heard about is the “Publish at your peril” approach – where work is uploaded for all to access and comment on at will. There are associated issues with this that make it problematic though – professional jealousy and bullying amongst them.
I should also point out somewhere that an important lesson I’ve learnt during my post-doc is that communication of ideas is often far more important than the quality of an idea (see also here. If you can communicate a poor idea in an appealing way (e.g., involve sex wherever possible), it is more likely to be published than a great idea communicated poorly. Neither situation is desirable, but one is still getting published, quite probably the wrong one.
I could be experiencing this because I’m working at the interface of rather different fields (maths & biology), which makes communicating to your peers awkward, as they tend to speak different languages. I think someone once said “If music be the food of love, then maths be the lingo of science”. If they didn’t, they should have. Or perhaps they did: Chi = dirac delta (kronecker multiplication) Matrix[music, 0; 0 maths].Vector[love;science].
So, is there another new alternative to the peer review process, that can improve on the current models? I didn’t want this to be a long moan about the publication process, but I’d love to hear about alternatives or plus sides that I’ve ignored or glossed over. With the increase in the number of papers submitted for consideration, can we really continue with the current review model as authors, reviewers and Editors?5
Why do we work as reviewers for free, then pay to publish and read articles for the profit of a publishing house? Let’s go on strike and/or only work for open access journals!
Is that you talking, or a rhetorical piece of irony?
Well, if someone likes to claim the review process is broken, before going too much into the alternatives it might make sense to look some of the underlying issue. Typical arguments against the present system often used are the sheer flood of refereeing requests as well as the work of the referees itself (bias, inexperience, incompetence).
As for the workload, well as Noah says in the blog you mention, journals have to be tougher in their initial editorial screening. At Nature Materials we reject about 80-85% of all submissions on an editorial basis (still providing a relatively safe margin compared to our overall acceptance rate of 6%).
As for the second part concerning the quality of the refereeing, well, as a journal one can do a lot to tackle such problems, too. Sitting in an office pushing manuscripts will not help much here, and personally I find it invaluable going out to labs and conferences, engaging the community of your authors. This way you “discover” good new referees, you learn more about the hot topics in the various fields, and you hear a lot of gossip, too… also, background knowledge gained there makes it easier to separate the well-told story containing no science from the convoluted paper that is a really significant advance. I guess it can be expensive doing lots of travelling for a journal, but if you want quality in the editorial process it is crucial to provider editors with the means to do a minimum of travel, I think.
Also, I certainly disagree that referees at journals such as ours are always from the same pool. The most I used any referee in 2008 was 4 times, and this includes revisions of manuscripts! Most I only use once or twice, i.e. on one study per year only.
As to the financial aspects of peer reviewing – well as a regular author you make contributions to the process but you also receive them. Simple calculation: each paper you submit to a journal gets peer reviewed by 2 or 3 referees. If you go down the chain a couple of times (from the likes of Nature to the journal a paper gets eventually published), this also adds to it. So essentially, everyone say should at least referee 5 times the number of papers they submit to journals every year. If you include co-authors as well, there should not be much of a problem here, particularly if editors provide a strong layer of prescreening.
For the majority of cases I think the present system offers a good premise, although from my perspective as an editor of course we need to constantly check our performance. There are a few options where refereeing becomes difficult and might quickly lose sense – either because the field is extremely small (posting on preprint servers may be considered sufficient), or the field is extremely fast-moving. Recent discoveries in a new class of superconductors meant that important new milestones were achieved almost every week, and nobody cares much in the short term where the stuff ends up a few months down the line – marking your claim and priority on a preprint server is more important in such cases as well. But these examples so far appear the exception rather than the norm.
As for alternatives in peer reviewing, nothing I heard so far for alternatives of actual peer reviewing has convinced me of their suitability. The only possible argument I could see is to abandon this process entirely, at least for the highly specific archival where “peer review” is a rather hollow phrase for a pseudo-process.
Thanks for the other viewpoints, Joerg. It’s good to hear what goes on from the other side, although I think there is considerable variation between fields, and among journals within a field. There was a paper1 10 years ago in one of my subject areas (community ecology) which identified an enormous problem with semantics that led to a lot of unnecessary confusion in the field. I still don’t think all the researchers in this field appreciate these problems yet, so confusion remains an unnecessary problem. I think the size of a field, the niches that exist within a field and the background of people within a field have an enormous impact on the review process.
@Maxine – I think there is a really interesting discussion about the financial costs and benefits of the current “goodwill” approach to reviewing. It’s something I wanted to keep separate from this post though, so I won’t answer your question here, I’m afraid. Nothing personal :)
1 Grimm & Wissel (1997) Babel, or the ecological stability discussions: an inventory and analysis of terminology and a guide for avoiding confusion. Oecologia 109 323-334. doi:10.1007/s004420050090
These guys list 163 definitions of 70 different “stability” concepts relating to Ecology!
Mike: you write I won’t answer your question here, but you wrote that statement in your blog post, and I’m asking whether you think this or if you are reporting someone else’s views as it was not clear in the text of your post. I don’t understand the problem of answering the question – as you wrote the post?!
I find it hard to distil a particular point from your post. However, I’d say that the peer-review process as practiced works very well for everyone concerned. People who peer-review mss know that they will get their mss peer-reviewed when they are authors themselves, so it is not a “free service” but a “community service”.
Peer-review has critics, and it is not a perfect process, but I have yet to see any better alternative put forward. It works in the sense that the scientific record is out there and built upon.
Most journals these days can peer-review and publish manuscirpts online very quickly. It is true that there are some journals, including the Nature titles, that have very high reject rates and so if authors submit they are taking a risk. But they know that when they choose to submit. These journals are open about the fact that they select on criteria other than technical accuracy as judged by peer-review, of necessity. If authors want better odds of getting a paper publsihed that passes techical peer review, then they can choose a journal that selects on those criteria – and there are many excellent such journals being published.
A small point of fact: you mention the neuroscience peer-review consortium. NPG has been operating such a system for many years, whereby authors whose papers have been declined can transfer referees’ reports with their resubmissions to another journal published by NPG (and do this directly without havin to go through all the submission logistics again to the second journal). Cell Press has a similar system, and so I believe do the PLOS journals.
Also to echo what Joerg writes: the vast majority referees used by the Nature journals, Nature included, referee once. Many referee twice over a year or two. A much smaller proportion referees more than that, but the maximum is around 5. To put it another way, each Nature journal has a database of tens of thousands of peer-reviewers.
I could write more about this but will spare you. I can’t see any clear reasoning for your comments about ideas being blocked, etc. It seems a rather blanket generalisation.
Plenty more information at the Nature journals’ peer review policy pages, the peer-review debate and peer to peer blog. One of the conclusions of our peer-review trial (open peer review) was not so much bullying or even fear of being scooped, but lack of time. People did not have any time or motivation to undertake a “wisdom of the crowds” approach to a ms being up for public peer review – and we tried hard to persuade people in the field to comment when we posted mss onto the server.
I need to clarify – the Nature journals never publish a ms that has not passed technical peer review. But they often have to decline those that have.
Maxine, the point I tried to avoid making was simply that for profit journals rely on the unpaid services of academics to maintain integrity and remain part of the established academic literature. I’m certainly not accusing Editors or other journal workers of benefiting from this practice, but someone, somewhere is, if the profit making part of the publisher is working properly.
Reviewing is hard work. I’m generally happy to do it in an academic sense – I learn from it, and I want to repay the favour to the people who review my work properly1. Publishing houses profit financially from this goodwill. Government (and private) funding bodies ultimately pay for it, through department overheads which pay journal page charges and library subsciptions… and in many cases, it’s still not freely available to the public after that.
My post is really long. I tried to chop it down, but didn’t find an easy way to do that I’m afraid. It’s one reason I’m blogging: to improve my clarity and become more concise. More practice required! But if I can distill it to a couple of major points, it’d probably be
1 I’ve really suffered in the past year or so from some bad reviews. Demonstrably bad, based on factual errors and misrepresentation by reviewers. This has delayed publication of my papers, which directly affects my access to grants and jobs. This may be specific to my field, or even my ability to communicate well, but I also read published papers that have basic errors in them, e.g., incomplete methods sections that don’t allow proper replication. This becomes very frustrating! How can we judge the “quality” of someone’s science if we can’t even try it for ourselves?
Commercial publishers spend a huge amount of money on managing the peer-review process, Mike. It is massively expensive. Count up how many professional editors are employed by these journals, how much is invested in the technical and office systems, etc.
I feel you are confounding issues here. A “publishing model” is surely irrelevant to the quality of the peer-review process. Would you say that for example society journals’ peer review process is invariably superior to the commercial publishers’ process?
We are clearly going to have to agree to disagree here, as I believe that “jouranls” (whatever their publishing model) manage the peer-review process “for the scientific community”. The peer-reviewers are the scientists, who are also the authors. Any journal or publisher that is not perceived by scientists (that is you, Mike) as providing a good community service will not get papers submitted to it or scientists to peer-review for it.
I have just, this afternoon, reviewed a paper. It was a resubmission in light of my and another referee’s comments. I was able to see these other comments in the revised MS as the author made reference to them in the cover letter. It was instructive to note that our comments did not overlap at all. The paper was multidisciplinary and our comments reflected what I assume is our backgrounds accross the disciplines. I think the resubmitted paper is a great improvement on the original and I trust (as I can only recommend) that it will now be published.
Refereeing helps sharpen your own paper writing skills and it is an essential aspect of the way we do science now.
When I was an Editor of a journal I got very irritated by people who replied to refee requests with variations on the following. “I have decided to referee N papers (N was usually around 3) papers this year. I have now reached this quota and I will do no more.” I get requests to act as a referee at least once a month, probably more often, I apply the following rules: 1) Am I currently active in the field (I sometimes get papers to referee on my PhD topic which was aeons ago), if not I don’t do it, 2) Do I have the time to return a review in the next 2 or 3 days because I know by experience if it slips more than that I will forget about it. If the answer is yes then I will usually do it.
@Mike: If you have a bad but wrong referees report say so in your reply. If it is factually incorrect and you can demonstrate it, then the referee will normally put it to one side. However, you should always ask whether the misunderstanding comes from some aspect of the way you have presented the work. Use referee’s comments constructively and you will write better papers in the future.
Good points, Brian, if I may say so.
Great post!
There is one particular issue that didn’t get much comments yet: it is surely harder to go against the “conventional wisdom” (let’s call it like that) in a particular field and get published anyway; and in this case it doesn’t seem to be necessarily related to the experience/understanding/degree of expertise of the reviewers but, on the contrary, to their attachment to certain views of what is relevant or worthy to be reported in their own field. It seems to me that what it takes for a novel approach to be taken seriously and deemed worthy to be used as a general framework for future studies is some big name backing it up, but this would essentially perpetuate already established hierarchies. Is the current system basically preventing innovation coming from smaller research groups? To what extent can editors go ahead with the publication of something they find worthy despite the opposition of the leading authorities in a given field?
I think this is one of the things that gets overlooked in the discussions and criticisms of peer review – the reviews are as much about improving the paper as about deciding whether to publish. It means that some one has (or should have) read it properly, and spotted the main problems. Even if the system isn’t perfect, it’s better than nothing. The idea that a manuscript is published and the “proper” peer review happens afterwards is appealing, but there’s no guarantee that a paper will be read and criticized.
Maxine’s point
People who peer-review mss know that they will get their mss peer-reviewed when they are authors themselves, so it is not a “free service” but a “community service”.
sums most of it up, for me.
Other points raised in the discussion include whether it is right o ask referees to do things for free when the journal is a commercial enterprise. Some of the answer is inherent in Maxine’s response above – referees who work ‘for free’ will expect to be able to submit their paper ‘for free’ to a subscription-only journal, but might have to pay submissions charges elsewhere. The point is that it costs money to produce a journal, and in the end that money has to come from somewhere.
Another interesting point raised above (by Christian) concerns innovation. At Nature nothing pleases us more to see a fabulous, entirely novel way of doing things (that’s what we’re here for) and it pleases me to find it come from people of whom I have never heard. But the same standards apply to every author, whether beginning graduate student or seasoned Nobelist.
Thanks for the responses, everyone. I feel the comments here show that my original post was a success in provoking a discussion, even though it’s kinda long winded and the main points might not be that obvious.
I think I’ll write a separate post about the financial aspects – I tried to write something here, but it started to become as long as the 1st post, and I feel it does differ a little from the question of how peer review works, and whether there are other models that may offer different pros & cons to the current system.
The motivation for the main question came from some experiences over the last couple of years where I felt that Editors had been almost anonymous in the review process, and I had to battle in darkness against reviewers that I thought were misrepresenting my work. I felt a lack of transparency in the process that was extremely frustrating, and the lack of editorial input left me wondering if they actually understood what the discussion was about. In some cases they didn’t, but their decision is what matters!
I came into work today to be greeted by the best editorial response I think I’ve ever had to a submission. The Ed. clearly outlined areas that we had to focus on based on reviewer comments, and demonstrated that they’d read and understood our MS and understood the relevance of the reviewer comments as well. Maybe the system isn’t as bad as I’d been feeling!
That sounds like an admission that the only papers of yours that are accepted are the ones the editors didn’t understand.
One thing I would really like to do if I ever get the time, is to post examples of “good” peer-review, with permission of authors and referees of course.
I am aware of various studies, mainly in the clinical trials/applied clinical literature, picking holes in published mss and concluding from that that the peer-review system is broken. But these studies are not looking at the right sample set. They should compare the submitted manuscript with the peer-reviewed version (as Bob and others are implying, and certainly true for authors published in Nature journals via our surveys, published authors overwhelmingly feel that peer-review has improved their mss).
This is a separate point from the “does peer-review stifle innovation?” question. Frankly I feel that sometimes it can do, but I can only throw the question back at the questioner, because the peer-reviewers are the scientists. They are judging each others’ work. It does not follow, to me, that if you jettison the peer-review process, you would then adopt new ideas more quickly – but perhaps someone can take that one up.
Of course, rejected authors don’t always feel so positive about the peer-review system that has judged them harshly on at least one occasion. Journals and editors should do all they can to ensure that their systems are fair, eg use two or three truly independent peer-reviewers per ms; show all the peer-reviewers all the comments on subsequent resubmissions and revisions; and allow reasoned appeals (i.e. appeals made on the basis of erroneous logic or other factual faults in the peer-reviewers’ arguments, rather than appeals based on emotion). Many journals publish details about their procedures on their websites so authors would be well advised to read these before submitting!
Sorry for the long comment.
@Bob—thanks, that gave me a belly laugh.
The paper I talked about in April took a hell of a long time to get through peer review. It was quite instructive, actually. Some really quite vicious reviewers obviously were defending their turf. But by realizing that the biologists didn’t understand the physics, and the physicists didn’t understand the biology, we were able to improve both parts.
Peer review isn’t, in itself, about testing whether something is correct. That’s called ‘reproducibility’. Peer review is about whether you’ve tested your (novel) idea correctly or not. Whether you’ve been logical, thorough: what you’ve missed. We had a new idea, and it took a long time to get it published, but now we’re pretty certain that it’s right (and it wasn’t published in Nature (you bastards) so we know it’s got a chance of being true).
@Maxine: Never apologize. Never.
As an editor I obviously disagree with the things said about editors not understanding papers. Either I pretty much understand the content of a paper and am able to take a decision based on that, or it is rather clear from evidence (scope, referee reports) that there is no need to understand a paper to the last detail. Obviously, the time editors spend on decisions, and hence the quality of the process, varies from journal to journal and publisher to publisher, and I leave comparisons up to your own experience.
As to compensating reviewers, as it’s been said, it is a service from the community to the community. If you think peer reviewing is for editorial decisions only and not also a service to the authors from your colleagues, it might explain some frustration you had in getting your papers published!
Finally, anyone thinking the peer review process is fundamentally flawed should launch their own journal based on a different assessment system. In the hard reality out there will quickly become obvious which system is the better. How would you run a journal as an editor, Mike?
Joerg, the simple answer to that question (and some shameless self promotion) can be found here (see also)
I would only agree to act as handling editor for a paper I felt I could understand in its correct context, hopefully even before reading any reviewer comments. We still have a relatively low submission rate so it’s not a problem I face too often, but this plug might change things…
The discussion here has been pretty supportive of the current peer review system. I was not advocating chucking it out entirely, merely highlighting potential problems and asking what alternatives exist?
I really think the sequential reviewing idea is an improvement – it allows editors to follow the different improvements that are suggested throughout a manuscript’s pre-publication life. From personal experience, this would have been extremely helpful!
From one paper’s initial submission, a reviewer suggested it was too abstract and I should add a further (simulation) study to make the paper more approachable to non-theoreticians. Later in a different journal, another reviewer said “The paper’s too long, chop out this ‘extra’ simulation bit”. Strangely, this same ref. complained that the paper was too theoretical and something had to be done to make it more approachable to non-theoreticians. This reviewer was acting for the Journal of theoretical Biology. Another reviewer for JTB suggested the ‘extra’ part was the only bit of interest in my study. Not a word of guidance from the Ed. about how I should deal with these opposing suggestions though. It’s just one example, but what is a boy to do? This is not a rhetorical question: how much guidance should an author reasonably expect from an Editor under those situations?
@Bob, Richard. Laugh it up, guys. I’m not even sure I understand some of what has my name on it anymore.
Just to answer your last question, in my opinion, yes, the editor certainly needs to provide guidance, and if not already done proactively in the decision letter, I am happy to provide assistance to authors on any query they may have.
As to sequential refereeing, I fully agree with Noah and Maxine’s comments over at the other post.
Maxine, that would be very useful. Because referee comments are confidential, you don’t get exposed to many before you have to do one yourself.
Rather than put up a specific reviewer’s comments, could you put some generic ones up that are amalgams of the best available? If you need a manuscript to “hang” them on, perhaps you could persuade Dr. Gee to finish his on the life cycle of carnostomids.