Following a conversation on the reasons for doing field vs lab experiments in evolutionary ecology, our lunch discussion today drifted inexorably back to the pros and cons of the Editorial/review process, that we all enjoy bitching about discussing so thoughtfully, from single and double blinding, to the depth of knowledge of subject editors and referees.
“Not again, Mike”, I hear you wail (and gnash your teeth), “you’ve been over this before, and the only people who gave a crap were the über sensitive editors”.
But wait, as Jimmy Carson was so fond of burring, “there’s more”.
It took an interesting twist, when a new Post-doc in our department, Dra. Maria del Mar Delgado was asked “How would you improve the review process?”
Immediately, she replied “Professional reviewers”. After a brief pause for thought, the other gnarled, bitter postdoc and I both agreed excitedly.
This is a revolutionary spark fascinating idea.
It doesn’t need to be a permanent position, but asking early and mid-career scientists to take a break from writing grant proposals (cos no-one ever actually has time to do research for themselves at this stage1) for a couple of years, and do nothing but get properly acquainted with the current and past literature while reviewing cutting edge research in a field, is likely to be an extremely attractive proposition to some people.
It would look great on the CV of those who wanted to continue in academia, or go into professional editorial jobs, or into other worlds beyond. It could help reduce the burden on less specialised subject editors and the unpaid work referees do at the moment, which gluttonously munches into their proposal writing time.
It could even work on a part time basis, if other funding could also be provided on the other part of time not spent reviewing professionally. Maybe some people only want a part time job anyway, a point that’s been raised in various discussions on NN.
There are still likely to be problems associated with this, but most of these are inherent in the current system2. But I like the idea. I’d be tempted to do it. And if I felt I could spend a reasonable amount of time (>1 day) really reviewing a paper thoroughly, I’d feel much better about the quality of my own work.
So, is this pie in the sky, or a potentially interesting development to improve the quality of published scientific literature?
There are worlds beyond?
It sounds like a great idea, but “who would pay”? is the obvious question. Ideas in Ecology and Evolution has a system where the referees get paid, but so far they only seem to have had one submission. It might work if an author pays for submitting an article, otherwise the costs could get silly (because someone has to pay for all those rejections). I guess if a publishing house has a large stable, and reviews can be passed around, it might work better.
There are worlds beyond,
DorothyBob. I’ve read about them in fairy stories and daily newspapers.Yep, the cost is an issue. But before giving up the idea because of perceived cost, I think a good discussion about other pros and cons would be interesting. Costs being an issue in other parts of the publishing process are being addressed in a variety of interesting ways, Open Access among them.
And following (secret?) discussion elsewhere, I like the shameless self promotion in your comment1. You could actually have linked to Ideas in Ecology and Evolution, but linked to your own post instead. Nice!
1 Don’t take this as an attack. Please. Then nobody would comment on my blogs…
Would these people be freelance, or employed by Universities, or by the journals? They will need very good access to the literature, which currently requires subscriptions to be in place.
Will they be seen as part of the research community, or part of the publishing community? “Us” or “them”?
The way I see it, the problem with such a break for an early or mid-career scientist is that they still need to get the lab funded, and also keep an eye on the projects going on in the lab too. In the current economic situation, if you stop writing grant proposals for a couple of years…how likely is it that you’ll still get them when you decide to go back in the game?
Frank: Freelance funding would be hard to come by for this sort of position I think. Universities provide a clear conflict of interest, which would negate one potential benefit of the system as I imagined it – namely, avoiding competition between labs in the review process (it may be a sound paper, but our group/department/faculty/uni has people working on a similar project so if I can hold this up I should).
I think the journals could be in the best position to pay for this. But really, I don’t care about the funding at this point in time, more about a discussion of the other pros and cons. Would it avoid petty conflict? Issues with blinding and anonymity? More importantly, would it provide a body of more rapid, more reliable, more knowledgeable reviewers?
Cristian: early career scientists (post-doc) won’t have a lab to run yet, and getting their head properly around the literature would be an enormous benefit in the future grant writing process. You can really identify where info is missing or a field needs to be synthesised after spending a couple of years just reading and thinking about it. I don’t see why “the current economic situation” would really be a hindrance to that.
Of course, I’m a theoretician at the moment, so running a lab means something a little different to me!
And Frank & Cristian, thanks for making my footnote to Bob irrelevant ;)
Hmmm…..thinks about a journal that uses the kind services of tens of thousands of refs.
Ahhh, but Maxine, if you paid a few reliable folks for each subject area, that would hopefully cut down the number you poor overworked editors had to deal with, the number of invited reviewers who don’t have time or have other reasons to avoid reviewing, the number who agree but do it late or don’t do it at all in the end…
Actually, freelancing could work rather nicely in this situation. The best freelancers would get more invitations.
Mike, two possible problems:
I am also worried that you suggest only professional reviewers would know the literature in their field properly…
Steffi, these are good points. I don’t see it as a universal option for all journals. As you say, it wouldn’t be that feasible for some smaller, more specialised journals.
But the larger journals, who get many more submissions than they can publish, could cut down on some of the problems of soliciting high quality, unbiased peer reviews by having their own professional referees, or contacting freelancers. If this happened, it would reduce the load on the rest of the community, who would then have more time to concentrate on their own projects and reviewing for more specialised journals (for no fee).
I have trouble keeping up with the literature in my fields – I make no bones about it. People who review my work also sometimes clearly don’t know the literature. I’m afraid I’m basing my suggestion about this on personal experience. It may not apply to everybody, but it does apply to some.
But please post a list of all these journals that have trouble filling their pages – I have a large pile of
drossunpublished work just waiting to find the right outlet!But please post a list of all these journals that have trouble filling their pages
Sorry, we don’t :)
When you say you have trouble keeping up, is this because there’s simply too much or you have no time at all for reading? And do you think this is the rule?
This is an interesting series of comments. However, many of the solutions proposed would strike at the heart of the concept of peer review. Only by using reviewers who are active researchers can we be sure that the review process won’t artificially hold science (publication) to a current path.
Refereeeing may be a chore but it is a necessary part of the way we have constructed the scientific project in the latter half of the 20th century and in the 21st century.
As tempting as your idea sounds from the point of view of a postdoc, I think there is potential difficulty for re-insertion should the individual taking time out want to get back to performing (or enabling the performance of) research. No matter when that is. (cf. Jennifer’s blog.)
Just taking four months off each for two babies at the end of my Ph.D./beginning of the postdoc period made it extremely hard to pick up again at the bench – but at the time I wasn’t doing much theoretical and most was hands-on. And the more time out, the less specialized and useful the referee becomes as a referee, and the more like a sub-editor. Which is a career in itself.
“cut down on some of the problems of soliciting high quality, unbiased peer reviews by having their own professional referees, or contacting freelancers”
Is is such a problem obtaining high quality, apparently unbiased peer reviews for the small donation of time to the scientific establishment that it currently represents? Because currently, journals get that input for free. So if everyone keeps bringing it back to funding, then it is because the added value of paying for reviews is not obvious when we project ourselves into the mindset of a publisher.
We scientists are the freelancers, and we perform the service for free. In my case, that is for a couple of reasons:
- I subscribe to a sort of civic idea, rather like paying taxes, that if I benefit from a service I should somehow pay into the system that gives it to me, and I would rather pay in kind than pay in cash as I have more time than money;
- I enjoy reading articles in my field before they get published; it makes me feel like I have a briefly privileged perspective, and I can integrate their ideas into my mindset that much sooner.
Senior scientists, who get more solicitations than they might be able to undertake, may be more interested in the money-for-service option. But usually they can and do farm out the review for free to their trainees – it gives the trainee practice, and saves the senior time (I only approve of it if the senior actually takes back the review and makes it their own before submitting it, otherwise it is more exploitation than a learning experience for the trainee).
As Heather put it, I don’t really see a period in the standard research career that would allow for this. When you’re still in grad school, you’re probably too inexperienced to be refereeing. During your post-doc years, everybody seems to agree that the more you publish, the better your chances to (eventually) get a faculty position, so you really want to spend as much time on the bench as possible then. And if you finally get that coveted position, things become even more frantic, with the need to setup your own lab, recruit and supervise people and apply for funding so you don’t have to close it down in a few years’ time. So, taking time off to be a professional referee would definitively hurt your career, unless it’s clear for you from the beginning that you’re not interested in a traditional academic career (it would be yet another alternative career for people with PhDs instead, which is fine, but then you’ll have to think about doing it for more than a couple of years…)
Brian:
I disagree. Researchers following any path are likely to hold their chosen methods above some others. If they review an article that doesn’t follow their methodology, they may reject it on those grounds, therefore holding science to a particular path. It doesn’t matter if you’re actively researching or not, you’re likely to have a favoured methodology. Having the time to properly get your head around alternative methods is likely to be extremely useful when it comes to reviewing unfamiliar work.
Heather: Reinsertion is certainly a problem in other situations, but I think most people agree that this should and can change. In fact, taking time of, e.g., to have and raise kids (for men or women), might provide an excellent pool of highly trained professional reviewers. You remain actively involved with the scientific community but are not tied to a lab bench. This could be the sort of initiative that help people return at a running pace after time away from the bench.
Elsewhere, Ian Brooks highlighted that only around 20% of the tens of thousands of post-doc researchers in North America end up in tenured positions. Why would we spend vast amounts of public money training and employing these guys over a number of years only to lose a hefty proportion of them from the scientific community? They also represent a potential pool of professional reviewers.
It may be that some fields are more suited to this than others. My field (theoretical ecology) perhaps doesn’t sprint along as fast as some others that benefit from rapid technological advances – although increases in computing power mean more complex models can be built and simulated in a shorter time, we have to ask a whole raft of new questions relating to this increased complexity in comparison with simpler models, which means understanding all the assumptions behind both simple and complex models.
Heather: I hope you’re not suggesting reviewers solicit advice from others (grad students) and take the credit from themselves? This is really dodgy – the article in review is not meant to be seen by anyone other than the reviewer. I think if someone doesn’t have enough time to do a review, they should say so to the Ed., then recommend their student/postdoc/colleague as an alternative, if they believe they are suitably qualified. But agreeing to review then passing it on to someone else is misrepresentation and not ethically sound for anyone involved.
I received a review a year or so ago where the referee claimed to understand what I was saying, but complained that a student of theirs did not understand all the terms in the MS. I had a couple of problems with that:
I’ll add that I rather enjoy refereeing articles. I particularly enjoy reading things that use similar methods but are in slightly different fields – I can safely evaluate the modelling part, but learning about new applications is always interesting. I’ve benefited and suffered as an author from the process as it currently works. It sort of works, I don’t disagree with that. I’m still amazed that some articles actually get published (including some of my own). But exploring alternatives is relevant and an interesting idea to me – which will hopefully avoid the scientific community becoming too dogmatic about its publication methods and standards.
You’re right – it was not a suggestion per se. (I agree with the sentiment in “But agreeing to review then passing it on to someone else is misrepresentation and not ethically sound for anyone involved”.) However, given that “if someone doesn’t have enough time to do a review, they should say so to the Ed., then recommend their student/postdoc/colleague as an alternative, if they believe they are suitably qualified” happens much less in my experience than the former situation, the best compromise is for the senior to at least reappropriate the review before submitting it. Like the junior colleague is a sort of reviewer’s reviewer – and the senior should sum up the most cogent parts of the junior’s argument (while checking that assertions are true!) before submitting the review in the end. I wouldn’t call that “taking credit” – reviewing is a pretty thankless task and I don’t think very useful on a CV – but I can imagine that it’s a very fine line.
It could be a teaching situation and might avoid some pretty awful reviews along the lines of personal attacks under the cover of anonymity, or the lazy reviewer who just checks off accept as is with no comments at all.
“Professional reviewer”.
Honestly, I don’t think I can dream up a science job I would want less than this. Just imagine the reams and reams of irrelevant background literature you’d have to read in order to do the job well.
Heather: I didn’t really think you were advocating that – I should have added some tongue in cheek punctuation to make that clear ;oP
But if you’ve got enough time to read, understand and rewrite a student’s interpretation of a paper, then show them how to do it properly, you should probably just review the MS yourself.
Richard: if it’s background literature you have to read to do the job well, then it’s probably not irrelevant…
But I’m sure there are plenty more people out there who are interested in/suited to this sort of work. Just as some researchers would make lousy lecturers, editors or public faces for their field.
Or perhaps I just missed your tongue filled cheek in the first read, Richard. Like I said, smiley punctuation works wonders in these situations.
Mmm, yes, perhaps I should have clarified…
‘irrelevant’ meaning ‘irrelevant to my scientific interests’
and yes, at least one ;) was in order :D :P etc.
I think a journal is more fresh, and more open to publishing papers with new concepts and challenging ideas, if it uses the widest possible reviewer base from within the scientific community – within its subject remit (which, for Nature, is large). Using only a small group of referees would, I think, tend towards conservativism, and result in less variety of perspective.