Straightforward rating?
Thomas Kluyver
Sunday, 15 February 2009 23:49 UTC
Amid all the problems that have been highlighted with impact factors based on citations, would it work just to have a number of humans rate a paper? I envisage something like a rapid peer-review process, where people are asked to give a score out of five for categories such as ‘methodological rigour’, ‘insight’ or ‘originality’. Of course, there are potential problems: maybe you can’t get a decent sample of raters (especially in very small specialised fields), or maybe the value of some science won’t be obvious until later. Does one try to blind the process, since researchers in related fields are likely to know one another*? To anyone who works in publishing, how variable are the reviews that typically come back on one paper?
*And just to play devil’s advocate again, we usually like to take personality/cronyism out of the equation, but would it encourage better co-operation if authors knew that reviewers could see their name, so had an incentive to befriend potential reviewers?
-
Replies
Jump to resultsResults
-
It is an interesting idea, but (1) how would you get them to do it; (2) how would you credentialise (weight) the person making the recommendation; and (3) how would you log it? A mechanism would be needed to control for all these factors and others.
-
Thomas, my big concern with this would be cost. You are essentially asking everyone to do a lot more peer review, even if it is lightweight. Indeed the UK Research Assessment Exercise has actually just abandoned an approach like this in favour of a “simpler” (certainly cheaper) metrics based approach primarily because of the cost of the running the RAE the old way.
Now if you could actually build this rating process into the things that we do whenever we handle a paper (i.e. tools for me to mark it as useful to me where the results of those ratings are available to other people), then we might get somewhere. But the key with these things is to find a way of incorporating it into what people are already doing. In a sense this is what tools like Connotea/Citeulike/Mendely/Papers are working towards.
See also: comments here and a related conversation
-
Getting people to do it:
How do we currently get peer reviewers? Payment? Could smaller payments be used for an equivalently smaller task? Or would it just end up with people clicking numbers to get cash? I’m aware that I’m answering questions with more questions.Alternatively, some sort of karma system? Not sure what one would buy with the accumulated karma, though.
I would personally avoid any weighting of ratings by credentials, except that, like peer-review, raters would be selected to have appropriate credentials.
Post-publication review:
I think this is likely to suffer from a similar sort of popularity effect as citations. You would also need quite a robust authentication system to avoid the sort of ‘click fraud’ that Christine Pikas pointed out (thanks for the links, Cameron).Crucially, this idea of reviewer ratings is different because: reviewers can’t see each other’s ratings when they rate, and reviewers would be encouraged to specifically rate quality and rigour, not just whether it’s a fascinating idea. (Judging by the comments on friendfeed, I don’t think I made this clear enough before)
Of course, this does have the downside that revolutionary science may be rated very low because it goes against the reviewers’ preconceived notions. Anyone working in publishing, do you encounter this with peer-review of revolutionary papers?
-
How we get people to peer review is an interesting question that keeps coming around because whenever anyone tries to get someone to do something similar they find it very difficult but very few people would refuse to do peer review. A lot of it is momentum and history of course but it is an interesting (and crucial) sociological issue. If you can solve this then a lot of people (not least journal editors looking for referees) will be very happy!
The other concern I would have is with timeframes. When is the best time to judge a paper? Just after publication, 5 years, 10 years, 20 years? Depends on the field and the nature of the paper I would guess. Some are obviously important, many are sleepers, and I guess many never wake up at all!
-
Isn’t this the kind of thing that Nature tried a few years back and it tanked? Maybe it was “too early” at the time, but still.
-
@Cameron
When I was an editor (academic) of a (high profile in my field) Journal, I found that a few people refused to referee. They tended to justify their actions by using an argument that went something like this… “i have decided to limit my reviewing to N paper this year, your request is number N+1 so i am returning it”. Needless to say N was small, typically <5. These were usually very research active people (one was an FRS) who would publish > N papers in a year. -
Peer reviewing is part of the golden circle. People see peer-review as a community service. They peer-review papers, and do it well, because they feel when they submit a paper, it will be well peer-reviewed by someone like them.
-
Very interesting. I wonder: if an editor were to experimentally get reviewers to rate papers along the lines I described above, how much would the ratings of different reviewers co-incide?
-
Thomas, I am sure you mean well but your suggestion seems to me to lack any sense of practicality. For it to work, you have to say how it would work, why people would do it, who would pay for it, etc.
It is all very well to criticise a system that seems imperfect, but part of the role of a critic is to suggest some practical alternative that will work. I haven’t read that here, yet! -
@Brian – I’m sure there have always been people who felt they were too busy to stoop to such things. But I am getting the distinct impression that there is a perception that the load is increasing and that people are increasingly putting things off or making excuses. Presumably the volume of peer reviewing must be going up as the number of articles does but it isn’t clear to me whether the overall burden really is increasing. Nonetheless I probably say that I can’t do about a third of the requests that come to me. I am concerned in the long term about the costs and whether they are sustainable – but then we’ve had that discussion before I think :-)
Results
-
