Straightforward rating?
Thomas Kluyver
Sunday, 15 February 2009 23:49 UTC
Amid all the problems that have been highlighted with impact factors based on citations, would it work just to have a number of humans rate a paper? I envisage something like a rapid peer-review process, where people are asked to give a score out of five for categories such as ‘methodological rigour’, ‘insight’ or ‘originality’. Of course, there are potential problems: maybe you can’t get a decent sample of raters (especially in very small specialised fields), or maybe the value of some science won’t be obvious until later. Does one try to blind the process, since researchers in related fields are likely to know one another*? To anyone who works in publishing, how variable are the reviews that typically come back on one paper?
*And just to play devil’s advocate again, we usually like to take personality/cronyism out of the equation, but would it encourage better co-operation if authors knew that reviewers could see their name, so had an incentive to befriend potential reviewers?
-
Replies
Jump to resultsResults
-
Just about every bit of work is already peer-reviewed twice. Once at the time you apply for the grant, and once when you submit the paper. I see no need to have yet another review, in the form (in the UK) of the RAE and the REF,
This is particularly true as the funding agencies move towards full economic costing of grants. Insofar as the full costs of the work are paid for, and peer-reviewed by, the funding agency, the assessment of research ceases to be anything to do with HEFCE. The third layer of review that they impose is not needed at all, and insofar as all the methods of assessment that have been proposed are desperately unsatisfactory, they can do only harm.
-
Maxine, sorry, I don’t have enough experience of the system to know what would work and what wouldn’t. Please consider this to be the online equivalent of thinking aloud!
-
Is this the sort of thing you had in mind (from Gunther Eysenbach’s blog last year):
The Medicine 2.0 conference in Toronto is currently conducting a unique experiment, comparing the traditional method of peer-reviewing submitted abstracts through invited experts, head-to-head with a “Digg”-like open peer-review mechanism to vote for submitted abstracts
This list of reading material on recent developments relevant to peer view may be useful background to the field.
-
Frank, that’s very interesting. Does anyone know how Eysenbach’s experiment turned out?
Results
-
