Nature Precedings forum: topic
This is a public forum
Nature Precedings needs to have a good rating system
Santosh Patnaik
Thursday, 21 June 2007 18:51 UTC
Nature Precedings needs to have a good rating system for open, community-based review to work well. Currently, submitted articles can be voted for, but that does not tell one how many would have voted against it. Nor does one get to know the negative points unless they go through the whole article themselves. Such negative points may have been mentioned in some comments but they are not easy to spot. Further, one is usually disinclined to write textual comments unless one has a strong interest to do so.
With open preprint systems, being able to find useful and reliable ideas and data in articles is perhaps more important than being able to submit one. This becomes apparent as the number of articles increase, when searching can return hundreds and thousands of articles. One can’t go through all of them, and a few ‘bad’ articles can easily cause frustration and distrust in the quality of the submissions.
But if search criteria can include objective measures of article quality, then one can indeed easily find valuable material. Nature Precedings should therefore opt for a point-based rating system where different aspects of articles can be appraised.
Thus, instead of just letting one vote for an article, one should be allowed to rate its different aspects on, say, a 1-5 scale. Such aspects can include:
1. clarity
2. originality
3. novelty
4. presence and quality of experimental data
5. logical procession
6. depth
7. proper referencing
In effect, this would be a proper peer-review system.
The ratings, both their average and their spread, should be displayed alongside articles.
A good review/rating system will discourage submission of bad articles, build trust in the usability and reliability of content in Nature Precedings, and encourage quality submissions.
-
Replies
-
Hi Santosh, so this is where you put it too :) as suggested in my blog 1. Below is a copy of what I replied in my blog:
"
Dear anonymous, I think these are good suggestions. Might you add the link where you posted these comments?I can agree about increasing the flexibility in browsing and getting informed about new submissions. At this moment, for example, you cannot browse just the papers, but always have the posters/presentation document types too.
It would also be a good idea to be able to read separately on those seven categories, to allow even more personalization of what you want to get informed about.
"1.http://chem-bla-ics.blogspot.com/2007/06/preprint-servers-cps-failed-how-will.html
-
I absolutely agree with your suggestions about improving the rating system and am frankly really curious why at least positive ~and negative voting weren’t allowed from the start. (I suspect there is a well-thought out reason and I’d like to hear what it was).
Its really interesting why there seems to be so much difficulty getting comments on N.P. papers when, in comparison, it seems relatively easy to get them on blogs. It was apparently difficult to get comments even on ‘real’ potential Nature papers ! I think comments are really the most valuable things, from the perspective of the authors, that come out of pre-publishing, so it seems critical for N.P. to improve on this somehow.
Some ideas:
1) develop ways to offer public recognition for commenters
- make it easy to see all of the comments from a particular person
- keep track of how many comments people post and display it somewhere on their profile
- add ratings for the comments themselves
2) make it easy for people not yet registered with nature to post comments
- possibly through OpenId
- allow anonymous comments
1 provides a simple, free, rewards system for commenting that sometimes works and 2 makes it easier for people to say the nasty things that need to be said. There is a good reason for normal peer review to be anonymous and I really don’t see why peer review on the web should be forced away from this.
Why can I hide my identity in this forum but not on N.P. ?
I’m sure there is an answer, lets hear it. -
Hi All,
This is an intersting suggestion, however, the idea of placing negative comments or negative ratings may not be constructive in the long-term. Many of the “seven” metrics you suggest are great, albeit very subjective (as are most measures of quality). To what standard are these articles being judged? Does the field that the article is targeted toward have a broad publication outlet or is it restricted to a few publications that are read by that group? In many instances I believe that places like NP are going to serve as “parking-spots” for data/manuscripts. This is to say that here is a forum for people to post their information that likely won’t be in the publication form for a long time (if ever). Therefore, there is likely a raw feel to the information because it is not “complete”. So rating on these criteria still may not give a good picture of the future impact of an article on a particular field or to a broader set of fields.
Craig
-
This is beginning to sound like the peer-review process ;-)
-
This content has been removed by the forum moderators.
-