An “PNAS envy” essay on Nobel Intent told me that:
…Past performance is, realistically, the only way to judge future performance. Past performance can only be assessed by looking at their publications. Were they in a similar area? Are they considered significant? Are they numerous? Curiously, though, the second question is also answered by looking at publications—if a topic is considered significant, then there will be lots of publications in that area, and those publications will be of more general interest, and so end up in higher ranking journals.
So we end up in the situation that the editors of major journals are in the position to influence the direction of scientific funding, meaning that there is a huge incentive for everyone to make damn sure that their work ends up in Science or Nature…
And the envy kept through the essay until the author addressed that:
[T]here are some steps that could be taken. First, Science and Nature should either eliminate their News and Views section, or implement a policy of not reporting on their own articles. This would open up one of the major sources of “science news for scientists” to stories originating in other journals.
Second, scientific competence should not be judged by how many citations the author’s work has received or where it was published. Instead, we should consider using a mathematical graph analysis to look at the networks of publications and citations, which should help us judge how central to a field a particular researcher is. This would have the positive influence of a publication mattering less than who thought it was important.
UPDATE: There’s already been a heated thread in the Publishing in the New Millennium: A Forum on Publishing in the Biosciences forum (quite a long title) started by Anna Kushnir
Isn’t the logic of the first excerpt wrong way round? Science journals are looking to publish the best science from what is submitted (by scientists) to them.
The News and Views editors at Nature, and I guess the Perspectives editors at Science, are looking to publish articles about the most exciting science. They are not looking at where the articles themselves are published, but at the science that is described in them.
I doubt these editors have the least interest in science funding, but they certainly are interested in publishing articles describing great science.
There is a very good article at the same website (Nobel Intent) here, by John Timmer, which you might also like to read (it is referred to in the Nature Network forum you mention in your post).
Incidentally, your first quoted sentence: “Past performance is, realistically, the only way to judge future performance” has been applied rather disastrously to financial investments and markets.
You’re putting the blame on the wrong place. The problem isn’t the journal or the editors. The problem is a culture that accepts this to be true
“Past performance can only be assessed by looking at their publications. Were they in a similar area? Are they considered significant? Are they numerous? Curiously, though, the second question is also answered by looking at publications—if a topic is considered significant, then there will be lots of publications in that area, and those publications will be of more general interest, and so end up in higher ranking journals.”
Did anybody notice nobody mentioned actually reading the publications? It’s just about counting: how many, how well cited, what’s the h-index, what’s the impact factor. This is simply bullshit. It’s a reliance on oversimplified measures that harms science tremendously if they become accepted and institutionalized for their own sake. It’s the same problem as in our economic system: a fast and easy assesment of ‘success’ that favours an incredible short-term thinking. It doesn’t result in sustainable progress.
For more info, read my post: We have only ourselves to judge on each other
I agree with Maxine. As a journal editor I am only interested in the science behind a paper, and the impact these findings have. It should not matter where you come from, what your back ground is, what you have published before, how many arms and legs you have, and so on. Otherwise, would you have trusted some patent examiner to come up with revolutionary scientific ideas?
Obviously, we have to be rather strict in our selection, and there are many really good paper that do not make the cut. So indeed, there is excellent science in many other journals, and as Sabine says, obsessing too much about Nature and Science is a disservice to science.
Nevertheless, there is of course some expectation that papers published in journals like ours do represent a certain quality. But indeed, this is only part of the story of scientific discovery, and at least in terms of numbers, a small piece of the cake.
Actually, there is a suggestion that applicants for funding, positions, etc. should provide a short list of say five of their most relevant publications, which are then considered in scientific detail by the relevant committee instead of metrics such as h-factors or number of publications. What do you think about something like that?
I certainly agree with Joerg and Sabine that a journal editor reads the papers that are submitted as the first and most important step, also looks at context (as a scientific paper is representative of a body of work). I do not know of any journal editor who looks at citation rankings of scientists in any way, h index or other, in deciding whether to consider a paper – as Sabine says, they are too busy reading it!
I agree that impact factors are misused. Here is a post I wrote at the Nature Network citation in science forum about that. There is a lot of discussion in that forum about citation metrics.
Joerg: So far as I know, “relevant publications” are used in many intsitutions for funding and other decisions. In the last UK research assessment exercise, impact factors were excluded. So I do not think this metric obsession is universal, though as everyone is incredibly busy always, I am sure that some indicators are used in some committees and panels as useful rules-of-thumb.
I think the Science/Nature/Cell “obsession” is limited to certain fields. Some disciplines are not so hung up on it and they are all the better for it. However, I can’t resist commenting that it is all too easy to “blame” journal editors (there are some ignorant comments at the Nobel Intent post to which Andrew links), but anyone can submit their work to any journal he or she likes.
As someone who has never (even tried) published in Science or Nature I can assure you that it is possible to get tenure and receive grants from a range of agencies without that particular badge of honour. In my opinion (with apologies to the masters of NN) the possession of a Nature/Science paper on your CV (resume) is no longer as powerful as it was 10 – 20 years ago. At present the h-factor is king of the (dubious) metrics and it is definitely that which is examined by appointment and promotion panels. I daresay that in 10 years from now, some other metric will be the rage.
The self-selected best five does seem to be the best measure of a person’s impact at present – although people do tend to select their papers from highest impact factor journals, rather than dispassionately selecting the five they believe to be their best.
I don’t think any apology is necessary, Brian!
Not all appointment/promotion panels use the h index – I know people on some of them who have never heard of the h index. I am sure they are used on some, though – and I trust you are showing them your own, various Nature, and other publications demonstrating some obvious flaws in that metric ;-)
errrr… I am not an advocate of that PNAS envy essay. I used blockquotes.
But thank you for your comments which are very informative to me!
In fact no one can be professional in all fields. By saying ‘We have only ourselves to judge on each other’, I think Sabine actually means that only processional in can properly judge other professionals in the same field. Is it true?
N&V is the reason Nature is as good and as widely read as it is. It’s too broad a church for everyone to read (let alone understand) every article, without N&V.
Oh, and Futures.
Andrew, I read the article too. Ars technica—you get what you pay for, and the emphasis is on the ‘ars’.
e