A cut too far
Maxine Clarke
Wednesday, 25 March 2009 19:14 UTC
UK researchers are rightly outraged at one funding council’s decision to exclude certain applicants, as described in an Editorial (free to access online) in Nature this week (458, 385-386; 2009). The Engineering and Physical Sciences Research Council (EPSRC), a national research-funding body, has decided to stop serially unsuccessful applicants from submitting any more grant proposals for a year (see news story on page 391 of the same issue; subscription required). By 24 March, more than 1,200 protesters had signed an online petition demanding that the policy be repealed. “The feeling in the community is that it is draconian and deeply unfair,” says Philip Moriarty, a physicist at the University of Nottingham, UK. He and other scientists contacted by Nature say they will refuse to review their colleagues’ work under such a system.
No other funding body, whether in the United Kingdom, the rest of Europe or the United States has attempted to formally exclude scientists in this way. The anger triggered by the move tells a cautionary tale: the policy is misguided, however urgently the EPSRC needs to relieve its overburdened peer-review system, states the Nature Editorial. “A ban on researchers — even those whose consistent lack of success disproportionately overburdens the system — is a clumsy way to try to break this cycle. When success rates are so low, the peer-review system cannot reliably identify the worst performers: rankings can vary so much from one reviewer to the next that many solid proposals end up being rejected along with the weak ones, just by the luck of the draw. And even if the system were reliable, the scientists involved have no time to adjust: the policy is being applied retrospectively. Worse still, this temporary ban could easily leave a permanent stain — particularly on the careers of young researchers.” Is the EPSRC alienating the very scientists its system depends on? What would be a better solution?
-
Replies
-
I’m reposting here Cameron Neylon’s and Philip Moriarty’s online comments to the Nature News story -
The statements made here by EPSRC and Wakeham seem to fail to address two critical issues which must surely like at the centre of a discussion of this proposal. Firstly is blacklisting morally and ethically acceptable? It is clear that many scientists feel strongly it is not. But far more importantly is the proposed criteria. Blacklisting will essentially destroy a researcher’s career. Therefore it is critical that any measure used to do this be robust, objective, and widely trusted. Peer review as a method of numerically ranking proposals and papers is known to be unreliable. Studies on this in peer review of grants (Cole et al, Science, 214, 881, 1981; Hodgson, J Clin Epedem, 50, 1189, 1998) and of papers (Rothwell and Martin, Brain, 123, 1954, 2000; Bornmann and Lutz, Learned Publishing, 22, 117, 2009) have consistently shown that peer review performs poorly at ranking, that it is subject to serious random effects, and specifically that it is poor at distinguishing between the majority of proposals and papers that fall between the very top and very bottom. Correlation between the results when a peer review process is repeated are marginally better than that expected by chance. Thus whether a given proposal falls in the top or bottom 50% of the panel ranking is to a very large extent determined by chance, and has little or no relation to any objective measure of the proposal’s quality. This is the key measure that EPSRC propose to use. The current proposal therefore amounts to destroying people’s careers on the basis of a measure which is only slightly better than flipping a coin. Disclaimer: This comment represents my personal opinion and does not represent the policy, opinion or position of either of my employers (I am employed by one of the UK research councils, STFC, and by the University of Southampton). Cameron Neylon.
and response to it from Philip Moriaty:
On Cameron’s rather important point above, it’s worth noting that David Delpy admitted at an open EPSRC meeting in Nottingham last year that, “We all know the peer review process is a lottery”. In addition, the suggestion that the changes to peer review were introduced as a response to the RCUK consultation on peer review is laughable, given that EPSRC chose to ignore extremely negative feedback via that “consultation” from practically every UK university on the subject of the imposition of economic impact criteria in peer review. This negative feedback is discussed at the blog of Steven Hill, the Head of RCUK’s Strategy Unit . Where in the consultation did RCUK ask the acadcemic community for their comments on the retrospective introducion of a ban on researchers applying for grants? Moreover, to suggest that EPSRC introduced this policy because it was listening to advice from the academic community begs the question as to why they didn’t follow the advice from UK universities on the imposition of economic impact criteria in peer review. Philip Moriaty.See also Katherine Haxton’s Nature Network post on this topic, and subsequent discussion .
-
I have applied twice for a Wellcome grant – a hugely time-consuming business – and although I got some very positive feedback each time I was turned down. It was not until I went to a session with the assessors of the award (which involved brain-storming and examples of applications of what was successful, as well as talks from individuals that had been successful) that I realised what I was doing wrong. It showed me that what I wanted to do didn’t actually match what they funded – although that was not at all clear from their description of the award.
Maybe such a day would be valuable here – the applicants and the council taking time to show what they would fund and what they are looking for.
I have to say that reading this piece was very disheartening. If only incremental changes are being funded then there will be no chance for real innovation requiring risk and vision (IMO!).
-
Only just picked up on this entry! Clare, the problem here isn’t that we don’t know what EPSRC fund, people have a good idea of that. The problem is simply there are too many proposals and that makes it a bit of a lottery. Now we whine about this all the time but at one level, you buy your ticket and you takes your chances and everyone knew that was the way it worked.
To turn around and change the rules in this kind of way, and to change them retrospectively, is only slightly short of barmy, and will penalize people in an appalling way for acting in a way which was essentially recommended six months ago. It’s a little like retrospectively introducing the death penalty for anyone getting a speeding ticket in the last two years.
-
What a powerful analogy!
There’s an update on The Great Beyond, the Nature News editors’ blog, on 3 April (links in post):
229 scientists face EPSRC exclusion
Two hundred and twenty-nine scientists will be barred from making grant applications to the UK’s Engineering and Physical Sciences Research Council (EPSRC) for a year, from 1 June.
The exclusion policy for serially unsuccessful applicants, announced by EPSRC on 12 March, aims to relieve the pressure on an overloaded system that currently peer reviews all grant applications.
An online petition protesting against the ban has attracted almost 1700 signatures as of 3 April. Some scientists thought many more than EPSRC’s ballpark estimate of 250 researchers would be affected.
EPSRC has now confirmed the exact number and tells Nature News it intends to send letters to individuals next week.
-
