I just got turned down for a grant. That’s nothing new, you win some and you lose some, and every senior professor has gotten used to that over time. This time, however, I cannot find it in myself to just say “oh well” and let it go at that. This time, I think I need to go public, because I think what happened shows an endemic problem with the US National Science Foundation and, I hope, points out some things they could do to fix it.
The proposal was for an NSF “Expeditions in Computing” grant, and we put together a team of which the final panel review said “the research team is among the best in the world” (as well as me, the PI team included three winners of MacArthur “genius” awards and one of the top researchers in theoretical computer science). They said our vision was “compelling,” and that the proposal “outlines an ambitious research program to attain this vision, while at the same time with realistic ambitions as to what can be done within the resources of the expedition.”
There are in fact, no real criticisms in the review until the summary which after saying more nice things about the team and the vision and the technology (a potential new breakthrough technology for the Web) said
The panel was concerned with the management plan, in particular how to ensure collaboration among the diverse research communities, especially given the number of strong intellectual leaders among the team members.
Or, in my words, they were worried how a team of such mavericks could pull together.
In the end, NSF told us it had been “very very close” but that they were just not quite willing to deal with the risk and they could not fund us. But here’s the thing — this is supposed to be “high risk, high reward” research. In the end, NSF simply could not bring itself to take the risk!!
OK, now if it just ended there I would expect, rightly, for you to say “sour grapes” – but I hope if you’re a US taxpayer, and looking at NSF turning down what the reviewers said could be breakthrough research in new ways of using the Web, you’re annoyed at this. “What,” you might ask, “could be done to improve the situation?”
Here’s some thoughts I have based on having been both a researcher and a funding agent at different times in my life:
1 – Many funding agencies in this situation would be able to come back to the team and work with us to craft a management plan that would make sense to both sides – we wanted enough freedom to use our judgment in which paths to follow, they wanted some better assurance we would pull together. In feedback from some NSF PDs they have ideas of exactly what we “should have said,” but they couldn’t give us this feedback until after we were rejected. In short, NSF cannot work with a “near miss” to craft a winner, they can only say yes or no.
2 – Having been turned down, we can, of course, submit again next year. But the amount of effort that went into a proposal of this magnitude (we had 36 researchers in 11 different fields at 8 universities) is quite high, and next year we would be started again from scratch. NSF has no means where we could say “last year we almost made it and this year we fixed the management plan” – so we’d have to do preproposals, proposal if invited (no sure thing), site visit if we made it that far, etc (and all for different review panels who would have different questions and issues). There would be no advantage to having made it so far in the first round. Most funding agencies have a way of giving some sort of extra consideration in these sorts of cases, NSF doesn’t. Starting from scratch with the high likelihood of the same outcome (because this team wouldn’t be interested in doing anything if it isn’t high risk) is not a compelling idea, and we won’t be doing it, and NSF has nothing to offer to encourage us to do so.
3 – NSF reviewing is now all done by panels. In the old days, it was all done by external reviewers (as many countries still do it). There is a lot of anecdotal, and some formal, evidence that panels tend to be more conservative. On the other hand, the workload of external review was way too high. The solution might be, particularly in situations like this, to solicit outside review to be compared with the panels (or to comment on specific matters such as “does the management plan meet what you know of these reviewers”). The panel who were at the site visit only included one person who has done published research directly relating to our proposal, and a couple more who knew parts of it. Getting some people who would have pointed out that many of the people in the proposal had coauthored with each other, and we were all known for working in large groups, might have helped alleviate NSF’s fear.
I know NSF does a great job, putting up with constantly changing oversight from Congress, dealing with a public that doesn’t understand science, and trying their best to fund the best work while keeping the bureaucrats happy. Still, I must tell you that I am bitterly disappointed that they turned down a grant that had, as the reviewers themselves described it, “the potential to transform the use of widescale, networked computing” with _"a dream team of Web researchers and also top researchers in foundational computer science and social science." I would have loved to work with this team, and I’m sure many of us will now go and find other ways to pursue pieces of our vision. Knowing the team, this setback will delay this research getting into practice, but won’t stop us — we’re too "unmanageable"to give up. But I sure hope that the agency can understand why this turn down is more of a condemnation of them than us, and I hope the constructive solutions above can lead to NSF’s rethinking their approach to funding high-risk research.
addendum July 16, 2009
I was beginning to enter the “acceptance” stage of this rejection when I came across a 2005 report by the National Research Council (the organization that coordinates the National Academies in the US – the highest level scientific societies we have) entitled “Facilitating Interdisciplinary Research.” Their number one recommendation with respect to funding was
Funding organizations should recognize and take into consideration
in their programs and processes the unique challenges faced by
IDR with respect to risk, organizational mode, and time.
exactly what NSF did not do in this case – in fact, the exact reason they said they rejected it. Ahh well, they didn’t trust our research team which included a number of members of the National Academies, why should I expect them to listen to the NRC. Sigh – back to denial, anger and depression…
Sorry about the proposal, James – it sounds exciting, and you’ve made me rather curious about it. Do you have an idea of what other proposals you were up against – is it possible that they were of the same quality and just fit NSF’s picture better without additional ‘effort’? Or are they funding several slightly smaller projects, rather than your very big one, to avoid putting all their eggs in one basket?
this was for a program where they were choosing about 3 of a large original set, we know we made the final group — definitely whatever we were up against deserved to be funded. However, these are for what NSF calls “expeditions in computing” which are expected to be big and bold visions, and it is a multiyear program, so that even if they decided not to this year, there could have been a mechanism for the future — my point isn’t that this particular proposal must have been funded, but that NSF has no mechanisms (at least in CISE) for any sort of additional effort. If a very very close miss on a “visionary” proposal with a “dream team” cannot be somehow turned into a funding possibility, then with its current rules NSF will continue to miss real opportunities for higher risk research (something they’ve been accused of in the past).
In item 3, you stated, “There is a lot of anecdotal, and some formal, evidence that panels tend to be more conservative.” Michael Nielsen had an interesting blog post about research that finds collective decision making scenarios (e.g., panels) lead to poor choices.
Cass Sunstein’s book “Infotopia” is an eye-opening account of situations where group decision-making fails. In particular, he makes a strong case that group decision-making performs poorly when decisions are being made are about questions whose answer is relatively subjective, and where the group behaviour is relatively unstructured. So, for example, if the question is “Is the square root of 2 irrational”, a group may perform well, because everyone can recognize progress on the question. But if the question is much more subjective (“Should this grant be funded”), groups don’t work so well, at least, not without a very structured process. There does seem to be one big gain from group decision-making in this situation: the group members end up relatively happy with the decisions. That is, the gain is political, not in improved quality of the decisions.
Oh, from my junior level, do I ever feel your pain. But your example seems even more flagrant than my anecdotal experience – it’s very hard to be original, if you haven’t got the bureaucracy covered as well.
“There would be no advantage to having made it so far in the first round. Most funding agencies have a way of giving some sort of extra consideration in these sorts of cases, NSF doesn’t. "
Neither does the funding mechanism in the E.U.‘s FP7 to my knowledge. Like you, we were wonderful in every way, but too fragmented among too many sites – and we totally could have and will, next round, tightened things up. Probably the other groups who made it past cutoff were wonderful, too, but in these sorts of situations you can only see what didn’t make it, and lament the alternative universe in which they might have been funded…
I agree that for such large stakes, confronting the POVs of external reviewers (there are disadvantages to that as well) with panels could be ideal, to get past the political side of things, and perhaps enable certain members of a panel to not be quite so consensual so as to fund differently deserving projects.
Chris thanks for the pointer to Michael’s blog – Michael, thanks for the pointer to the book – I knew about the book, but haven’t read it yet – I didn’t know of that specific result, but I think it is a good example of the sort of things I’ve seen in the past. There’s a number of other studies he cites which I did know about, which tend to show the same result. If this goes further I will definitely do my homework and a quote from that book sounds like a good way to help
Heather – I agree that the EU also doesn’t have such a mechanism, I’ve been on some EU panels. It definitely is the case that there will always be cases where something that was close didn’t get funded – this certainly isn’t my first time for that. However, the backstory on this is that there have been questions raised in the past as to whether NSF funds enough high-risk work – and I think that in this case the key is not so much the quality of the researchers (although if you knew who they were I’d bet you’d say “I can’t believe they rejected that!”) but the panel itself saying that this could be extremely high impact work and that we had the capability to deliver. If you compare this with what they funded last year (and of course I won’t know this year’s for a while yet) it is clear that they have played it pretty safe – and I understand the pressures that lead to that – but if we as scientists settle for this, we cede our role as innovators to industry – and I think that would be a big mistake!
James – The first half of the book is largely about the limitations of group decision-making. Unfortunately, there’s many questions Sunstein doesn’t address, but within the parameters he operates it’s a pretty useful resource. I also found it very readable – I read the whole book sitting in a Borders in Boston, before buying the book in deference to the author’s ability to keep me in my seat.
This is an excellent thread, and for me (sadly) emphasizes the inherent conservatism that rises in “organizations” during tough times, at exactly the time when they must to embrace radical change. This is the sad but eternal truth corporations, which unfortunately has spread to NSF funding.
Long before Clayton Christensen’s classic Innovator’s Dilemma, I heard a quote attributed to the great Gordon Bell: “…committees are the white blood cells of corporations…” (I cannot find the source, but it wasn’t from his book on high tech ventures). His specific context was the challenge of “intrapreneuring” within corporations; in companies like Digital (where I was at the time) moving ideas forward required (and requires!) an arduous journey through a series of committees whose only purpose seems to be to kill exciting but risky proposals by picking at them until they are dead (in spirit if not in reality).
I think this is analogous to the “local optimum” problem that Michael so wonderfully discusses in his blog depicted here. The problem is getting the committee to stop clinging to a local optimum representing its current paradigm; it is impossible for a group, thinking as a group, to discard its old paradigm and leap across the “chasm” (cf. Crossing the Chasm, Geoffrey Moore) to any new, “radical” paradigm embodied by “risky” proposals
-even if the sponsors are well-known!“Playing it safe” and staying on that first optimum kills off the good opportunities with the bad. Or put another way, if you are drowning and someone throws you a lifeline, you don’t ask them for their ship’s menu, or their CV as a rescuer, or the repair records of their boat! No, you accept the risk that the food might stink, or the rescuer is a rookie, or even that her rescue boat might turn out to have a leak!
Similarly, funding organizations seeking to establish new research tracks must accept risks, especially when the proposals are backed by PIs with proven records of success!
Thanks John – Really good points. I think there’s an interesting connection between this point and the one Michael made before – as the US has moved more and more to panel review, the it’s gotten harder for them to fund new stuff – I think that follows from the point Michael made – in an established field the methodologies and measurables are well understood – so the panel can evaluate the risk – but when the work is new, and part of the work will require creating the methodologies, we are in the subjective space where panels are known to fail. There are some EU programs where the panels pick some outside reviewers – again, the mix makes sense to get some expertise into the equation (and, aggh, I just realized that my reviews are due soon – so gotta go!)
-Jim H
I am sorry about your grant and the hope that is set aside for another day. I found this an enlightening post in regards to what will feel threatening to a review panel and why ordinary gets a popular vote. I think the key is to see this as a system rather than an incident and that a system can be built to counter intellectual and security based resistance. It seems to be about making a safe choice rather than a significant contribution…in other words if the panel can not envision its external control it must have organizational flaws. I guess this is expected as new forces demand space and create resistance…so rather than changing the NSF which is an unlikely task how would you design the next project to circumvent the difficulty….just a beginner here who is considering how to frame a grant but it has occurred to me people could go through years of producing quite acceptable proposals, knowledge that produces results and still be a near miss
Best
Amy
This post is discussed in the latest Coast to Coast Bio Podcast – Episode 22 commencing at 26.00 minutes.
Graham – thanks for letting me know c2cbio is having server issues (at least I cannot seem to connect) – look forward to hearing it some time – JH
managed to hear it today, don’t agree with all they said, but anyone saying such nice things about me deserves a mention :-)
Did anyone see this? It appears the external reviewer issue may be widespread
http://www.nature.com/news/2009/090722/full/460440a.html