Tomorrow's Giants: a conference hosted by the Royal Society and Nature forum: topic
This is a public forum
Performance indicators for policy
Rachel Oliver
Friday, 18 September 2009 07:55 UTC
Having attend an interesting “Tomorrow’s Giants” meeting in Cambridge yesterday, a question came to me which has occured to me before at similar science policy events: it’s all very well dreaming up exciting new policies for funding or organising science, but how do we know whether we’ve done a good job?
As scientists, we’re used to testing our ideas. I work on semiconductor devices, but I don’t get to launch a new device on the unsuspecting public, without thoroughly testing it and proving its usefulness. Similarly, in planning science and applying for grants, I’m expected to identify objectives and then use milestones and performance indicators to demonstrate that I am reaching those objectives. However, new ideas in science policy don’t seem to get tested at all. If an organisation introduces a new initiative for funding science, it presumably does so with some objectives? How does it test whether those objectives have been met? If, for example, the objective is to increase publication rates from British Universities, how do we know that any observed increase is actually due to the initiative and wouldn’t have happened anyway? Without a control group of some description it’s difficult to test even a fairly concrete objective, so how on earth would one assess success on more nebulous objectives like “supporting excellence”? Even worse, if we have multiple objectives, how do we check that in forging our way towards one, we’re not going to damage our chances of reaching another?
I think questions like this are “the elephant in the room” when discussing science policy: the big problem which we all ignore. We can come up with no end of clever ideas for transforming the future of science, but without a way of checking whether we’re meeting our objectives, we’ll never know whether we’re doing more harm than good. What can we do about this? Can we identify meaningful performance indicators for policy?
-
Replies
-
The call for a proof of usefulness of new policies is too frequently wrongly used as an argument against a change. Existing policies have never shown to be better than new ones either.
Any statistic can only reveal a correlation, but not a causal connection.
However, there is a clear demand for changing policies that are obviously not meeting the objectives. A way forward can be constructed by finding out why policies fail, and how chances will affect the outcomes.
In fact, I would consider it obvious that publication rates or any form of bibliometrics are not only a poor indicator of research excellence (shouldn’t quality come before quantity?), but aiming at such an objective is even detrimental to foster curiosity and innovation.
Defining meaningful objectives, would, in my opinion, need to go before defining meaningful indicators on whether these are achieved.
-
I think that there are ways that future, current and past policies get tested and reviewed. First by academic researchers in the social sciences, secondly by small organisations/think tanks/learned societies and other interested groups, thirdly government/policy makers who commission reviews and undertake researche to see how current policies are working and fourthly by researchers’ themselves who are asked to respond to consultations on existing and future policies. Also there is a lot of horizon scanning done by a number of organisation to figure out how to improve existing policy and what other ones will be required in the future.
I am not just saying this because I am involved in policy (I left bench research last year); however, it does give me the unique position to see how the whole thing fits together and where the checks and balances are. I am not saying it is perfect, but they are there. The work we do at the Research Information Network specifically looks at enhancing and broadening the understanding of how researchers create and use information resources and services and how this is changing in response to technology and policy developments. Ultimately we aim to ensure that current and future policies not only reflect the technological changes that are occurring, but also reflect the behaviours, needs and expectations of researchers. Our latest report looks at how researchers publish and why, and looks at the influence of research assessment on this process.
Yes more can be done – it isn’t perfect, but there are groups out their trying and many (such as ourselves) want researchers to be involved and are actively trying to get researchers into the discussions, because they are happening, just maybe not with all the people that should be involved. Also I think a problem lies in fact that these discussions, on the checks and balances of current and future policy, just aren’t high profile enough.
-
HEFCE is launching a consultation on the REF, the new process for assessing research in UK higher education institutions, which will replace the RAE.
For more information click here
-
Although an easy one, just the number of publications does not reveal much about the qualities of a researcher, in which case scientists working in industrial research would be poor scientists – however, they keep the economy going which enables us to do our research.
Measuring performance of policies enabling science and measuring the performance/outcome of research projects should have ultimately the same goal:
They should aim to improve the average quality of life (if this is the most important variable) in our society. In a sense we are employed by and should be accountable to the tax payer. The questions are:1. Is it possible to measure the quality of life?
2. Can we measure how a new policy/research project alters the quality of life?
3. How can we estimate the future impact of the new policy/research projects on the quality of life.On a smaller scale, that’s what companies do (or attempt to do) to justify funding some research projects and not others. The equivalent answers to the above questions would
be:
1. Profit
2. Change of profit
3. Contribution of an invention to the overall value of a product, ie market share and profit margin of the product (feeding back to 1. and 2.).Is it possible to capture the value of science to society in a similar way, incorporating aspects like GDP per capita, health (via life expectancy?), social tension (via number of prisoners per capita?) etc?
-
Thanks everyone for your very interesting thoughts.
I’m not sure I can adress them all in one post, but a couple of comments which occur to me are:
- I agree with Martin Domink that the first issue is to define meaningful objectives and that improvements in bibliometrics aren’t really the right objectives. However, objectives such as “supporting excellence” seem even more unhelpful. So, what objectives would be sensible and how could they be measured?
- I’m interested in Branwen Hide’s comments about policies being researched and reviewed. I’m just not sure to what extent it is true at a practical level. I’m not necessarily talking about government, or even the civil service, but about those organisations which are notionally independent of government, such as the research councils, whose policies and strategies often have an enormous day-to-day (and long term) influence. I’m not sure where I’d look for information on the evidence base for research council strategy changes. Can anyone tell me?
All the best,
Rachel.
-
