Overview
- What is MRC e-Val?
- Who responded to MRC e-Val?
- How many outputs were reported to MRC e-Val?
- Understanding and interpreting the data
What is MRC e-Val?
MRC e-Val is an online survey designed to gather feedback from MRC-funded researchers about the results of their work. The aim is to compile accurate information about the outputs of MRC research and to capture impact as it occurs. This information will then be used to communicate the benefits of MRC funding; support evaluations of the economic, social and academic impact of MRC research; and provide evidence for strategy development.
The MRC’s Strategic Plan covers the period 2009-2014 and defines four strategic aims. MRC e-Val collects data that will monitor and evidence our progress toward these strategic aims.
The implementation of MRC e-Val brought several important developments to the MRC process for gathering output data. Previously some data had been collected using final grant report forms, but this provided an incomplete snapshot of output. MRC e-Val will seek feedback from researchers throughout the lifetime of their MRC support, and for a number of years thereafter. This recognises that outputs may accumulate after grant funding has terminated. Also MRC e-Val is addressed to all MRC grant holders, as well as all MRC fellowship holders and all MRC Unit/Institute programme leaders. MRC e-Val will be used to gain a comprehensive picture of research progress, productivity and quality across the entire MRC portfolio.
Data was captured through an easy-to-use on-line form. In order to reduce the burden on the principal investigator, care was taken to seek data that we knew was needed for evaluation, and to keep detail to a minimum. Many questions could be answered using “drop-down” and “pick-list” menus.
The full MRC e-Val question set is available here. The data in MRC e-Val adheres to a set of principles of use. These principles outline how we collect, store and analyse the data.
If you have any queries about MRC e-Val, please contact the evaluation team:
Email: evaluation@headoffice.mrc.ac.uk
Who responded to MRC e-Val?
Information was sought across the whole MRC portfolio from all MRC researchers that had held MRC support since 2006 (approximately 3000 principal investigators). A full set of responses was submitted for 2541 Awards. This represents 83% of Awards that were invited to complete MRC e-Val.
The table below compares the total MRC Science Spend to the Spend for awards that submitted data to MRC e-Val (for financial years 2006-07, 2007-08 and 2008-09). It shows that we have gathered data which covers 83.6% of our Science Spend.
Type |
£m |
MRC Science Gross Spend (£m) |
1412.5 |
Spend for awards with Submitted data (£m) |
1180.2 |
% coverage in MRC e-Val |
83.6 |
Notes
- Spend data for MRC Institutes and Units, down to the level of individual research programmes is estimated retrospectively each year.
- The data above is drawn from MRC Research Data Warehouse (RDW), the majority of which is displayed publicly in the MRC online Research Portfolio.
- The ‘science spend’ above does not include contribution to special research programmes (including strategic partnerships with other research funding agencies, such as UK Biobank, the MRC contributed £37 million to these arrangements in 2008/09), or international subscriptions (£15 million). Where feasible it is planned to include these investments in the MRC e-Val survey nest year. MRC e-Val did not gather feedback on output realised as a result of supporting studentships (£68 million in 2008/09), as HEIs are required to provide this via the RCUK JeS portal.
How many outputs were reported to MRC e-Val?
Over 44,000 outputs were reported via MRC e-Val. Nearly half the outputs (21,823) were publications. This is shown in the table below:
MRC e-Val section |
Total (sum of all awards) |
Publications |
21,823 |
Collaborations |
3,958 |
Further Funding |
4,038 |
Next destination |
3,151 |
Dissemination |
3,982 |
Influence on Policy |
928 |
Research Materials |
1,926 |
Intellectual Property |
358 |
Products and Interventions |
303 |
Spin-out Companies |
30 |
Awards and Recognition |
2,243 |
Significant Knowledge |
1,682 |
Total (all sections) |
44,422 |
In addition researchers provided feedback on a question in section 4 asking for details of whether they had difficulty in recruiting particular skills. We received 549 responses against this question.
It should be noted that this table presents the sum of outputs reported. Some outputs may be reported multiple times, having been entered independently by more than one researcher. In the case of publications it is relatively easy to identify duplicate outputs, and we can show that the 21,823 reports of publications represent almost 19,000 unique papers. For other outputs this is more challenging, and this is addressed as part of the analysis for each section.
MRC e-Val provided the facility for researchers to report an output once, but note that contributions came from a number of Awards either funded by the MRC or not. We therefore have to ensure that some unique outputs should be included in more than one analysis, depending on which “portfolio” of input awards is used.
Data from 2541 Awards, covering £1,180 million of expenditure on research gives us a fairly comprehensive view of way research is delivering output across the MRC portfolio. This first analysis has highlighted further ways in which we can improve the survey, ensuring that the data collected is aligned to business need, and that the data collection process is kept as simple as possible for researchers.
We aim to continue to work with the research community to increase the feedback we receive. We are accountable for 100% of the public funds we deploy in support of medical research, and so will be looking for a significant increase in coverage in 2010/11.
Understanding and interpreting the data
We present here the initial findings for sections of MRC e-Val from all Awards that submitted data. More specific results and analysis will follow as we continue our analysis.
There are a number of challenges to tracking research over the long term, and the diagram below illustrates some of these. The diagram shows inputs, outputs, outcomes and impacts from research over time. It shows that there will be a time-lag between input and outcomes/impacts being realised. Some outputs have a greater time lag (between input and outcome/impact) than others.
The realisation of research outcomes is likely to involve contributions from multiple research projects (the blue arrows in the diagram above), which may be funded by different funding agencies. As research is translated it is more likely that research teams collaborate on, or “pass the baton” for, developing knowledge into new treatments to other research teams or industry, worldwide. MRC influence throughout this process will vary significantly as the work at various stages is conducted with MRC support, in collaboration, or entirely externally.
Translation is also not as simple as a linear process as illustrated above. The pathway to impact is likely to have multiple feedback loops. Progress, or failure in more applied stages, not only informs progress to application, but also informs fundamental research. It will be challenging to track the progress of research through these complex networks of inputs, stakeholders, and over the long term.
Our approach is to collect data relevant to a number of indicators of progress in the above process, and to as far as possible, link this to MRC inputs. Only over time will we be able to determine whether some indicators are better markers of progress than others, and the extent to which we can establish the MRC contribution to this.