Publications
- Overview
- Numbers of publications arising from MRC research
- Citation impact of MRC research publications 2006-2008 rebased to end of 2009
- What does this tell us?
- MRC attributed publications have a higher citation impact and fewer papers are uncited, than output from the rest of UK medical research
- F1000 evaluation of MRC publication output
- F1000 evaluations are an indicator of future citation impact
- What is the paper that currently has the highest citation impact?
- Links to information on other papers with high citation impact
- Next Steps for the analysis of publications
Overview
Peer reviewed journal articles and reviews are an important primary output from research, recording new knowledge gained, new methods established, or new insights from a synthesis of existing work. There is a large amount of interest in measuring the extent to which this knowledge is used. Most approaches are based on capturing the way subsequent publications cite this knowledge.
Using MRC e-Val, we have for the first time assembled a comprehensive bibliography of publications reported as arising from MRC research. This data improves upon previous approaches to assemble this data from final grant reports and annual returns from MRC Units and Institutes. We have explored a number of approaches to analyse this data and found this useful for identifying research that is recognised by peers as having, or likely to have, wider impact.
Using bibliometric analysis, papers that were reported as arising from MRC funded research were shown to have significantly higher “citation impact” than UK health related, or biological sciences research. Papers with a contribution from MRC funded research were shown to have over two times the world average citation impact.
We will continue to collect this data, and examine the citation impact of the MRC portfolio over time. However it is our aim that, in future all research publications arising from MRC research should be available in an open access repository (such as UK PubMed Central), with a complete record of contributing funding agencies, appropriately acknowledged.
Further information
- Numbers of publications arising from MRC research
- Citation impact of MRC papers
- F1000 evaluation of publication output
- Examples of papers with high citation impact
- Download the 2009 analysis of MRC publication output
Numbers of publications arising from MRC research
16,041 unique publications, published between 2006 and 2009 were reported via MRC e-Val.
MRC e-Val includes a link to the PubMed website, through which researchers can quickly and easily enter details of their publications. Pubmed (service provided by the US National Library of Medicine and comprises more than 19 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites) is an extensive bibliographic database of the abstracts of published biomedical research papers and includes the majority of abstracted articles and reviews relevant to medical research.
Researchers are asked to record publications in MRC e-Val that, in their view, resulted wholly, or in part, from MRC support. Publications are reported by entering an 8 digit PubMed Identifier (PMID), this triggers an automated fetch of the information from PubMed, and the researcher is then asked to confirm that this is the publication they intended to enter. This meant that researchers who knew their PMID did not have to spend time typing in title and author details for each publication. For journals not abstracted by PubMed, researchers were encouraged to provide brief bibliographic details, in order that these outputs are recorded.
After the MRC e-Val data gathering period, bibliographic details of unique papers were provided to Evidence Ltd who returned citation impact information for every publication they could match to the Thompson ISI Web of Science database. While the ISI database does not include all journals in which MRC researchers publish, citation data was returned for around 90% of all papers reported via MRC e-Val.
The figure below shows the number of unique publications captured by MRC e-Val by year, and by whether we obtained citation impact data from Evidence Ltd, or whether the publication could not be matched to the Thompson ISI database.
Year |
No citation data |
Citation data |
Total |
2006 |
311 |
2957 |
3268 |
2007 |
398 |
3704 |
4102 |
2008 |
384 |
4108 |
4492 |
2009 |
343 |
3836 |
4179* |
Total |
1436 |
14605 |
16041 |
* The data gathering period ended in December 2009. Due to the time it takes for records to be updated in PubMed the data for 2009 will not be complete. We expect the number of publications for 2009 to rise when we next collect data from researchers. For analysis, data on papers published in the three years between 2006 and 2008 were used.
Between 2006 and 2008, MRC funding resulted in 11,862 unique publications. Of these, over 10,000 journal articles and reviews have been matched by Evidence Ltd to records in the Thompson ISI database and citation data has been captured for these papers.
Citation impact of MRC research publications 2006-2008 rebased to end of 2009
Based on the number of times an article has been cited compared to the number of times we would expect the article to be cited given the journal and year in which it was published, a Rebased Impact (RBI) is calculated for each publication. For example, an RBI of 1 means the article has been cited exactly the number of times expected (or as world average); an article with RBI of more than 4 is generally considered to be a highly-cited article (four times higher than the world average).
An Impact Profile™ (Adams J, Gurney K & Marshall S (2007) Profiling citation impact: A new methodology. Scientometrics 72:325-344) is a method of presenting the citation impact of a publication set. It shows the distribution of a publication set according to the RBI of the publications. As with other output data, this information is most useful when compared to a relevant benchmark set of data. The Impact Profile™ below presents these 2006-2008 publications according to their citation impact at the end of 2009 compared to the citation impact of all UK clinical/health related papers (minus MRC output). The graph of citation impact from UK clinical/health related papers includes data from 96,977 articles and reviews (criteria used to distinguish between reviews and articles in the Thompson ISI database includes).
Data and Analysis: Evidence Ltd, Thomson Reuters.
What does this tell us?
This is the first time that the MRC has been able to compile a near complete record of publication output from MRC research, across a number of years. While it is encouraging to see that MRC research is contributing to large numbers of research papers, measures of quality that can be used to benchmark this output are essential.
MRC attributed publications have a higher citation impact and fewer papers are uncited, than output from the rest of UK medical research
About half the world output of papers and a quarter of UK papers are never cited. Data from the MRC e-Val dataset shows that around 6% of MRC papers published between 2006 and 2008 fall into this category. This is significantly lower than the figure for the UK Clinical/Health and Medically-related research portfolio (17%).
The analysis of MRC e-Val data shows that 13% of MRC papers are cited over 4 times the world average, and this compares favourably to 6% of papers in this category for the UK Clinical/Health and Medically-related research portfolio.
Overall publications attributed to MRC funded projects have an average citation impact of more than twice the world average (2.35).
Together these findings are extremely encouraging, they show that the results from MRC research are highly skewed toward higher impact publications, and that MRC papers are less likely to remain uncited 1-3 years after publication.
Although it was made clear that information gathered via MRC e-Val was for the purpose of evaluating productivity and quality of research output across the entire MRC portfolio, we were concerned to check to see if there was any selective reporting of publications by researchers. It is possible for researchers to omit papers if it is known that they had failed to accumulate many citations, and this would help to skew the results toward a higher citation impact. In future we will be collecting this information each year, in advance of citations accumulating, and so there will be no opportunity to select publications on prior knowledge of their citations. We compared publication lists provided from MRC institutes between 2003 and 2008 with data compiled by Evidence Ltd based on address searches from the Thompson ISI database. In this specific case there was close agreement between the datasets, and so no evidence of under-reporting.
We suggest that this indicates excellent research output from MRC funded research, in particular the contribution that MRC research makes to the publicly available body of knowledge. We are interested in following changes in publication output, the citation impact of this over time, and as we accumulate more information analysing this by subject area.
F1000 evaluation of MRC publication output
Faculty of 1000 (F1000) Biology and F1000 Medicine are online tools which aim to highlight and review the most interesting papers published in the biological sciences and medicine, based on the recommendations of a faculty of well over 4,500 selected leading researchers ("Faculty Members"). While F1000 biology has been running for longer, F1000 Medicine was launched in 2006, and so both tools have gathered information on papers relevant to the period covered by MRC e-Val.
It is argued that not all papers that are of wider interest are published in a high impact journal. It is also argued that not all papers of wider interest will necessarily accumulate high citation counts. F1000 rates individual papers according to their merit, irrespective of where they are published. More than a third of papers awarded the top "Exceptional" rating by members of Faculty of 1000 Biology are from journals other than Nature, Science and Cell. Conversely, between 20% and 25% of eligible papers from these three journals are not recommended by members of Faculty of 1000 Biology.
Faculty members select and evaluate noteworthy papers soon after they are published. As citations arise through the publication of subsequent papers, and citation impact is calculated on an annual basis, there is a long lag between publication and recognition that a paper has a high citation impact. F1000 evaluations may therefore give an earlier indication of high quality papers.
While both citation impact and F1000 evaluations are subjective, it is worth noting that the F1000 score is based on a positive recommendation by named, hand-picked scientists who are not the authors. This mitigates against the problems of self citation and negative citations. Approximately 2% of research papers are rated by faculty members, and we were interested in the qualities of these high profile papers and the use of the F1000 evaluation as a measure distinct from citation impact.
Faculty members can apply a number of labels to the papers they rate (such as “controversial”, or “changes clinical practice”) and we were interested in whether this information could assist us in identifying papers which highlighted work providing wider outputs and outcomes, and whether this matched with information provided in other sections of MRC e-Val.
We commissioned F1000 to match papers that were reported via MRC e-Val and published between 2006 and 2008 to papers with F1000 evaluations. This was achieved using the unique PubMed identifier (PMID), as F1000 evaluations are linked to bibliographic information in PubMed. F1000 identified 1281 unique MRC e-Val publications from their database (approx 6% of publications in MRC e-Val), although only 288 (22%) of these had attracted more than one independent review from a F1000 member. We examined the citation profile of these 1281 papers compared with the overall population of MRC attributed papers, UK biological sciences papers and clinical papers.
F1000 evaluations are an indicator of future citation impact
It is clear that those papers chosen for evaluation by faculty members do subsequently accumulate a high citation impact. This suggests that those papers selected by F1000 faculty may well be those to watch in future, and we should be interested in MRC attributed papers which gain a high evaluation score, and attract a number of independent evaluations from F1000.
Our analysis of the labels applied by F1000 faculty was less helpful. F1000 have not set out clear objective criteria for the use of these categories, and they are therefore used inconsistently by faculty members. The categorisation of papers in a consistent way while evaluating them, is a good opportunity to add value to this information, and we would like to continue to work with F1000 to select criteria that will be helpful to the research community in finding literature likely to make a difference to their field.
F1000 citation |
Average F1000 rating (number of evaluations) |
More information |
Faculty of 1000 Biology: evaluations for Rasmussen SG et al Nature 2007 Nov 15 450 (7168) :383-7 |
11.3 (8) |
Breakthrough reveals important human receptor structure that will aid drug development |
Faculty of 1000 Medicine: evaluations for The ASTRAL Investigators ATRAL investigators N Engl J Med 2009 Nov 12 361 :1953-1962 |
8.8 (6) |
Treatment to open blocked kidney arteries shows no real benefit to patients |
Faculty of 1000 Medicine: evaluations for Dennis M et al Lancet 2009 Jun 6 373 :1958-65 |
8.2 (3) |
What is the paper that currently has the highest citation impact?
Genome-wide association study of 14,000 cases of seven common diseases and 3,000 shared controls (Wellcome Trust Case Control Consortium, Nature 2007, RBI 151.3)
This important study was primarily funded by the Wellcome Trust. The Wellcome Trust Case Control Consortium involves over 50 research centres, and the paper acknowledged hundreds of authors, a number of which reported their involvement in this work via MRC e-Val. The MRC provided significant funding for the disease cohorts used by this study, funded the 1958 birth cohort which contributed half the controls, and substantially funded the UK DNA banking network which stores the samples used. A number of key contributors to the consortium also received personal support from the MRC. Given the size of this programme, and wide ranging impact of this work it is not surprising that this paper gained the highest citation impact of all publications collected via MRC e-Val. Research using large genome-wide association studies feature strongly in the list of papers with the highest citation impact.
Links to information on other papers with high citation impact
Paper |
RBI (2009) |
More information |
Link to MRC Research |
TDP-43 mutations in familial and sporadic amyotrophic lateral sclerosis (Department of Clinical Neuroscience, King's College London, Medical Research Council (MRC) Centre for Neurodegeneration Research, and Institute of Psychiatry, London, Science 2008, |
51.9 |
MRC funded research contributes to crucial landmark in identifying the cause of Lou Gehrig’s disease |
|
The selectivity of protein kinase inhibitors: a further update (MRC Protein Phosphorylation Unit, Dundee, Biochemical Journal 2007 |
49.9 |
MRC Researchers publish key data on kinase inhibitors to help drive basic research and drug discovery |
|
A common variant in the FTO gene is associated with body mass index and predisposes to childhood and adult obesity (Contributions from the MRC Functional Genomics Unit, and MRC Centre for Obesity and Related Metabolic Diseases, Science 2007) |
60.3 |
Next Steps for the analysis of publications
It is important that MRC can compare the productivity, progress, and strengths of its research portfolio against other organisations, and over time. Currently publication output is one of the few areas where this is practical.
Publication data is routinely collected and analysed in generic reports such as International Comparative Performance of the UK Research Base, which is commissioned by BIS every year (add link). The MRC Evaluation Team is working with the Wellcome Trust, and other funding agencies to be able to disaggregate this data to show the contributions of individual funding agencies.
By working jointly with other funding agencies it is hoped that we will develop the power to demonstrate associations between changes in productivity, progress and quality of output and strategic initiatives.
We will continue to accumulate information on the publication output from MRC-funded research, calculate the citation impact of this, and work with other approaches such as F1000 to see if better indicators can be constructed.
We may publish follow up analyses of publication output by subject field, although this will be more informative if we do this collaboratively with other funding agencies.
Ideally we will in future collate publication output from publicly available repositories such as PubMed and UK PubMed Central (UK PMC) without the need for researchers to separately report this information. To ensure that this can occur, all researchers funded by MRC will need to note the contribution the MRC has made to their publications according to accepted guidance (Research Information Network (RIN) guidelines on the acknowledgment of research funders). Acknowledgments that include the name of the funding body and a unique identifier for the relevant grant (in the case of MRC this should be the “file reference” number) can be automatically identified by PubMed. As all MRC funded researchers must comply with the MRC open access mandate all MRC attributed output should shortly become available in UK PMC. When we can demonstrate consistent acknowledgement of MRC funding in papers we will be able to cease collecting publication output via MRC e-Val. At present less than a third of publications captured by MRC e-Val include a formal acknowledgement of MRC funding that can be automatically extracted by PubMed, and a third of papers captured by MRC e-Val are present in full in UK PMC.