Methodological research to improve the quality of national clinical audits




National clinical audits (NCAs) seek to measure and assess the quality of clinical practice across all practitioners and providers of care in the UK or, more usually, in one or more jurisdiction. They focus on a particular disease (e.g. stroke, diabetes, hip fracture) or procedure (e.g. cardiac surgery) and assess either the outcome of care (mortality, morbidity, disability, health related quality of life) or the extent of adherence to evidence-based guidelines or a combination of both. Key to their success is the engagement and ownership by the relevant clinicians, the involvement of methodologists (epidemiology, statistics, outcome measurement), the participation of patients and carers, and IT expertise. Traditionally NCAs have concentrated on secondary and tertiary care but the emphasis (particularly for long term conditions) is changing to primary care.

In the last ten years the number of NCAs has rapidly expanded throughout the UK. In England, 46 NCAs are included in the 2015-16 Quality Accounts (the annual reports that NHS trusts have to produce about the quality of the services they deliver); in Scotland audits cover topics such as stroke, trauma and renal replacement therapy.

In addition, NCAs are used throughout the UK for medical revalidation and for providing staff with information for stimulating redesign of their services and providing the public with information to assist their choice of providers. In England, NCAs are also used nationally for regulation by the Care Quality Commission, by NHS Choices for reporting consultants' outcomes, by the Department of Health to assess NHS England via the NHS Outcomes Framework, and by Clinical Reference Groups for Clinical Dashboards.

NCAs face a number of methodological challenges, many of which are directly linked to their extensive scale and the limitations of the resources that are available for recruiting patients and collecting data. It is essential that these methodological challenges are addressed to ensure that NCAs can provide rigorous, meaningful and credible information about the performance of healthcare providers.

Highlight Notice

MRC and NIHR invite applications through the Methodology Research Programme to conduct research into methodologies to optimise national clinical audits, focussing on the following areas:

1. Benefits and costs of different methods to check case ascertainment and the quality (completeness, accuracy) of audit data.

The traditional approach to case ascertainment is to check the number of patients recruited against hospital administrative data or review a representative sample of case notes. Data quality checks have involved reviewing the plausibility and internal consistency of data values.

If case ascertainment or data quality differs across providers (almost inevitable), we need to know how these differences should be taken into account when comparing outcomes. These adjustments could have an impact on the uncertainty estimates of the results of the audit itself (e.g. produce a widening the confidence intervals) or lead to changes in the actual results.

Methodological research is needed to:

  • Determine the efficiency of different approaches to checking case ascertainment, data completeness and accuracy according to clinical area, nature, scale and scope of the audit, and the type of data used.
  • Develop statistical methods that can be used to incorporate data quality into the audit results through adjusting the results or inflating the variance estimates.

2. Linkage across audits to construct and assess pathways of care (meta-audit).

Patients may be included in more than one NCA. For example, a patient with bowel cancer who had a major resection may be included in audits focused on bowel cancer, emergency laparotomy and critical care. Linking the records of such a patient in three NCAs would allow assessment of their care pathway. Also, a patient may have more than one health problem and be included in several audits. Such linkage (or meta-audit) has the potential to assess service organisation and extent of integration of care.

NCAs can also benefit from linkage to other national databases, such as cancer registries, administrative databases (e.g. HES, PEDW, Scottish Morbidity Record) and primary care databases. Such linkage would not only allow an assessment of care pathways for a specific condition but would also help to identify other morbidities and healthcare issues that patients may have.

Methodological research is needed to:

  • Identify, develop and validate methodologies to enable one to identify clinical areas with potential for linkage of NCAs (and with other databases) and assess the feasibility and cost-benefit of such linkages

3. Using multiple audits to look at a provider’s overall performance.

NCAs are mostly used in isolation to compare the performance of providers for specific clinical activities. Combining data from audits, either grouped according to conditions (e.g. cancer, cardiac disease) or type of service (e.g. surgical procedure), might provide meaningful and useful indicators of a provider’s overall performance.

One possible methodological approach could be based on the assumption that the outputs of audits all contribute to the measurement of “quality” construed as a single latent trait of a provider. This would mean that regression approaches can be applied to model this latent variable. Other approaches could involve the use of non-parametric approaches, for example creating clusters of providers that have similar patterns of audit results.

Methodological research is needed to:

  • Investigate the extent to which the results of NCAs are correlated within providers and whether any association depends on clinical area or type of services.
  • Explore the feasibility and value of different statistical approaches to distinguish providers with different levels of performance.

Application process and schedule

Applications for projects are invited through the normal MRC funding grant schemes (research grant, new investigator research grant etc.) and will be considered at the regular MRP Panel meetings. These will be in competition with other applications received, but the Panel will be mindful of the strategic importance of this area.

MRC-NIHR Methodology Research Programme

Contact and guidance

The titles of all applications in response to this highlight should be prefixed with HNA: when filling out the JES form, and on any attachments, e.g. “HNA: A method for…”

It is essential to discuss your proposals with MRC Head Office at an early stage. All applications must be approved by the Methodology Programme Manager prior to submission. Please contact:

Dr Sam Rowley