Abstract

Purpose

The purposes were to find and synthesize available literature on explicit or implicit standards for the design and conduct of a national activity that involves measuring and facilitating improvement of the quality of patient care, such as a national clinical audit or a quality improvement (QI) study, and to develop proposed standards for the design and conduct of the national activity.

Data sources, selection and analysis

The literature was searched to identify key aspects of good practice in the conduct of national or international clinical audits, QI studies, performance or quality indicator measurements or equivalent national initiatives that have the purpose of driving improvement in the quality of care provided in a healthcare system. Key aspects of good practice in design or operation of these activities were abstracted from the literature, and organized logically into standard statements according to the stages in the design or conduct of such an activity.

Results

Thirty standards for the design and conduct of a national clinical audit or QI study were derived from the published literature. The standards are on structural, process and outcome aspects of any national activity that involves measuring and improving healthcare services. Most of the standards focus on measurement processes.

Conclusion

It is hoped that these proposed standards for a national clinical audit or QI study will facilitate debate on how to assure the quality of these national activities. Activities that meet accepted standards may be more effective in influencing participating sites to achieve improvements in the quality of care.

Introduction

Clinical audit has been defined as a quality improvement (QI) process intended to measure and improve the care and outcomes patients experience [1]. Clinical audits are normally carried out by a group of peers working in the same healthcare service or organization. National groups in the UK have carried out national clinical audits since 1994 [2]. In England in 2012, there were as many as 70 national clinical audits being conducted; National Health Service (NHS) organizations are mandated to participate in 29 of these audits [3]. In other countries in which clinical audit does not have a high profile, equivalent activities, such as international and national quality indicator projects, QI studies, performance measurement or monitoring projects or registries, have been carried out for several years.

Some individual countries have set out guidance for the design of registries [4, 5]. The SQUIRE (Standards of Quality Improvement Reporting Excellence) guidelines [6] were developed in recognition of the variation in content and quality of QI work in health care. The guidelines describe what should be included in published reports on QI studies, and good practice for carrying out a QI project is embedded in the guidelines.

However, guidelines or standards against which national clinical audits or QI studies can be assessed and assured have not been available internationally. In the absence of agreement on best practice for these activities, variation in their potential effectiveness and cost–benefit is inevitable. For example, if staff in participating sites are unable to reliably collect the data required or do not receive timely reports comparing their performance with equivalent sites, they may be less likely to make changes in their clinical practice based on the findings.

Purpose

The purposes of this work were to find and synthesize available literature on explicit or implicit standards for the design and conduct of a national activity that involves measuring and facilitating the improvement of the quality of patient care, such as a national clinical audit or a QI study, and to develop proposed standards for the design and conduct of such a national activity.

Method

Search strategy

For purposes of identifying publications describing good practice relating to the design and conduct of a national clinical audit or QI study, any activity that involves the measurement of an aspect of the quality of patient care, including clinical processes or outcomes, was included. The types of activities involved included national clinical audits, national outcome studies, national performance indicator monitoring, national or international quality indicators measurement, national QI studies and registries.

Data sources and selection

The following sources were used: Medline and CINAHL databases using PubMed, Ovid and EBSCOHost as search engines; websites of organizations known to carry out or report on national clinical audits or QI studies; journals known to publish national or international studies related to measuring and improving the quality; professional organizations known to be concerned with measuring and improving the quality of healthcare services; and Google and Yahoo search engines.

In combination with the range of activities specified, the terms used for searches included: criteria or standards or indicators or measures with clinical audit or QI or outcomes; performance or quality indicators or measures; scope of work or statement of work and national quality indicators and registry. Where limits could be set, the years searched were limited to 2000 to 2011. Reference lists in publications retrieved or on websites searched were scanned for additional relevant sources.

Limitations of the search and the literature

The literature on the conduct of national studies of the quality of care is not uniform or systematic in the description of methodology used in carrying out the work. Much of the published material about how to design and carry out a clinical audit or QI study is written for clinicians carrying out their own projects in individual clinical services in single healthcare organizations. This material was scanned and included only if a point made was thought by the reviewers to be relevant to the conduct of a national project. Descriptions of national clinical audits or QI studies or performance indicator projects were not included unless they included details relating to quality of the performance of the project.

Data extraction, analysis and synthesis

Documents that described a performance expectation on any aspect of the design or operation of a national audit or equivalent activity were identified and screened. One reviewer screened documents retrieved and abstracted information from relevant documents and a second reviewer validated the screening of documents and abstracts. Key themes among the performance expectations were identified. Standard statements were derived from references that related to each theme. The themes and standard statements were arranged in a logical sequence in accordance with the stages in designing and conducting a national project.

Results: proposed standards for a national clinical audit

A summary of proposed standards for a national clinical audit or equivalent activity derived from available literature is in Table 1. The standards cover structure, process and outcome aspects of such a project.

Table 1

Standards for a national clinical audit or equivalent activity

No. Topic Standard 
Structural aspects 
 1 Ethical basis The subject of a national clinical audit has an ethically based justification for the conduct of the audit as a means of achieving improvement in the quality of care provided to patients 
 2 Governance There is continuous and effective governance of a national clinical audit by a group that assumes accountability for the effectiveness of the audit 
 3 Stakeholder involvement All key stakeholders in a national clinical audit are identified and actively involved in all aspects of the design and conduct of the audit. Key stakeholders include patients or service users or their representatives 
 4 Resources Resources allocated to a national clinical audit at a national level are appropriate for the valid and reliable measurement and proactive facilitation of improvement of the quality of patient care for the subject of the audit 
 5 Roles and responsibilities and project plan Roles and responsibilities for the design and conduct of the national clinical audit at national and participating site levels are comprehensively described and there is a published project plan for the execution of all aspects of the audit 
Process aspects 
 6 Recruitment The national clinical audit is continuously proactive in recruiting the participation of all sites intended for inclusion 
 7 Improvement-driven aims and objectives The stated aims and objectives of a national clinical audit reflect an intention to drive improvements in the quality of patient care 
 8 Population or sample The population or sample selected for a national clinical audit is suitable for drawing valid conclusions in relation to the stated objectives of the audit and for minimizing bias in the findings 
 9 Data collection strategy The strategy selected for collecting data for a national clinical audit is suitable for the stated objectives of the audit 
 10 Quality-of-care measures Quality-of-care measures used in a national clinical audit meet the criteria for such measures that are established by the governance board and are capable of producing valid and reliable data about the quality of care 
 11 Data elements and data sources Data specified for the collection for a national clinical audit permit judging compliance with the quality-of-care measures in accordance with the objectives of the audit, using currently available data to the extent possible to avoid an unreasonable data collection burden on participating sites 
 12 Data collection and handling protocol or manual A comprehensive protocol describes in detail all aspects of data collection and handling for a national clinical audit and is accessible by all stakeholders and all participants in the audit 
 13 Protection of patient identity Systems and processes are designed and fully implemented to protect the identity of patients whose care is assessed in a national clinical audit consistent with applicable national laws and regulations 
 14 Identification and handling of ethical issues Any ethics-related issues in a national clinical audit are recognized quickly and acted on appropriately 
 15 Training and support for data collectors Data collectors for a national clinical audit are trained to follow the data collection protocol and are supported to supply complete, accurate, reliable and timely data 
 16 Pilot testing The design, the data collection process, protocol and tools to be used in a national clinical audit are pilot tested before all intended sites are invited to participate and action is taken on the lessons learned from the pilot test 
 17 Reliability testing The reliability of the data collected for a national clinical audit is tested and the findings are included in reports 
 18 Data linkages If a national clinical audit relies on linkages with electronic records or databases that contain information about individual patients, systems are implemented to ensure the security of the linkages 
 19 Data quality management, analysis and assurance Data submitted for a national clinical audit are managed, analysed appropriately and quality assured. Findings related to data quality and their implications are reported 
 20 Preliminary data and peer review Preliminary data are available on a timely basis to participating sites to enable a peer review and feedback process in participating sites before findings are published. The local peer review includes analysis of findings and any cases not consistent with good practice to identify and feed back to the national organization any clinically acceptable exceptions not previously acknowledged in data collection 
 21 Identification of good practice and shortcomings in the quality of care Areas of good practice and shortcomings between what is done and what should be done in the delivery of care are identified and reported 
 22 Analysis of causes of shortcomings in the quality Participating sites are supported in identifying root causes of any shortcomings in the quality of care 
 23 Facilitation of improvements A strategy is adopted and implemented for proactively facilitating the implementation of improvements in the quality 
 24 Reports Complete, timely and useful reports are provided for all stakeholders 
 25 Communication Communication with all stakeholders is maintained throughout all stages of the clinical audit 
Outcome aspects 
 26 Level of participation A target is set and met for the number of eligible organizations and services that participate in a national clinical audit 
 27 Reliability of data Reliability testing or independent validation of data collection demonstrates that the reliability of data collected for the clinical audit meets or exceeds an established target 
 28 Timeliness of reports on preliminary data Reports of preliminary data collected are supplied to participating sites promptly following the completion of data submission, that is, within a short number of weeks of the deadline for submission of the data 
 29 Timeliness of complete reports Reports of final data collected are provided promptly, that is, within a short number of weeks of the deadline for submission by participating sites of any additional data following review of preliminary findings and cases that did not appear to meet the quality-of-care measures 
 30 Evidence of improvements in the quality of care Improvements in the quality of care from one time period, for example, 1 year, to the next of a national clinical audit are sufficient to justify continuation of the audit 
No. Topic Standard 
Structural aspects 
 1 Ethical basis The subject of a national clinical audit has an ethically based justification for the conduct of the audit as a means of achieving improvement in the quality of care provided to patients 
 2 Governance There is continuous and effective governance of a national clinical audit by a group that assumes accountability for the effectiveness of the audit 
 3 Stakeholder involvement All key stakeholders in a national clinical audit are identified and actively involved in all aspects of the design and conduct of the audit. Key stakeholders include patients or service users or their representatives 
 4 Resources Resources allocated to a national clinical audit at a national level are appropriate for the valid and reliable measurement and proactive facilitation of improvement of the quality of patient care for the subject of the audit 
 5 Roles and responsibilities and project plan Roles and responsibilities for the design and conduct of the national clinical audit at national and participating site levels are comprehensively described and there is a published project plan for the execution of all aspects of the audit 
Process aspects 
 6 Recruitment The national clinical audit is continuously proactive in recruiting the participation of all sites intended for inclusion 
 7 Improvement-driven aims and objectives The stated aims and objectives of a national clinical audit reflect an intention to drive improvements in the quality of patient care 
 8 Population or sample The population or sample selected for a national clinical audit is suitable for drawing valid conclusions in relation to the stated objectives of the audit and for minimizing bias in the findings 
 9 Data collection strategy The strategy selected for collecting data for a national clinical audit is suitable for the stated objectives of the audit 
 10 Quality-of-care measures Quality-of-care measures used in a national clinical audit meet the criteria for such measures that are established by the governance board and are capable of producing valid and reliable data about the quality of care 
 11 Data elements and data sources Data specified for the collection for a national clinical audit permit judging compliance with the quality-of-care measures in accordance with the objectives of the audit, using currently available data to the extent possible to avoid an unreasonable data collection burden on participating sites 
 12 Data collection and handling protocol or manual A comprehensive protocol describes in detail all aspects of data collection and handling for a national clinical audit and is accessible by all stakeholders and all participants in the audit 
 13 Protection of patient identity Systems and processes are designed and fully implemented to protect the identity of patients whose care is assessed in a national clinical audit consistent with applicable national laws and regulations 
 14 Identification and handling of ethical issues Any ethics-related issues in a national clinical audit are recognized quickly and acted on appropriately 
 15 Training and support for data collectors Data collectors for a national clinical audit are trained to follow the data collection protocol and are supported to supply complete, accurate, reliable and timely data 
 16 Pilot testing The design, the data collection process, protocol and tools to be used in a national clinical audit are pilot tested before all intended sites are invited to participate and action is taken on the lessons learned from the pilot test 
 17 Reliability testing The reliability of the data collected for a national clinical audit is tested and the findings are included in reports 
 18 Data linkages If a national clinical audit relies on linkages with electronic records or databases that contain information about individual patients, systems are implemented to ensure the security of the linkages 
 19 Data quality management, analysis and assurance Data submitted for a national clinical audit are managed, analysed appropriately and quality assured. Findings related to data quality and their implications are reported 
 20 Preliminary data and peer review Preliminary data are available on a timely basis to participating sites to enable a peer review and feedback process in participating sites before findings are published. The local peer review includes analysis of findings and any cases not consistent with good practice to identify and feed back to the national organization any clinically acceptable exceptions not previously acknowledged in data collection 
 21 Identification of good practice and shortcomings in the quality of care Areas of good practice and shortcomings between what is done and what should be done in the delivery of care are identified and reported 
 22 Analysis of causes of shortcomings in the quality Participating sites are supported in identifying root causes of any shortcomings in the quality of care 
 23 Facilitation of improvements A strategy is adopted and implemented for proactively facilitating the implementation of improvements in the quality 
 24 Reports Complete, timely and useful reports are provided for all stakeholders 
 25 Communication Communication with all stakeholders is maintained throughout all stages of the clinical audit 
Outcome aspects 
 26 Level of participation A target is set and met for the number of eligible organizations and services that participate in a national clinical audit 
 27 Reliability of data Reliability testing or independent validation of data collection demonstrates that the reliability of data collected for the clinical audit meets or exceeds an established target 
 28 Timeliness of reports on preliminary data Reports of preliminary data collected are supplied to participating sites promptly following the completion of data submission, that is, within a short number of weeks of the deadline for submission of the data 
 29 Timeliness of complete reports Reports of final data collected are provided promptly, that is, within a short number of weeks of the deadline for submission by participating sites of any additional data following review of preliminary findings and cases that did not appear to meet the quality-of-care measures 
 30 Evidence of improvements in the quality of care Improvements in the quality of care from one time period, for example, 1 year, to the next of a national clinical audit are sufficient to justify continuation of the audit 

Summary of literature supporting the proposed standards

Structural aspects

Ethical basis

Benefits and risks to patients in the current healthcare delivery system and whether or not the proposed audit responds to the risk–benefit analysis should be determined [7, 8]. A range of considerations can be included in the risk–benefit analysis [9, 10], including the incidence or prevalence of the condition that is the subject of the work; the impact the condition has for patients, families and the health and social care systems; availability of evidence that current quality of care is not consistent with best practice or there is significant variation in practice among healthcare organizations or evidence that meaningful improvements in the quality of care can be achieved.

Governance

Governance should provide for transparency to all stakeholders in the audit on its operation, accountability for decision-making and reporting of results [4, 5]. A named governance board for a national clinical audit should include representatives of all key stakeholders in the audit including patients or their immediate representatives [4] and representatives of participating sites.

Stakeholder involvement

All relevant stakeholders, including patients and their representatives, should be involved from the beginning of the clinical audit through to the proactive facilitation of improvements [9, 11].

Resources

The resources needed to operate an audit at a national level and in participating sites should be considered carefully when the audit is designed [4, 5]. The most efficient methods that are likely to be effective in meeting the audit objectives should be used. Population-based audits are not justified when extensive time is required for data collection and validation, particularly when appropriate statistically determined samples could be used. Resources allocated should consider the need to facilitate changes in practice to improve the quality of patient care in participating sites.

Roles and responsibilities and project plan

The roles and responsibilities of everyone involved in an audit at national and site levels should be defined comprehensively [5], including demonstration of commitment to and leadership of participation in the audit and implementation of improvements in care [9]. The project plan should include timelines for the national organization and participating sites and the published timelines should be met routinely [11].

Process aspects

Recruitment

Goals and processes for recruitment and retention of participating sites should be described clearly and their effectiveness assessed periodically. Variations between intended and actual levels of recruitment and participation should be continuously evaluated for their risk of introducing bias in the audit findings [4]. Incentives for participation [4, 12] and reasons for non-participation and non-continuation in the audit should be identified and acted on by the governance board. The recruitment process should provide for early and full disclosure of the requirements for participation to potential participating sites, including access to the complete project plan, defined roles and responsibilities, data collection tools to be used and a realistic estimate of resources needed for participation.

Improvement-driven aims and objectives

The stated aims and objectives of a national clinical audit should be explicit about the intentions to have participating sites use the audit to drive improvements in the quality of care and the organization operating the audit to lead and support the implementation of needed improvements in the quality. The design and conduct of the audit should use several strategies that are likely to facilitate improvements in the quality of care in participating sites [13].

Population or sample

The population or sample should be clearly specified [13–15] and selected to approximate the relevant characteristics of the target population as much as possible [4]. The desired number of cases should consider the magnitude of an expected, clinically meaningful effect; practicality; cost; and the intention to show the effects of interventions to achieve improvement from one data collection period to another [4]. Also, the population or sample, along with the data collection strategy and data sources should be selected to enhance representativeness and minimize potential sources of bias [4], and if consistent with the audit's objectives, enable international comparisons. The number of cases in the population or sample should not place an unnecessary burden of data collection on participating sites [13].

Data collection strategy

The data collection strategy should be suitable for the audit objectives and the availability and accessibility of data needed, and should not put an undue burden on those supplying data.

Quality-of-care measures

Measures enable numeric quantification, usually reported as percentages or ratios, of healthcare quality for the subject and objectives of a clinical audit [16]. The governance board for the national audit should establish criteria for the selection of quality-of-care measures to be used in the audit, such as: [16–23] evidence based [5, 17, 24] or agreed by consensus in the absence of valid evidence [25–29]; scientifically acceptable, for example, that the measures have been tested and demonstrated to produce valid and reliable information about the quality of care [17, 25, 30–33]; important to measure and report; usable and relevant; feasible to collect; efficient and avoiding risks to patients [17]. The validity [32, 34–39] and sensitivity or positive predictive value (PPV) of the measures should be determined [40–47]. Cases identified as ‘false positives’ should be reviewed individually to identify how the measures need to be improved [40–47].

Data elements and data sources

Data to be collected should be derived from the quality-of-care measures, be carefully defined [32, 48, 49] and permit judging compliance with quality-of-care measures in accordance with the audit objectives, using currently available data to the extent possible to avoid an undue burden on participating sites [5, 9, 13]. Data elements should be able to be used to support QI efforts at local level [13]. The collection of ‘nice to know’ or optional data should be kept to a minimum [4]. The reliability of any data retrieved from secondary sources should be tested [32, 50]. Measurement of patient-reported outcomes or patient experiences should use tools that have been designed and validated for this purpose [5].

Data collection protocol or manual

All aspects of data collection, collation, analysis and control for a national audit should be described in a protocol or manual [5, 9, 25]. The protocol or manual should be easily accessible to all stakeholders in an audit, particularly to healthcare organizations prior to their need to register for participation in the audit.

Protection of patient identity

The governance board of a national clinical audit must assure that the systems designed to protect the identity of patients whose care is assessed in the audit are robust and consistent with applicable national laws and regulations, and that they are continuously implemented in all sites participating in the audit.

Identification and handling of ethical issues

Provision should be made to handle ethical issues that emerge at each of these stages of a national audit [8, 51, 52]: the design of the audit; when the findings are available; and when findings from repeat data collection are available. The governance board should assume responsibility for decisions made on the handling of any ethical issues in the audit [8].

Training and support for data collectors

Training on carrying out data collection and submission for a national audit should be available for all clinical or non-clinical staff collecting and submitting data [4, 5, 13, 25, 32, 53–55]. Training manuals and live support should be easily accessible [4]. Feedback from data collectors about the value of the training, materials and support provided should be regularly sought and the findings acted on.

Pilot testing

The design, data collection process, protocol and tools to be used in a national audit should be thoroughly pilot tested in several sites eligible for participation in the audit before the audit involves all intended sites [4, 25, 32, 33, 53, 56]. Action should be taken on the lessons learned from the pilot test.

Reliability testing

Data collected for a national clinical audit should be tested for reliability, sometimes referred to as re-abstracting data [25, 32, 54, 55, 57–62], independently by the national clinical audit [5, 54, 55, 57, 58, 63]. The reliability of data should be reported routinely when the findings are reported [55, 57, 60].

Data linkages

When a national clinical audit uses linkages with electronic records or databases to capture patient care information, robust systems must be implemented to ensure the continuous security of the linkages [5, 9].

Data quality management, analysis and assurance

The quality of data submitted for a national audit should be monitored and assured [5], preferably using systems that use front-end logic to check data quality [4, 5, 11, 13]. Findings of quality assurance checks should be reported to the governance board, used to make changes in the conduct of the audit as needed [4], and included in reports.

Preliminary data and peer review of cases

Participating sites should have access to and review their preliminary data and cases not consistent with good practice, through a local peer group process [5, 54], and prepare for implementing any needed improvements [64, 65]. Sites should be able to correct their data in the national audit prior to publication, if evidence of error or inappropriate judgement during data collection is supplied to the national audit.

Identification of good practice and shortcomings in the quality of patient care

A national audit should identify areas of good practice as demonstrated by compliance with the quality-of-care measures used. Gaps or shortcomings between the actual quality of care and best practice as represented by the measures also should be explicitly identified [5, 66–68]. Participating sites should easily be able to determine their levels of performance in comparison with other comparable sites.

Analyses of causes of shortcomings in quality

A national clinical audit should help participating sites to identify and understand the causes of their shortcomings in care through providing examples of analysis of variation [69, 70], root cause analysis of problems revealed and case studies [31, 71].

Facilitation of improvements

A national clinical audit should adopt and implement a framework for spread of good practice [72, 73], identify what works best among improvement initiatives and encourage the rapid adoption of those initiatives [13, 54, 55, 74]. The national audit should create and market practical tools to help healthcare organizations improve their performance [13, 28, 75]. Examples of improved performance should be disseminated through professional meetings and awards recognizing substantially improved practice [13, 54, 55, 76].

Timely, complete and informative reports

Reports should be timely and reflect the SQUIRE guidelines to the extent feasible [6]. They should be designed to help sites quickly and easily identify their areas of needed improvement in relation to their peers [13]. Reports on national clinical audits should be provided for groups of identified stakeholders, including patients [9].

Communication among all stakeholders

Communication with all stakeholders should be maintained throughout every stage of the clinical audit [9], particularly with management, clinicians and data collectors in participating sites.

Outcome aspects

Level of participation

A target level of participation in a national clinical audit by eligible sites should be established by the governance board, which should publish the participation rate in comparison with the target.

Reliability of data

A target level of reliability of data collected should be established by the governance board, which should publish the findings of reliability testing of data collected. Reliability testing or independent validation of data collection should demonstrate that the reliability of data collected for the clinical audit is at least 90% or an equivalent kappa value [55, 77].

Timeliness of reports on preliminary data and complete reports

Reports of preliminary data collected should be supplied to participating sites for local review in real time or within weeks following the submission of the data. Complete reports of national clinical audits should be supplied to participating sites and other stakeholders within weeks of the review of preliminary data by participating sites.

Evidence of improvements in quality

The governance board should establish and apply measures for judging the effectiveness of the audit. Depending on the audit findings, the board should determine how data are being used to drive QI or make changes in the audit to influence the implementation of improvements needed in participating sites.

Discussion

The Cochrane systematic review on audit and feedback concluded that ‘audit and feedback generally leads to small but potentially important improvements in professional practice. The effectiveness of audit and feedback seems to depend on baseline performance and how the feedback is provided’ [78]. This conclusion is based on a rigorous review of published trials of audit and feedback, which is defined as ‘a summary of clinical performance over a specified period of time’.

The review does not specifically address national clinical audits or QI studies. However, in the absence of standardization of the work carried out, particularly the quality of the processes used to measure the quality and provide information to participating sites on their own performance, it is unclear what conclusions can be drawn about the potential effectiveness of national projects that are intended to measure and improve the quality of health care. Given the substantial economic resources involved in these projects and the constraints on funding of healthcare systems around the world, it is essential that these activities provide optimal cost–benefit to healthcare systems. Their potential value should be judged only when they are carried out to the highest standard.

The proposed standards should be achievable by organizations carrying out national clinical audits or QI studies because they represent current practice as described in the relevant literature. The biggest issue is that potentially not every national audit or QI activity complies with every standard. A major shortcoming is likely to be the informality of many of the processes involved, such as training data collectors, and the lack of full documentation of many of the measurement processes such as a complete data collection and quality control manual and a realistic project plan for both the national organization operating the activity and participating sites.

Conclusion

National clinical audits or QI studies have been carried out in the UK and other countries for over 15 years. In the absence of consensus agreement on best practice in the design and conduct of these activities, the audits or QI projects can vary in their effectiveness and cost–benefit, particularly at participating site level. Synthesis of available literature indicates that there are many aspects of the design and operation of national clinical audits or QI studies for which good practice has been defined and some published examples of standards for the design and operation of an audit or QI study being achieved. A set of 30 standards for the design and conduct of a national clinical audit or equivalent activity is proposed, based on published literature and examples of best practice. It is hoped that the standards will be debated among stakeholders in national clinical audits and equivalent activities as the basis for undertaking these projects in the future.

Funding

The work was supported in entirety by Healthcare Quality Quest Ltd, prompted by a request from Parkinson's UK to evaluate the Parkinson's national clinical audit.

Acknowledgements

Mary Pearce of Healthcare Quality Quest carried out the searches of literature for this work and validated the abstracting of the evidence.

References

1
National Institute for Clinical Excellence
Principles for best practice in clinical audit
2002
Abingdon
Radcliffe Medical Press
2
Amess
M
Walshe
K
Shaw
C
, et al.  . 
The audit activities of the medical royal colleges and their faculties in England
1995
London
CASPE Research
3
National Clinical Audit and Patient Outcomes Programme
 
4
Gliklich
RE
Dreyer
NA
Registries for Evaluating Patient Outcomes: A User's Guide
 
2nd edn
Rockville, MD
Agency for Healthcare Research and Quality
 
(Prepared by Outcome DEcIDE Center [Outcome Sciences, Inc. d/b/a Outcome] under Contract No. HHSA29020050035I TO3) AHRQ Publication No.10-EHC049. 2010 www.effectivehealthcare.ahrq.gov (28 October 2012, date last accessed)
5
Australian Commission on Safety and Quality in Healthcare
Strategic principles for a national approach to clinical quality registries and operating principles for australian clinical quality registries
 
6
Ogrinc
G
Mooney
SE
Estrada
C
, et al.  . 
The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration
Qual Saf Health Care
 , 
2008
, vol. 
17
 
Suppl. I
(pg. 
i13
-
32
)
7
Casarett
D
Fox
E
Tulsky
JA
Recommendations for the ethical conduct of quality improvement. A Report of the National Ethics Committee of the Veterans Health Administration
2002
National Center for Ethics in Health Care, Veterans Health Administration
8
Dixon
N
Review of ethics issues related to clinical audit and quality improvement activities
2009
London
Healthcare Quality Improvement Partnership
 
9
Assessing and improving quality in national clinical audits
2011
London
Healthcare Quality Improvement Partnership
 
10
Department of Health
Criteria for selecting and prioritising topics for national clinical audits to be funded by DH NCAPOP
2009
 
11
Pearson
M
Lessons from the management of acute myocardial infarction
Heart
 , 
2005
, vol. 
91
 
Suppl. 2
(pg. 
ii28
-
30
)
12
Cornish
JA
Tilney
HS
Tan
E
, et al.  . 
The National Bowel Cancer Audit Project: what do trusts think of the National Bowel Cancer Audit and how can it be improved?
Tech Coloprocto
 , 
2011
, vol. 
15
 (pg. 
53
-
9
)
13
Peterson
ED
Roe
MT
Rumsfeld
JS
, et al.  . 
A Call to ACTION (acute coronary treatment and intervention outcomes network): a national effort to promote timely clinical feedback and support continuous quality improvement for acute myocardial infarction
Circ Cardiovasc Qual Outcomes
 , 
2009
, vol. 
2
 (pg. 
491
-
9
)
14
Harris
AH
Reeder
RN
Ellerbe
LS
, et al.  . 
Validation of the treatment identification strategy of the HEDIS addiction quality measures: concordance with medical record review
BMC Health Serv Res
 , 
2011
, vol. 
11
 pg. 
73
 
15
Jones
RC
Dickson-Spillmann
M
Mather
MJ
, et al.  . 
Accuracy of diagnostic registers and management of chronic obstructive pulmonary disease: the Devon primary care audit
Respir Res
 , 
2008
, vol. 
9
 pg. 
62
 
16
Remus
D
Fraser
I
Guidance for using the AHRQ quality indicators for hospital-level public reporting or payment
2004
Rockville, MD
Agency for Healthcare Research and Quality
 
17
Chassin
MR
Loeb
JM
Schmaltz
SP
, et al.  . 
Accountability measures—using measurement to promote quality improvement
New Engl J Med
 , 
2010
, vol. 
363
 (pg. 
683
-
8
)
18
Criteria for Evaluation
Measure evaluation criteria
2011
 
National Quality Forum http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx (28 October 2012, date last accessed)
19
Kesmodel
US
Jølving
LR
Measuring and improving quality in obstetrics—the implementation of national indicators in Denmark
Acta Obstet Gynecol Scand
 , 
2011
, vol. 
90
 (pg. 
295
-
304
Epub 4 March 2011 doi:10.1111/j/1600–0412.2011.01078.x
20
Krumholz
HM
Keenan
PS
Brush
JE
Jr
, et al.  . 
American Heart Association Interdisciplinary Council on Quality of Care and Outcomes Research, American College of Cardiology Foundation
Standards for measures used for public reporting of efficiency in health care: a scientific statement from the American Heart Association Interdisciplinary Council on Quality of Care and Outcomes Research and the American College of Cardiology Foundation
J Am Coll Cardiol
 , 
2008
, vol. 
52
 (pg. 
1518
-
26
)
21
Maggard
MA
McGory
ML
Ko
CY
Development of quality indicators: Lessons learned in bariatric surgery
Am Surg
 , 
2006
, vol. 
72
 (pg. 
870
-
4
)
22
Mainz
J
Developing evidence-based clinical indicators: a state of the art methods primer
Int J Qual Health Care
 , 
2003
, vol. 
15
 
Suppl. 1
(pg. 
i5
-
11
)
23
Wollersheim
H
Hermens
R
Hulscher
M
, et al.  . 
Clinical indicators: Development and applications
Neth J Med
 , 
2007
, vol. 
65
 (pg. 
15
-
22
)
24
van der Geer
E
van Tuijl
HF
Rutte
CG
Performance management in healthcare: Performance indicator development, task uncertainty, and types of performance indicators
Soc Sci Med
 , 
2009
, vol. 
69
 (pg. 
1523
-
30
Epub 24 September 2009
25
Travaglia
J
Debono
D
Clinical audit: a comprehensive review of the literature
2009
Sydney, Australia
Centre for Clinical Governance Research, University of New South Wales
 
26
Cryer
C
Injury outcome indicators—validation matters
Int J Inj Contr Saf Promot
 , 
2005
, vol. 
12
 (pg. 
219
-
24
)
27
Ko
DT
Wijeysundera
HC
Zhu
X
, et al.  . 
National Expert Panel
. Canadian quality indicators for percutaneous coronary interventions
Can J Cardiol
 , 
2008
, vol. 
24
 (pg. 
899
-
903
)
28
Lee
DS
Tran
C
Flintoft
V
, et al.  . 
Canadian Cardiovascular Outcomes Research Team/Canadian Cardiovascular Society Heart Failure Quality Indicator Panel
CCORT/CCS quality indicators for congestive heart failure
Can J Cardiol
 , 
2003
, vol. 
19
 (pg. 
357
-
64
)
29
Mian
S
Wagg
A
Irwin
P
, et al.  . 
National audit of continence care: laying the foundation
J Eval Clin Pract
 , 
2005
, vol. 
11
 (pg. 
533
-
43
)
30
Hearnshaw
H
Harker
R
Cheater
F
, et al.  . 
A study of the methods used to select review criteria for clinical audit
Health Technol Assess
 , 
2002
, vol. 
6
 (pg. 
1
-
78
)
31
Campbell
SM
Hann
M
Hacker
J
, et al.  . 
Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes
Qual Saf Health Care
 , 
2002
, vol. 
11
 (pg. 
125
-
30
)
32
Dixon
N
Pearce
M
Guide to ensuring data quality in clinical audits
2010
London
Healthcare Improvement Partnership
 
33
Wagg
A
Mian
S
Lowe
D
, et al.  . 
Continence Programme Working Party
National audit of continence care for older people: results of a pilot study
J Eval Clin Pract
 , 
2005
, vol. 
11
 (pg. 
525
-
32
)
34
Persell
SD
Dolan
NC
Friesema
EM
, et al.  . 
Frequency of inappropriate medical exceptions to quality measures
Ann Intern Med
 , 
2010
, vol. 
152
 (pg. 
225
-
31
)
35
Davies
S
McDonald
KM
Schmidt
E
, et al.  . 
Expanding the uses of AHRQ's prevention quality indicators: validity from the clinician perspective
Med Care
 , 
2011
, vol. 
49
 (pg. 
679
-
85
)
36
Harboe
KM
Anthonsen
K
Bardram
L
Validation of data and indicators in the Danish Cholecystectomy Database
Int J Qual Health Care
 , 
2009
, vol. 
21
 (pg. 
160
-
8
Epub 21 March 2009
37
Head
BJ
Aquilino
ML
Johnson
M
, et al.  . 
Content validity and nursing sensitivity of community-level outcomes from the Nursing Outcomes Classification (NOC)
J Nurs Scholarsh
 , 
2004
, vol. 
36
 (pg. 
251
-
9
)
38
Idvall
E
Hamrin
E
Siöström
B
, et al.  . 
Quality indicators in postoperative pain management: a validation study
Scand J Caring Sci
 , 
2001
, vol. 
15
 (pg. 
331
-
8
)
39
Maas
M
Johnson
M
Moorhead
S
, et al.  . 
Evaluation of the reliability and validity of nursing outcomes classification patient outcomes and measures
J Nurs Meas
 , 
2003
, vol. 
11
 (pg. 
97
-
117
)
40
Borzecki
AM
Cevasco
M
Chen
Q
, et al.  . 
How valid is the AHRQ Patient Safety Indicator ‘postoperative physiologic and metabolic derangement?
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
968
-
76.e1-2
Epub 13 April 2011
41
Borzecki
AM
Kaafarani
H
Cevasco
M
, et al.  . 
How valid is the AHRQ Patient Safety Indicator ‘postoperative hemorrhage or hematoma?
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
946
-
53.e1-2
Epub 7 April 2011
42
Borzecki
AM
Kaafarani
HM
Utter
GH
, et al.  . 
How valid is the AHRQ Patient Safety Indicator ‘postoperative respiratory failure?
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
935
-
45
Epub 7 April 2011
43
Cevasco
M
Borzecki
AM
Chen
Q
, et al.  . 
Positive predictive value of the AHRQ Patient Safety Indicator ‘postoperative sepsis’: implications for practice and policy
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
954
-
61
Epub 7 April 2011
44
Cevasco
M
Borzecki
AM
McClusky
DA
III
, et al.  . 
Positive predictive value of the AHRQ Patient Safety Indicator ‘postoperative wound dehiscence
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
962
-
7
Epub 13 April 2011
45
Cevasco
M
Borzecki
AM
O'Brien
QJ
, et al.  . 
Validity of the AHRQ Patient Safety Indicator ‘central venous catheter-related blood stream infections
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
984
-
90
Epub 13 April 2011
46
Chen
Q
Rosen
AK
Cevasco
M
, et al.  . 
Detecting patient safety indicators: how valid is ‘foreign body left during procedure’ in the Veterans Health Administration?
J Am Coll Surg
 , 
2011
, vol. 
212
 (pg. 
977
-
83
Epub 13 April 2011
47
Scanlon
MC
Harris
JM
II
Levy
F
, et al.  . 
Evaluation of the Agency for Healthcare Research and Quality pediatric quality indicators
Pediatrics
 , 
2008
, vol. 
121
 (pg. 
e1723
-
31
Epub 12 May 2008
48
Clarke
RT
Warnick
J
Stretton
K
, et al.  . 
Improving the immediate management of neutropenic sepsis in the UK: lessons from a national audit
Br J Haematol
 , 
2011
, vol. 
153
 (pg. 
773
-
9
)
49
Sketcher-Baker
KM
Kamp
MC
Connors
JA
, et al.  . 
Using the quality improvement cycle on clinical indicators—improve or remove?
Med J Aust
 , 
2010
, vol. 
193
 
Suppl.
(pg. 
S104
-
6
)
50
Kelly
M
Lamah
M
Evaluating the accuracy of data entry in a regional colorectal cancer database: implications for national audit
Colorectal Dis
 , 
2007
, vol. 
9
 (pg. 
337
-
9
)
51
Dixon
N
Burgess
R
Ethics and clinical audit
New Principles of Best Practice in Clinical Audit
 , 
2011
Oxford
Radcliffe Publishing
52
Dixon
N
Ethics and Clinical Audit and Quality Improvement (QI)—A Guide for NHS Organisations
 , 
2009
London
Healthcare Quality Improvement Partnership
 
53
Indicators.
International quality indicator project
 
http://www.internationalqip.com/indicators.aspx (28 October 2012, date last accessed)
54
Bridgewater
B
Keogh
B
Kinsman
R
, et al.  . 
Sixth National Adult Cardiac Surgical Database Report 2008, Demonstrating Quality
2009
The Society for Cardiothoracic Surgery in Great Britain and Ireland and Dendrite Clinical Systems Ltd
 
http://www.e-dendrite.com (28 October 2012, date last accessed)
55
Shiloach
M
Frencher
SK
Jr
Steeger
JE
, et al.  . 
Toward robust information: Data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program
J Am Coll Surg
 , 
2010
, vol. 
210
 (pg. 
6
-
16
)
56
Roberts
CM
Lowe
D
Barnes
S
, et al.  . 
A prospective study of the practical issues of local involvement in national audit of COPD
J Eval Clin Pract
 , 
2004
, vol. 
10
 (pg. 
281
-
90
)
57
Gompertz
PH
Irwin
P
Morris
R
, et al.  . 
Reliability and validity of the Intercollegiate Stroke Audit Package
J Eval Clin Pract
 , 
2001
, vol. 
7
 (pg. 
1
-
11
)
58
Harrison
DA
Brady
AR
Rowan
K
Case mix, outcome and length of stay for admissions to adult, general critical care units in England, Wales and Northern Ireland: the Intensive Care National Audit and Research Centre Case Mix Programme Database
Crit Care
 , 
2004
, vol. 
8
 (pg. 
R99
-
111
)
59
Hart
S
Berqquist
S
Gajewski
B
, et al.  . 
Reliability testing of the National Database of Nursing Quality Indicators pressure ulcer indicator
J Nurs Care Qual
 , 
2006
, vol. 
21
 (pg. 
256
-
65
)
60
Herrett
E
Smeeth
L
Walker
L
, et al.  . 
MINAP Academic Group
The Myocardial Ischaemia National Audit Project (MINAP)
Heart
 , 
2010
, vol. 
96
 (pg. 
1264
-
7
)
61
Willis
CD
Jolley
DJ
McNeil
JJ
, et al.  . 
Identifying and improving unreliable items in registries through data auditing
Int J Qual Health Care
 , 
2011
, vol. 
23
 (pg. 
317
-
23
Epub 25 March 2011
62
To
T
Estrabillo
E
Wang
C
, et al.  . 
Examining intra-rater and inter-rater response agreement: a medical chart abstraction study of a community-based asthma care program
BMC Med Res Methodol
 , 
2008
, vol. 
8
 pg. 
29
  
http://www.biomedcentral.com/1471-2288/8/29 (28 October 2012, date last accessed)
63
Kragsterman
B
Pärsson
H
Lindbäck
J
, et al.  . 
Swedish Vascular Registry (Swedvasc)
Outcomes of carotid endarterectomy for asymptomatic stenosis in Sweden are improving: results from a population-based registry
J Vasc Surg
 , 
2006
, vol. 
44
 (pg. 
79
-
85
)
64
Mainz
J
Krog
BR
Bjørnshave
B
, et al.  . 
Nationwide continuous quality improvement using clinical indicators: the Danish National Indicator Project
Int J Qual Health Care
 , 
2004
, vol. 
16
 
Suppl. 1
(pg. 
i45
-
50
)
65
Wightman
R
Fielding
J
Green
S
Audit of antipsychotic prescribing in dementia: Cambridgeshire results and lessons learnt
Psychiatr Danub
 , 
2011
, vol. 
23
 
Suppl. 1
(pg. 
S126
-
9
)
66
Seddon
M
Buchanan
J
EPIQ
Quality improvement in New Zealand healthcare. Part 3: achieving effective care through clinical audit
N Z Med J
 , 
2006
, vol. 
119
 pg. 
U2108
 
67
Gale
CP
Roberts
AP
Batin
PD
, et al.  . 
Funnel plots, performance variation and the Myocardial Infarction National Audit Project 2003–2004
BMC Cardiovas Disord
 , 
2006
, vol. 
6
 pg. 
34
 
68
Qvist
P
Rasmussen
L
Bonnevie
B
, et al.  . 
Repeated measurement of generic indicators: a Danish national program to benchmark and improve quality of care
Int J Qual Health Care
 , 
2004
, vol. 
16
 (pg. 
141
-
8
)
69
Barter
S
Drinkwater
K
Remedios
D
National audit of provision of MRI services 2006/07
Clin Radiol
 , 
2009
, vol. 
64
 (pg. 
2384
-
90
)
70
Tawn
DJ
Squire
CJ
Mohammed
MA
, et al.  . 
National audit of the sensitivity of double-contrast barium enema for colorectal carcinoma, using control charts for the Royal College of Radiologists Clinical Radiology Audit Sub-Committee
Clin Radiol
 , 
2005
, vol. 
60
 (pg. 
558
-
64
)
71
Dixon
N
Getting clinical audit right to benefit patients
2011
Romsey, Hampshire UK
Healthcare Quality Quest
72
Massoud
MR
Nielsen
GA
Nolan
K
, et al.  . 
A framework for spread: from local improvements to system-wide change
2006
Cambridge, MA
Institute for Healthcare Improvement
 
IHI Innovation Series White Paper http://www.IHI.org/knowledge/Pages/IHIWhitePapers/AFrameworkforSpreadWhitePaper.aspx (28 October 2012, date last accessed)
73
Nolan
K
Schall
MQ
Erb
F
, et al.  . 
Using a framework for spread: the case of patient access in the Veterans Health Administration
Jt Comm J Qual Patient Saf
 , 
2005
, vol. 
31
 (pg. 
339
-
47
)
74
Rudd
AG
Lowe
D
Irwin
P
, et al.  . 
Intercollegiate Stroke Working Party
National stroke audit: a tool for change?
Qual Health Care
 , 
2001
, vol. 
10
 (pg. 
141
-
51
)
75
Grant
RL
Batty
GM
Aggarwal
R
, et al.  . 
National sentinel clinical audit of evidence-based prescribing for older people: methodology and development
J Eval Clin Pract
 , 
2002
, vol. 
8
 (pg. 
189
-
98
)
76
Husk
J
Achieving changes in practice from national audit: national audit of the organization of services for falls and bone health in older people
J Eval Clin Pract
 , 
2008
, vol. 
14
 (pg. 
974
-
8
)
77
Williams
SC
Watt
A
Schmaltz
SP
, et al.  . 
Assessing the reliability of standardized performance indicators
Int J Qual Health Care
 , 
2006
, vol. 
18
 (pg. 
246
-
55
)
78
Ivers
N
Jamtvedt
G
Flottorp
S
, et al.  . 
Audit and feedback: effects on professional practice and healthcare outcomes
Cochrane Database Syst Rev
 , 
2012
 
Issue 6. Art. No.: CD000259 doi:10.1002/14651858.CD000259.pub3