External quality assurance (EQA) or proficiency testing is a system for objectively checking the laboratory’s performance using an external agency or facility for evaluation of participant performance against pre-established criteria by means of interlaboratory comparison.
According to ISO/IEC 17043:2010 the EQA provider reports the findings to the participants in a way that facilitates the verification of trueness, to establish the corrective actions needed and to assess the success of prior corrective actions. Further, ISO 15189:2012 requires that a laboratory “shall monitor the performance of its examination methods, by participation in external quality assessment, appropriate to the examination and interpretation of examination results, including Point of Care Testing [POCT] examination methods.”
It is important to note that there are 2 different types of EQA scheme based on the fundamental purposes. The first type is regulatory. Because of the importance of pathology results in healthcare, many jurisdictions require the performance of pathology laboratories to achieve a basic minimum standard and failure to achieve these standards may result in significant consequences for the laboratory’s right to practice. The need to “pass” this type of EQA can lead to unintended consequences. For example, laboratories may treat these EQA specimens differently to patient samples to ensure acceptable performance. In addition, the performance criteria are usually able to be achieved by nearly all laboratories.
The other type of EQA programs are described as “aspirational or educational.” The aim of these programs is to improve the quality of laboratory testing, and therefore not all laboratories will achieve the performance goals at the time of implementation. Educational programs may also offer support to participants in the form of additional troubleshooting advice, webinars on interpretation of QC and EQA, and workshops on addressing measurement problems identified in the EQA program.
The structure of an EQA program is defined by the EQA samples, the frequency of samples, the analytical performance specifications (APS), and how the target values are established. The sample may be verified commutable with clinical samples, have a range of concentrations, and be repeated during the EQA cycle. Recall that an EQA material (EQAM) used as an EQA sample is considered commutable when a measuring system (MS) produces the same result for an EQAM as it does for an authentic patient sample that contained the same analyte concentration. Additionally, there may be samples offered in the program that test the interaction between the measurand and hemolysis, lipemia, bilirubin, or common interfering substances. The APS may be set using biological variation, clinical outcome studies, or state of the art derived from the statistics of the participants results (SD, or z score). The target may be set using a higher order reference measurement procedure or using the mean or median of all participants in the case of commutable EQAM, or be set as the mean of the participants’ method group in the case of noncommutable EQAM.
This discussion was under the auspices of the Working Group on Traceability Education and Promotion of the Joint Committee on Traceability in Laboratory Medicine.
What are the pros and cons of different EQA schemes (regulatory vs aspirational/educational)?
Michael Spannagl: In Germany, both regulatory and educational EQA schemes are offered. Concerning the regulatory scheme, 50 years ago the scientific community of laboratory physicians and clinical chemists, the regulatory bodies, and the general board of physicians decided to introduce a mandatory project for internal and external quality control in laboratory medicine (RiliBÄK). Starting in clinical chemistry, it has been extended to many other fields of laboratory medicine. The RiliBÄK covers the fields of hematology, hemostasis, oncology, immunology, infectious disease, and all kinds of diagnostics on the gene level. Furthermore, there are special applications like sperm diagnostics or newborn screening. In the regulatory system of the RiliBÄK measurands, acceptance limits and frequency of participation are strictly defined. The limits set by RiliBÄK have been balanced between state of the art and biological variation and are continuously adapted, mainly based on scientific progress and long-term results of proficiency testing in Germany. In the area of clinical chemistry, the reference measurement procedure concept is established as an evaluation criterion. In this way, the metrological traceability chain is directly linked to the routine laboratories. The challenge here, however, is that the commutability of the EQA material must be ensured, which means a large effort in terms of logistics and costs. It must be emphasized that the mandatory approach in Germany is also performed with educational aspects by the organizing reference institutions and the scientific communities in all fields of laboratory medicine.
The educational EQA schemes, which are outside the regulatory system of the RiliBÄK, are highly flexible and can be quickly adapted to the current needs of laboratory medicine. They can provide the basis for harmonization and standardization processes of new measurands.
From the beginning, RiliBÄK started as a mandatory project. However, it is performed by the scientific communities, which means there is a considerable effort in education and criteria established to fulfill scientific criteria.
Andrew Quintenz: A regulatory scheme, or program, is one that is intended for laboratories to demonstrate they achieve minimum quality requirements as measured by EQA and required in a government regulation. The regulation may call out performance criteria, but more likely, frequency to perform EQA, number of failures allowed before a license is in jeopardy, identification of analytes that require EQA, and/or authorized EQA providers. In some countries, there may be a requirement to use a specific EQA provider for certain analytes.
PRO: These schemes ensure that government officials are aware of failures and action is taken to prevent the deficient laboratory from providing inaccurate results that could lead to patient harm.
CON: This approach only comes with a stick and no carrot. A laboratory may approach an EQA challenge sample with trepidation and not run it entirely as a normal patient sample, thus preventing the laboratory from obtaining a true assessment of their normal quality. The scheme may also not provide additional statistics or resources to help the laboratory understand their full placement amongst their peers and what they could do to incrementally improve their quality.
An aspirational scheme is typically created with the intent of being a partner with the laboratories by providing them insights about their testing quality and deeper reports and tools to help improve their performance.
PRO: Performance criteria may be set tighter (the goal) than regulatory schemes (the floor). Laboratories subscribing to these programs are often focused on quality improvement rather than checking off a box. Aspirational schemes may have resources to help laboratories and guide them on their quality journey.
CON: Aspirational schemes rely on the good faith of the laboratories to assess their results and make changes as needed. A laboratory that continually fails a given analyte, conceivably, could continue testing without intervention, until they cause an error when a clinician acts on an incorrect result.
Graham Jones: EQA is recognized as a vital component of obtaining and verifying the similarity of results from different laboratories and is recognized as such in the ISO standard 15189. The large number of variables involved in medical laboratory testing gives a wide scope for laboratories to produce different results for the same measurand. Indeed, as a personal opinion, I expect laboratories to be giving different results, unless there is evidence through EQA that the results are similar. Behind this statement, there is the need to define what “similar” means for laboratory tests. EQA providers can supply APS to allow assessment of acceptable variation between laboratories. Additionally, EQA providers must supply a central target for results. Providers can use a range of approaches to setting targets and APS, and the purpose of the EQA program plays an important role in this.
If the goal of EQA is to reduce between-laboratory variation, and variation from an agreed target, then the question is how best to achieve this. Different approaches may be described as regulatory, improvement, or aspirational. A regulatory approach is to set a standard required by all laboratories and require a passing performance for laboratories to continue to practice. An improvement approach would be to set standards which are achievable by “good performing laboratories” and flag poorer performing laboratories with the aim of self-improvement. An aspirational program is one setting the optimal standard, even if not easily achievable, with the aim of stimulating improvement in the industry.
A pro for aspirational programs is that they can continually send a message to enrolled laboratories about the possibility of improvement. A con may be that laboratories become accustomed to a failure to meet the standard and become complacent.
A clear pro for regulatory programs is that poorly performing laboratories may be removed from providing patient care. This outcome can be seen as protecting patients by ensuring analytical quality. Clearly, if the standards are set too tightly, too many laboratories may be suspended with consequent loss of service. A particular con may be the behavioral response to this type of program. With the threat of regulatory action, laboratories may be tempted to optimize the measurement process, for example, testing just after calibration, in duplicate, by a senior analyst. Thus, the results do not reflect the actual quality delivered for patient care. An improvement program, by using achievable targets and without threat of regulatory responses, may be more likely to be performed with the goal of representing actual performance, i.e., measuring at different times of day, with different analysts, together with patient samples. The hoped-for outcome is that responses to out-of-range results lead to improvements on routine testing and thus patient care. Some years ago, the Australian Government approached the Royal College of Pathologists of Australasia Quality Assurance Programs (RCPAQAP) with the aim of using its (non-regulatory) programs to identify “poorly performing laboratories,” which could then be subject to earlier review by an accreditation team. Even this modest potential additional use of EQA results stimulated concern amongst users that the data in the program may not reflect actual routine performance. This additional use was not brought into practice, and it was noted that a truly educational program must be separate from a regulatory program for best effect.
Sverre Sandberg: Most importantly, an EQA scheme should encourage laboratories to improve their quality. It should, in principle, not be used as a system to “punish” laboratories that don’t fulfil the “acceptance” criteria of the scheme. I don’t have much experience with regulatory EQA schemes. They usually have wide “acceptance” limits that are usually not related to medical need or biological variation. And we can ask: what are they related to, to major risks? The EQA providers organizing “regulatory schemes” should state the reason for using the limits they use. In regulatory schemes, it might be more tempting for the participants to cheat with the results and not treat the samples as normal patient samples. EQA schemes should be dynamic and educational, which means that they should change depending on the “problem” that should be solved. The relationship between the laboratory and the EQA provider should reflect mutual trust so that the EQA provider can help the laboratory to solve problems—then we have external quality assurance.
Mauro Panteghini: My immediate thought about the alternatives of regulatory or educational EQA is if we should continue to reiterate it. First, the scope of currently available EQA schemes is not always quite clear or explicitly stated. People consider as “regulatory,” schemes in which some minimum standard should be fulfilled (but how to objectively define these standards?). However, the sole scope of the EQA schemes should be to check the suitability of laboratory results for their clinical use. Second, to apply one of the two definitions to different EQA schemes is not always possible. Indeed, some “regulatory” schemes would also like to educate participants, and some “educational” schemes would like to see their approach in some way applied in the regulatory setting.
Greg Miller: Results from regulatory EQA schemes are used to monitor the quality of a laboratory’s test results as one aspect of accreditation and maintaining a license to provide laboratory services. The regulatory agency typically defines the APS that must be met to be considered acceptable performance. A pro for a regulatory scheme is that all laboratories are required to participate and the process is intended to ensure all participants meet minimally acceptable performance. A con is that the APS may be set large enough to enable a pass rate, such as >95%, that may not identify poorly performing measurement procedures used by some laboratories.
Results from aspirational/educational EQA schemes are also used to monitor the quality of a laboratory’s test results, but the APS can be chosen to reflect more closely what is needed for good clinical management. A pro for an educational scheme is EQA samples intended to challenge the limitations, e.g., interfering substances, of measurement procedures can be included without fear that a non-passing result will jeopardize a laboratory’s accreditation status. Instead, the result can be used to educate a laboratory that a measurement procedure has a limitation they might not have been aware of. A potential con of an educational EQA scheme is a laboratory may fail repeatedly without jeopardy of being shut down, although most accreditation agencies will review EQA for acceptable performance even without strict regulatory requirements for the EQA scheme itself.
A challenge shared by regulatory and aspirational/educational EQA schemes is that EQA samples that are commutable with clinical samples are only available for a relatively small number of measurands, possibly 20 to 30 of the thousands in a test order catalog. Consequently, most measurement procedures for a measurand are grouped into “peer groups” that are expected to have the same noncommutability bias, and thus results are likely to agree. Results from one laboratory are compared to the “peer group” mean which confirms that laboratory is performing similarly to others using the same measurement procedure. However, there is no information to determine if results from different “peer groups” of measurement procedures agree with each other or if measurement procedures are correctly calibrated to a higher order reference system.
How does the purpose of the EQA (regulatory or aspirational/educational) impact on the design of these EQA programs?
Michael Spannagl: Regulatory EQA schemes with defined acceptance limits and rules offer the possibility of horizontal evaluations as long-term observations and continuous improvement. Educational EQA schemes can be more experimental and address additional aspects outside of the main routine analysis.
At INSTAND, responsibilities in the framework of RiliBÄK are clearly separated:
(a), Organization and logistics are provided as a comprehensive platform for all surveys.
(b), Statistical evaluation is strictly performed by defined statistical process according to ISO 13528.
(c), In certain circumstances, deviation is possible that has to be interpreted and documented by the responsible physician (medical survey manager from the scientific community) or the diagnostic group of the scientific society, respectively.
Furthermore, several educational “EQA” have been introduced with a focus on pre- and post-analytical issues or orphan disease (control material not available). The possibility to transfer and process big data allows for more and more imaging in education and quality control for medical diagnostics.
Andrew Quintenz: Regulatory schemes, often like laboratory regulations, are designed with criteria that ensure the laboratories with poor quality fail and the rest pass with little attention as to the degree with which a laboratory passed. Regulatory schemes may have only a few challenge samples over the course of a year as the goal is assessing suitability to continue testing, as opposed to information to guide quality improvements.
A regulatory scheme may also impact the design of the materials. A scheme that is specific to one country, province, or region, may be able to be designed for that demographic. That is, there may be a smaller variety of instruments used and the EQA sample preparation may not require as much manipulation to meet a wide variety of instruments/methods.
Similarly, for samples that can be shipped short distances, use of fewer stabilizing techniques may lead to improved commutability. Additionally, if shipping distances are short, and less costly, samples may be prepared at the time of the challenge and shipped under refrigerated conditions, thus preserving the similarity to patient samples.
Aspirational schemes are often multinational and may be shipping to a wide variety of climates at the same time. Thus, the same challenge needs to traverse to hot, humid climates at the same time as shipping into cold winter climates. These shipping conditions may be multiple days, and therefore need to be prepared for longer storage and may be frozen or lyophilized to ensure equal access across the globe.
Aspirational schemes may offer more frequent samples than regulatory schemes to allow the laboratory to assess their impact of quality improvements deriving from previous challenges. Similarly, they may provide for larger sample volumes to allow the laboratories to retain some amount for follow-up evaluation should the performance not be what was anticipated.
Graham Jones: The key factors that discriminate these types of EQA programs are the applied APS, and the response to a failure to meet the standard defined by those APS. Typically, regulatory programs have wider APS, but severe consequences for failure. By contrast, an aspirational program will have tighter APS but no punitive consequences for failure. Improvement programs fit between these extremes. Analytical quality, here taken to encompass precision, bias, and analytical selectivity, can be seen as a continuum, from a perfect result through to a grossly deviating result. As the MS performance deteriorates from perfect to grossly deviant, the effect on the risk of harming a patient changes. Minor variation has no effect on patient outcome. An example of this would be precision at the optimal level relative to biological variation. As performance deteriorates, extra “noise” is added to the results, which does not reflect changes in the patient. As this noise increases, the risk to patient care increases. Regulatory programs seek to avoid performance sufficiently bad that it creates a high risk of patient harm. Aspirational programs aim for the lowest possible risk of patient harm, and improvement programs aim to reduce the greatest risks and promote changes to minimize risk. Regulatory programs are linked to an accreditation process or similar, where failure can lead to laboratory closure. Aspirational programs seek to highlight all measurement weaknesses, but if the optimal performance is not possible, failure to meet the standard is expected and no punitive consequences can be applied. An improvement program would generally set a standard that is achievable by many laboratories but aims to improve the less-well performing laboratories and MSs. Thus, the response in the laboratory to a failure should be “how can we improve this assay.”
Sverre Sandberg: A regulatory scheme has to be more “stable”/fixed compared to an educational scheme. An educational scheme should be able to change according to the present needs of the laboratory profession. Most important, the goal for an EQA scheme should be to assure that the results produced by the laboratory are suitable to be used for clinical purposes. The schemes should therefore be designed to be flexible, both concerning the number of surveys per year as well as the analytical “problem” addressed.
Mauro Panteghini: If the times are really changing and we need to rethink EQA schemes in a new way, avoiding old contrapositions, we should work to consolidate EQA programs in only one design, which is answering questions about the clinical utility of laboratory results. All the involved stakeholders should be oriented on configuring and implementing schemes that are effective in the post-market verification of in vitro device MS suitability. In addition to meeting specific requirements, such as true value assignment on commutable materials, effective EQA requires that acceptable performance of participating laboratories should be defined by criteria that specify (in numerical terms) the analytical quality required to deliver laboratory test information that would satisfy clinical needs for improving health outcomes. The criteria should only accept the analytical uncertainty for which the risk of medical harm can go undetected. This will permit EQA participants to understand the effect that the quality of laboratory data has on the manner in which it is used in patient care, including the traceability of the calibration and the test result equivalence among laboratories (i.e., result standardization).
I believe that today, all major impediments to realizing the objective to assess performance relative to patient care needs have crumbled, and the time has come to design EQA schemes able to provide this outcome, adding substantial value to the practice of laboratory medicine. Some years ago, the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) started important work in this direction, but, unfortunately, no deliverables were produced to be practically promoted amongst EQA providers. There is indeed an urgent need for an official document describing, with sufficient uniformity, the structure of EQA schemes that may provide information on how laboratory measurement results are clinically suitable. EQAS organizers (and the laboratory professionals too) have to ask themselves what is the ultimate scope of what they are offering. EQA providers should retune, if needed, the design to assess the quality of measurement results defined as suitable for clinical use, independent of the MS type and the possibility to just fulfill the manufacturer’s specifications, and act accordingly.
Greg Miller: A regulatory scheme typically includes a set number of specimens and mailings for all measurands specified in the regulations. Consequently, there is less flexibility, and the concentrations do not challenge limits of the measuring interval or if measurement procedures may be influenced by interfering substances. Aspirational/educational schemes have flexibility to vary the measurands examined in a particular mailing and to prepare samples designed to give information on performance limitations that may be important for patient management. Countries or regions with primarily regulatory programs should reconsider the value and importance of shifting to more educational EQA schemes.
What is the role of EQA in Accreditation?
Michael Spannagl: The German RiliBÄK helps the national accreditation body to apply formal but transparent criteria in the accreditation process in medical laboratories. There is a regular exchange in respective committees, leading to a process of continuous improvement. Regulatory bodies and scientific societies are involved. Together with third-party payers and politicians, the perspective of patient outcomes and cost effectiveness is consequently addressed.
Andrew Quintenz: EQA in an accreditation scheme should not be punitive. As opposed to laboratory regulations, a goal of laboratory accreditation is “not about who the best is, but who has a system of standard procedures with aim to improve the quality and patient safety.”
EQA should be required by accreditation standards along with established EQA goals, requirements to demonstrate evaluation of EQA results, and record of actions taken when goals were not attained, regardless of the EQA schemes pass/fail result. The EQA scheme should not dictate laboratory activity but rather be the information the laboratory uses to guide its own quality improvements.
One of the challenges of accreditation is that when accreditation is adopted by a country in lieu of regulations, this has led to originally aspirational schemes being used to meet accreditation requirements and concern by laboratories of failing these EQA schemes that were designed as aspirational. There has been some pressure on aspirational schemes to adjust APS/criteria to avoid accrediting bodies' citations.
Graham Jones: The accreditation system is designed to support good laboratory practice with the aim of optimizing patient health and safety. EQA is an absolute requirement for a laboratory to ensure high quality results and so making assessment of EQA participation and performance a criterion for accreditation is highly appropriate. The accreditation process, however, must aim to ensure the EQA is being performed, that performance does reflect the performance that would be achieved with patient samples, and that responses to out-of-range EQA are applied, where necessary, to routine testing. With this in mind, the accreditation process should focus as much on the process of performing the EQA testing, and the response to EQA results, as well as on the actual results themselves. Accreditation can also play a role when an appropriate EQA is not available by ensuring other procedures are put in place to confirm analytical quality, such as sample swaps or “average of normals” analysis.
Sverre Sandberg: Accreditation bodies need education in how to interpret EQA results. Very often, they just observe if an EQA result is outside limits of the target value and ask questions about what actions have been taken. Accreditation bodies should take the quality of the EQA scheme into account when they evaluate the results. They should focus on what EQAM has been used, is it commutable, how is the target value established, how many participants are there in the scheme, etc. The national EQA provider should organize meetings with the accreditation bodies where they inform about the different types of EQA schemes. They should underline that, in most cases, having a commutable EQAM with target values set by a reference measurement procedure is more important than circulating a lot of samples each year. When the schemes use noncommutable material, accreditation bodies should investigate how the peer groups are established and how many participants are included in the group. But maybe even more important is that the accreditation bodies should pay much more attention to pre- and post-analytical schemes and should encourage laboratories to participate in those whenever possible. The performance in pre- and post-analytical schemes is as important as the performance in the analytical schemes.
Mauro Panteghini: Accreditation usually is checking medical laboratories according to standards that do not mention the criteria for selecting EQA schemes. They simply ask for the evidence of participation in schemes with a performance that does not affect laboratory accreditation. However, this approach is not evaluating at all the information about the clinical suitability of laboratory measurements. The link between laboratory accreditation and participation in EQA schemes is just formal and, frankly, I do not expect that the accreditation bodies will urge the desired revolution in EQA scheme design.
Greg Miller: Participation in EQA schemes is required for accreditation by ISO 15189 and by most national requirements. The key issue is if the regulations control the frequency of EQA challenges and the grading criteria that can restrict the flexibility of an EQA scheme to provide maximum feedback to a laboratory on how it may be able to improve its measurements.
Nonstandard Abbreviations
EQA, external quality assurance; APS, analytical performance specifications; EQAM, external quality assurance material; MS, measuring system.
Author Contributions
All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.
Authors’ Disclosures or Potential Conflicts of Interest
No authors declared any potential conflicts of interest.
Employment or Leadership
S. Sandberg, President, the European Porphyria Network; A. Quintenz, Bio-Rad Laboratories, Inc., Board of Directors, CLSI; G.R.D. Jones, JCTLM (Joint Committee for Traceability in Laboratory Medicine); W.G. Miller, Clinical Chemistry, AACC; M. Spannagl, Chairman INSTAND, Germany.
Consultant or Advisory Role
A. Quintenz, member, AACC Corporate Advisory Board; G.R.D. Jones, RCPAQAP – Chemical Pathology Advisory Committee.
Stock Ownership
None declared.
Honoraria
S. Sandberg, payment or honoraria for presentations at Recordati Rare Diseases, Bio-Rad, and Technopath.
Research Funding
None declared.
Expert Testimony
None declared.
Patents
None declared.
