Improving the quality of radiological examinations: effectiveness of an internal participatory approach

Abstract

Quality problem or issue

To assess the quality of radiological examinations (REs) and to evaluate the effectiveness of a participatory continuous improvement approach to ensure best practices in a Portuguese hospital imaging department.

Initial assessment

At baseline, we found 232 (10.2%) non-compliances, mostly related to the criteria image centering and framing in chest radiography (CXR), proper use of radiological protection equipment in other conventional RE (CR) and X-ray beam collimation (CXR/CR).

Choice of solution

A baseline and three consecutive evaluations of the RE quality were conducted. Each assessment was followed by participatory focused interventions for improvement.

Implementation

For each evaluation, we selected a random sample (n = 60) of cases for four types of examination (total n = 240 for each assessment, and 960 for the whole project). Both the building of quality criteria and the design of interventions for improvement were participatory, involving the radiology technicians. Estimates of criteria compliance were calculated with 95% confidence intervals. The statistical significance of absolute and relative improvements was tested using one-tail z-tests.

Evaluation

After the intervention, non-compliances decreased to 48 (2.1%). Compliance estimates improved in 25 of 38 criteria assessed, with statistical significance for 5 criteria in CXR and 3 in CR and digestive examination.

Lessons learned

The internal participatory approach enabled the identification of existing quality problems and, by focusing on the more frequent quality defects, was effective in improving the quality of RE.

Quality improvement < Quality management, Hospital care < Setting of care, Teamwork < Human resources, Quality circles < Quality management

Introduction

In the current socio-economic context, with increasing pressure for cost reductions and increases in productivity, the scarcity of resources is one of the triggers for the structuring of institutional programs aimed at improving the quality of health services [1, 2]. In addition to the potential positive economic impact of these programs [1, 2], other factors such as its influence on patient safety, satisfaction [3, 4] and legal regulation [5], among others, were the key drivers of this project.

The legal context varies among countries, but the inclusion of quality metrics in the healthcare system, by government, councils or professional associations, and specifically, in radiology services, is a recent reality still in progress [6, 7]. Quality problems in radiology can affect the effectiveness of diagnosis [3] and, consequently, the normal clinical evolution of patients. Furthermore, efficiency can also be impaired [3], as well as the patients’ safety and satisfaction [3]. The escalation of costs [1, 4] associated with technological development in this field and the need for a rational use of the resources have also contributed to an increasing interest in the implementation of quality programs, which may potentially address all these problems [1, 3].

The technical criteria for quality in radiological examinations (REs) are in general clearly defined [8–10] and well known by most radiology technicians, but the variability of inter- and intra-department compliance as well as the type and frequency of problems remain widely unknown. In radiology, supervision and quality assurance activities are focused mainly on the equipment [1, 4], following legally dictated quality control programs [5]. The use of particular approaches to quality improvement such as Six Sigma [11, 12] and Failure Mode and Effects Analysis [13] has been also reported, as well as other initiatives to improve professional practices [6, 7]. However, studies and reports on the specific magnitude of quality problems, as well as their direct consequences to patients, are still scarce. Ignorance regarding actual quality problems favors the perpetuation of existing errors and a lack of impetus for change.

It seems that the processes of certification or accreditation by external entities are not enough to solve this problem, and that they cannot ensure the adoption of good practices [2, 5, 14]. However, a potentially effective strategy is to conduct structured and systematic participatory quality improvement cycles [14–17], which are characterized by the identification and analysis of specific quality problems, intervention on the more frequent and relevant ones and reevaluation to confirm their resolution. This approach is relatively new to the field of radiology [3, 12, 17], while there are still some doubts regarding its actual effectiveness [18].

This project was designed using an internal and participatory continuous quality improvement approach, by defining and evaluating a system of technical procedures leading to an almost inevitable production of quality REs, in compliance with standardized technical requirements [8–10].

The overall objectives were (i) to assess the quality level of REs, identifying the most frequent and relevant failures and (ii) to implement and analyze the effectiveness of internal participatory evaluation and improvement cycles for continuous quality improvements in radiology.

Methods

Background and study design

This project was implemented in the general radiology department of a medium-sized (~300 beds) general hospital in Lisbon (Portugal), which recently started the process of ISO9001:2008 [19] certification. This process raised awareness about quality issues and the need for an internal approach to improve specific quality problems following the steps and tools of a quality improvement cycle. These include the identification of quality problems and their assessment based on previously defined criteria, the design and implementation of improvement interventions focused on the most relevant and frequent non-compliances, and reassessment of the situation to document the improvement or eventual need for further improvement. According to the research policy at the hospital, this work met criteria for operational improvement activities exempt from ethics review.

To actively involve health professionals from the outset, we used the nominal group technique [20], a participatory group consensus technique, for the selection and prioritization of improvement opportunities. The prioritization criteria were frequency of the problem, influence on patient satisfaction, influence on professional satisfaction, feasibility of measurement and feasibility of resolution. Eventually, the decision was made to focus on the improvement of the technical quality of the REs, for which quality criteria were defined to assess the baseline situation which would drive the appropriate improvement intervention. Quarterly assessments were planned to control and plan improvements.

Development of quality criteria

We built criteria for radiological image quality based on internationally defined and standardized recommendations [8–10]. Other potential quality problems related to the patient, the equipment and the facilities, which may also influence final image quality, were considered and displayed in a cause-and-effect diagram, but were eventually excluded from this study given that they were not amenable to immediate intervention by the professionals who undertook the improvement initiative. The defined quality criteria were assessed for face, content and criterion validity, and piloted for reliability using the kappa coefficient (n = 30) with three evaluators. Eventually, 13 valid and reliable quality criteria of radiological images were defined and accepted as measures of quality for the 4 types of examinations targeted for improvement. They relate to examination identification, patient preparation, patient positioning and collaboration, image acquisition, protection and collimation, and image processing and archiving (Table 1); we built and assessed 13 quality criteria, 8 of which applicable to all examinations, 1 applicable to chest radiography (CXR) and other conventional radiology (CR) examinations and 1 specific for each of the procedures: digestive examination (DE), abdominal/pelvic computed tomography (CT), CXR and CR (see first column of Table 3).

Table 1

Quality criteria developed to evaluate the quality of radiology examinations

Criteria	Exceptions	Clarifications	Reliability (kappa)
A. Examination information
1. Examination information and identification		Full name, process number, patient birth date and/or age, date and time of execution and responsible professional, available and visible	0.78
B. Patient preparation
2. Absence of artifacts	Urgent examinations, inability to remove objects from the patient and/or objects whose removal is medically inadvisable		1
3. Adequate bowel preparation, in DEs	Urgent examinations, compromising the final quality of the examination		1
4. In abdominal or pelvic CT, requiring the ingestion of oral contrast, correct opacification of the digestive tract	Patient incapable of performing the appropriate procedures		1
C. Positioning and collaboration of the patient
5. Patient positioning	Patient incapable of performing the appropriate procedures	According to the performance criteriaa	1
6. Information related to patient positioning		Radiological imaging with information on the position of the patient right/left, anterior/posterior. Oblique must be identified using the position anterior or posterior oblique, right or left	1
7. Correct inspiratory apnea in chest examinations	Patients incapable of performing the appropriate procedures or examinations in expiratory apnea	According to the performance criteriaa	1
D. Image acquisition
8. Adequacy of exposure parameters		Check the adequacy of the exposure parameters by evaluating the value of the exposure index, which appears in the table provided by the supplier	0.86
E. Protection and collimation
9. Proper use of radiological protection equipment	When the use of protection prevents the observation of anatomical regions under study.		0.84
10. X-ray beam collimation in accordance with the requirements of the examination	When manual collimation is not performed, because it is not possible to confirm the intention of meeting the criterion	According to the performance criteriaa	0.86
F. Image processing
11. Image presentation		The image must be presented in the anatomical position and the window and level settings must be in accordance with the objective of the examination	1
12. Image centering and framing		According to the performance criteriaa	1
G. Archive
13. Sending images to the archive (PACS)		The image available in the archive must contain all the information concerning the features and patient data, procedure, image acquisition and post-processing	1

^aPerformance criteria applicable to the examinations in question [8–10].

Population and sample

The population for the quality assessments were all the CR, CXR, DE and CT performed in the 3 months before each assessment at our department over a period of 1 year (four successive assessments). Other radiological techniques were excluded from the study because of the complexity and variability of the specifications for some of the criteria used (such as positioning and collimation).

In each of the 4 evaluations, 4 random samples of 60 cases for each type of examination were selected from the list of examinations for the period (quarter), making a total n = 240 for each evaluation and 960 for the whole project.

Evaluation procedures

The quality evaluations were quarterly and retrospectively assessed on the four samples of radiological images accessible using the equipment where the examination was performed, except for post-processing and image archiving, which needed further verification in the picture archiving and communication system (PACS). The evaluations were performed by three trained radiology technicians (more than 10 years of experience) who were members of the project team.

Multifaceted intervention

Data from the baseline evaluation were analyzed and used in the participatory planning of intervention focused on the more frequent (‘few vital’) quality defects highlighted using a Pareto diagram on non-compliance. The multifaceted intervention proposed by the working group was organized into three groups represented in an affinity diagram (Table 2) [21].

Table 2

Affinity diagram with the interventions proposed by the working group

1	Training of employees (radiologists, technicians, assistants and administrative personnel)	To raise awareness on the radiological image quality criteria and the importance of its compliance
2	Definition of service procedures	Work instructions, under the responsibility and commitment of the technical and medical leadership of the department, to achieve quality radiological images in all REs
3	Presentation and discussion of the results of the assessments (feedback)	To periodically promote and raise awareness on required actions, and to monitor improvement

A storyboard was created and periodically updated, including the study schedule and responsibility/accountability (Gantt diagram), frequency, dates and results of the assessments (feedback), the defined corrective actions and people responsible for the implementation.

Interventions after the second and third reviews were adjusted to and focused on the respective priority criteria with more frequent non-compliance highlighted by the Pareto diagram, and were discussed and disclosed in the same way.

Data analysis

In each evaluation and for each of the four types of examinations assessed, estimates of compliance were calculated with 95% confidence intervals. To analyze the effect of the interventions, absolute (difference in criteria compliance) and relative improvements (difference in criteria compliance as percentage of the room for improvement at baseline) [22] were estimated. Differences in compliance before and after the intervention were tested for statistical significance by calculating the z-value (one-tail) for the alternative hypothesis of existence of improvement, which is accepted when the probability of the null hypothesis is P < 0.05. Non-compliance data from subsequent assessments are analyzed using a before-and-after Pareto chart, in order to visualize improvement and to prioritize quality criteria to target for further improvement after each evaluation. The before-and-after Pareto chart [22] compares the charts representing non-compliances before and after interventions. Charts have two vertical axes: the left one for the absolute number of non-compliances (with a scale from zero to the total number of non-compliances found in the assessment), and the right one for the corresponding relative frequencies calculated as percentage over the total number of non-compliances; the horizontal axis is used for a bar chart of the different criteria, ordered from more to less frequent incidents of non-compliance. The curves drawn in the charts represent the cumulative frequency of non-compliances and are used as a reference to point out the criteria responsible for the majority of quality failures, which should be prioritized in the design of the improvement intervention. The improvement achieved after the intervention is represented in the before-and-after Pareto chart by the area above the total non-compliances in the latest assessment. This area will increase as the number of non-compliances decreases, as it is proportional to the difference in non-compliances between the evaluations before and after the interventions. At the same time, each individual Pareto chart underscores the criteria which should be prioritized for further improvement.

Results

Baseline level of quality and improvement priorities

At baseline, criteria compliance ranged from 31% (19/60) to 100% (60/60). Most of the quality criteria (33 out of 38) had compliance rates above 80% (48/60), and some of them (absence of artifacts, positioning, positioning-related information, exposure parameters, presentation of the image and archive on PACS) showed above 90% (54/60) compliance in all examination types evaluated.

In general, abdominal/pelvic CT had the highest compliance rates, with 100% compliance in five of the nine applicable criteria, and none of them below 80%. The greatest variability (inter-quartile range) was seen in CR examinations (9.3%), followed by CXR (7%). The lowest criteria compliance (<50%) was found in these two types of REs. A detailed description of the initial level of criteria compliance for each type of examination is shown in Table 3.

Table 3

Comparison between first and fourth evaluations—absolute, relative improvement and statistical significance

Criteria (preceding number corresponds to the one assigned in Table 1)	First evaluation	Fourth evaluation	Absolute improvement	Relative improvement	Statistical significance
	p1 (CI 95%)	p4 (CI 95%)	p4–p1	$\frac{p4-p1}{100-p1}$	P-value
CXR	80.7 (31.7–96.9)	97.5 (95–100)	16.8	87	0.023
1. Information and identification	88 (80–96)	95 (94–96)	7	58	NS
2. Artifact absence	96 (92–100)	100	4	100	NS
5. Positioning	92 (85–99)	97 (96–97)	5	63	NS
6. Positioning-related information	92 (87–97)	99 (98–99)	7	88	0048
7. Inspiratory apnea	87 (78–95)	95 (94–96)	8	62	NS
8. Exposure parameters	95 (90–100)	97 (96–97)	2	40	NS
10. Beam collimation	40 (28–52)	95 (94–96)	55	92	0.001
11. Image presentation	90 (82–98)	99 (98–99)	9	90	0.026
12. Centering and framing	32 (20–44)	100	67	99	0.000
13. PACS archiving	95 (90–100)	100	5	100	0.039
CR	87.3 (33.3–100)	97.5 (91.7–100)	10.2	80.3	0.013
1. Information and identification	88(80–96)	97 (95–98)	9	75	0.04
2. Artifact absence	98 (95–100)	98 (97–99)	0	0	NS
5. Positioning	100	97 (95–98)	−3	0	NS
6. Positioning-related information	96 (92–100)	100	4	100	NS
8. Exposure parameters	96 (92–100)	100	4	100	NS
9. Radiological protection	33 (21–45)	92 (89–94)	59	88	0.000
10. Beam collimation	60 (48–72)	94 (91–96)	34	85	0.000
11. Image presentation	96 (92–100)	100	4	100	NS
12. Centering and framing	100	98 (97–99)	−2	0	NS
13. PACS archiving	100	100	0	0	NS
DE	95.6 (88.3–100)	99.4 (96.7–100)	3.8	86.4	0.021
1. Information and identification	88 (80–96)	100	12	100	0.003
2. Artifact absence	98 (95–100)	100	2	100	NS
3. Bowel preparation	91 (85–97)	98 (95–100)	7	78	0.048
5. Positioning	98 (95–100)	97 (93–100)	−1	−50	NS
6. Positioning-related information	91 (85–97)	100	9	100	0.011
8. Exposure parameters	96 (92–100)	100	4	100	NS
11. Image presentation	96 (92–100)	100	4	100	NS
12. Centering and framing	95 (90–100)	100	5	100	NS
13. PACS archiving	100	100	0	0	NS
Abdominal-pelvic CT	98.1 (88.3–100)	97.8 (93.3–100)	−0.3	−15.8	0
1. Information and identification	96 (92–100)	96 (95–97)	0	0	NS
2. Artifact absence	98 (95–100)	94 (92–95)	−4	−20	NS
4. Digestive tube opacification	88 (80–96)	94 (92–95)	6	50	NS
5. Positioning	100	100	0	0	NS
6. Positioning-related information	100	100	0	0	NS
8. Exposure parameters	100	100	0	0	NS
11. Image presentation	100	100	0	0	NS
12. Centering and framing	100	100	0	0	NS
13. PACS archiving	96 (92–100)	96 (92–100)	0	0	NS
Total	89.8 (89.5–90.1)	97.9 (97.3–98.5)	8.1	78.4	0.000

p1 and p4: compliance rates in the first and fourth evaluations, respectively. 95% CI, confidence interval 95%; NS, not significant (P > 0.05).

Considering the improvement opportunities, ~1 in 10 individual criteria showed quality failures, with a total of 232 non-compliances (10.2% of maximum possible non-compliances). According to the Pareto chart (first chart in Fig. 1), the criteria that should be prioritized for improvement concerned the dimensions image processing and patient safety. The most frequent problem was image centering and framing in CXR, followed by the proper use of radiological protection equipment in CR, and X-ray beam collimation, both in CXR and CR. Sixty-three percent of the basal non-compliances corresponded to these criteria.

Figure 1

Pareto charts present absolute frequencies (AF), relative frequencies (RF) and accumulation of failures in criteria compliance in the four evaluations. The bars represent AF of failures in each criterion and the lines represent the cumulative frequency. The total number of non-compliances in the first evaluation sets the reference scale for subsequent assessments and their improvement areas. The reference numbers correspond to the numbering of the criteria in Table 1.

Effectiveness of the evaluation and improvement cycles

Estimates of absolute and relative improvements between the initial and final evaluations, as well as their statistical significance, showed the positive effect of this project (Table 3). The absolute improvement (AI) in compliance between the first and last evaluations was of 16.8% for CXR, 10.2% for CR, 3.8% for DE and none (−0.3%) for abdominal/pelvic CT, which showed the highest (close to 100%) compliance at baseline. The relative improvement figures provided an even clearer picture of the project's impact on the quality of the REs, as it considered baseline estimates and their consequent potential room for improvement. The data showed an absolute overall compliance improvement of 8.1%, and overall relative improvement (share of the initial potential room for improvement) of 78.4%. By type of examination, overall compliances in the final evaluation were 97.5% for CXR, 97.5% for CR, 99.4% for DE and 97.8% for CT.

As for the individual criteria, increased estimates of compliance were found in 25 of the 38 criteria evaluated in the 4 types of examination. The improvement was statistically significant (P < 0.05) for 11 of the 25 criteria, 5 in CXR and 3 each in CR and DE. Important improvements were achieved on those criteria with lower compliance (below 85%) at baseline: image centering and framing (CXR) (AI = 68%; P = 0.000), radiological protection equipment (CR) (AI = 58%, P = 0.000) and X-ray beam collimation (CXR) (AI = 55%, P = 0.001). In the final assessment, all criteria showed a level of compliance higher than 88%, and 19 of the 38 criteria evaluated had 100% compliance.

The benefits of this project were also made explicit by the reduction in non-compliances. This was shown by the evolution of the Pareto charts in the four consecutive evaluations (Fig. 1). Total non-compliance in the samples continuously decreased from 232 (10.2% of the maximum possible number of non-compliances) to 48 (2.1%). After the intervention, the criteria responsible for the majority of failures showed a significant improvement (P < 0.01), although three of them remained in the group of ‘vital few’ in all evaluations. The criterion on centering and framing of CXR was no longer included in this troubled group in the last evaluation, with non-compliance reduced to zero. In the remaining criteria, non-compliance was reduced by >50% in the second assessment and continued to register reductions accomplishing, in the last evaluation, values below 15% of those initially recorded.

Four criteria showed lower (but statistically non-significant) estimates of compliance at the end (artifacts on CT, positioning on DE and CR and centering and framing in CR). However, it is important to note that two of them had 100% initial compliance, the other two close to 100% and all maintained compliance levels above 93%. Also, the criteria with no improvement (0%) had initially and maintained compliance greater than 96%.

Discussion

This study contributes to the understanding of possible quality problems in REs and provides information on the potential of participatory and continuous evaluation cycles as a tool for changing professional behavior and ensuring good practices in a radiology department. The study describes an effective way to solve problems related to human behavior in radiology. The major and consistent improvement throughout the project (Fig. 1), particularly on the most problematic criteria, argues positively for the effectiveness of this approach. We believe that the positive results could be at least partially attributed to the participatory nature of the internally decided project. The internal participatory approach was shown to be effective in early studies supported by WHO [23, 24], incorporated some years later to the methodology proposed and widely used by the Institute for Health Improvement [25, 26], and with successful application in a variety of settings both in developed [14–16, 27] and developing [17, 28] countries. Overall, it seems that professionalism and intrinsic motivation are key, and that training health professionals on quality improvement skills emerges once again as a desirable objective in all health systems [29].

In general, the results showed a relatively high initial quality level (with 89.8% compliance), particularly in CT. It is likely that this result is due to multiple verifications performed by a number of professionals with requirements for compliance between each other, as well as the introduction of software with mandatory filling fields which precludes the occurrence of some of these errors. The quality level verified in CT apparently justifies a lower priority for improvement on these aspects.

However, the explicit assessment identified relevant flaws (i.e. beam collimation and adequate radiological protection) which could remain unnoticed and overlooked. These problems were mainly concentrated in CR and CXR, a result consistent with previous studies [30–32]. The fact that these types of examinations are the most frequent in many radiology departments underscores the importance of their explicit measurement and control to overcome a higher probability of failure, perhaps associated with repetitive work and increased workload.

Besides beam collimation (in CR and CXR) and the proper use of radiological protection equipment (in CR), the lowest compliance at baseline was in image centering and framing (in CXR). All of them were preventable failures when uncovered and explicitly addressed by the radiological team, as evidenced by the achieved improvement (78%). The absence of active involvement in improvement puts patients and the healthcare providers itself at risk of serious consequences such as excessive exposure to radiation, the repetition of tests due to poor visualization, diagnostic errors and unnecessarily increased costs [31, 32].

Before-and-after designs without a control group have been pointed out as one of the methodological limitations of quality improvement studies [18]. This could also be considered a limitation for our conclusions. However, the continuous nature of the improvement, not just comparing measurements before and after a given intervention, suggests that our results may have not been achieved without the intervention as it was planned. Our project underscores the importance of internal communication and participatory teamwork to mobilize intrinsic motivation for improvement. The right choice for a project that addresses the mistakes of the team [15], without individualizing, and not intending to point out responsibilities but rather the causes and interventions which could lead to the improvement of the entire group, may have been decisive in achieving a degree of support and commitment higher than expected. The quarterly reviews allowed us to focus subsequent interventions on the correction of more problematic criteria. This continuous audit and feedback to the professionals [15, 16], included in a multifaceted intervention, may have also contributed to the effectiveness of the improvement cycle methodology, achieving significantly better compliance levels in the most problematic criteria.

There are not many published studies on improving the quality of radiological services, but our results compare favorably with the improvements achieved in other studies we found [17, 30, 31], where the reported relative improvements were 33% [17, 30] and 60% [26], much lower than the relative improvement of 78.4% in our project. Additionally, the greater availability of resources and time described in other studies [13, 33] further enhances the relevance of the improvement we could achieve.

On the relatively negative side, it is worth underlining that some criteria which affect directly patient safety [8, 10], such as patient protection and beam collimation, still account for a relevant proportion of quality failures, with <95% compliance, justifying a more specific approach for the achievement of better results, which other studies have shown to be achievable [17].

Some possible limitations of our study data include the potential bias in identifying some criteria exceptions defined at the beginning, examinations hampered by the difficulty of patient collaboration and equipment problems. These may contribute to overestimated non-compliance. Also, the results may be considered contextual. However, we think that quality improvement initiatives in different contexts may consider reproducing the methodology we used, taking advantage of their proven ability to significantly reduce quality problems in REs. Eventually, this could be tested in a larger number of centers with a control group of institutions [18]. Additionally, further monitoring may be needed to control the stability of the improvement over time.

In summary, although we agree in that there is no ‘magic bullet’ for quality improvement [34], the results of this study demonstrate the importance of the participation of the staff to raise the quality culture and achieve consistent improvements. We believe that the achieved success can be attributed to (i) the internal initiative for prioritizing the topic for improvement, and for the construction and validation of the evaluation criteria, contributing to the necessary professionals’ prominence and ownership; (ii) continuous feedback of the results of the assessments [15, 16], followed by participatory design of the intervention for improvement and (iii) the prioritization of the interventions to improve focus on the most problematic criteria, favoring the effectiveness of improvement efforts. All these principles may also be applied to new improvement projects, targeting other techniques or types of examinations.