Refinement of indicators and criteria in a quality tool for assessing quality in primary care in Canada: a Delphi Panel study

Abstract

Purpose.

Primary care is the cornerstone of the health care system and increasingly countries are developing indicators for assessing quality in primary care practices. The ‘Quality Tool’, developed in Ontario, Canada, provides a framework for assessing practices and consists of indicators and criteria. The purpose of this study was to validate the indicators and simplify the Quality Tool.

Methods.

This study involved a systematic comparison of indicators in the Quality Tool with those in other local and international tools to determine common indicators to include as valid in the Quality Tool. A Delphi process was used to help reach consensus for inclusion of any indicators that were not included in the comparison exercise.

Setting.

Primary care in Ontario, Canada.

Subjects.

Key informants were those with known expertise and experience in quality assessment in primary care.

Main outcome.

Validated set of indicators for inclusion in an updated Quality Tool.

Results.

Twenty-three stakeholders participated in the Delphi panel. Forty-four indicators were included as valid after the systematic comparison of similar indicators in other assessment tools. Of the 63 indicators brought to the Delphi panel, 37 were included as valid, 15 were excluded and 11 became criteria for other included indicators.

Conclusions.

The study resulted in a set of 81 validated primary care indicators. The validation of the indicators provided a strong foundation for the next version of the Quality Tool and may be used for quality assessment in primary care.

Delphi technique, family practice, general practice, primary health care, quality improvement, quality indicators.

Introduction

Most health care is provided in family practice settings where the prevention, diagnosis, management and outcomes of care depend on the quality of care provided(1). Decision makers, funders and clinicians themselves have been challenged to provide an environment that fosters the best possible care(2). In Canada, the Quality in Family Practice Program (Quality Program) was developed in 2005, to assess the quality of care provided in family practice settings, through a voluntary accreditation program(3). Internationally, many countries have adopted accreditation programs to promote quality improvement. These programs externally assess and monitor standards or indicators of quality in primary care. General practice accreditation programs have been widely implemented in Australia and New Zealand(4,5), in some countries in Europe(6–8) and are proposed in the UK (9). Although not considered an accreditation program, the Quality and Outcomes Framework in the UK was initiated in 2003 to improve patient access to care and patient health through a pay-for-performance system(10). Systematic reviews of studies of accreditation have reported limited and inconclusive findings about the effects of accreditation(11).

There have been criticisms that these quality programs are too prescriptive, do not take into account the complexity of practice, drive clinical behaviour for financial reward, promote evidence rather than clinical judgement and reduce continuity of care(12,13). There is also evidence to show that these initiatives have had a positive effect on chronic conditions(13). In each program mentioned above, assessment tools have been developed to measure the quality of performance of the practice through external assessment(14–17). One way to address these concerns is to ensure that the authoritative standards and indicators to formally assess practices are comprehensive, meaningful and foster continuity of care.

The Quality program, developed a comprehensive quality assessment tool, called the Quality Tool in 2005 as a practical guide for assessment of family practices(3). At the same time in Canada, the Canadian Institute of Health Information (CIHI) undertook a collaborative consensus process with experts to develop primary health care indicators in 2006(18).

The Quality Tool was based on New Zealand’s ‘Aiming for Excellence’ tool that had its roots in Australia’s Standards for General Practice(4,5). The Quality Tool was refined for the Canadian context, and attempted to address the criticisms about quality programs by developing indicators that incorporated the four principles of family medicine(19,20). In both the pilot and the field test of the Quality Program and Tool, participants found the implementation of the Quality Program and Quality Tool to be a positive, useful and valuable experience and suggested simplifying the tool(21,22).

The Quality Tool indicators, while based on other established programs that had been subjected to peer-review or validation had not undergone similar study. The purpose of this project was to validate a set of indicators to be included for the next version of the Quality Tool and simplify the tool.

Methods

There were three main objectives to this validation study:

• 1. To include as face valid the indicators in the Quality Tool that were the same or similar to indicators in other validated/consensus-derived primary care quality assessment tools, both in Canada and internationally.

• 2. To list unique indicators from the Quality Tool and other Canadian and international tools examined, and new indicators from emerging programs in Canada.

• 3. To perform a Delphi panel exercise on the unique and new indicators, and include the indicators that met our definition of consensus, as content valid, and remove the indicators that fell below this cut point.

Including as face valid the same or similar indicators from other tools

The Quality Tool had 80 indicators and over 300 criteria. Indicators represented standards of practice performance. Criteria were processes and interpretations that could be counted or measured to assess the performance of the indicator(23). For example indicator B.5.6 ‘[t]he practice ensures that only authorised people have access to prescription medications’, has an associated criterion B.5.6.6 ‘[p]rescription pads are not accessible to unauthorised persons’ and an interpretation ‘[t]here are no prescription pads left in the examining rooms where patients can access them’ (23).

First, we listed and compared the Quality Tool indicators and criteria with indicators from three validated/consensus-derived tools [European Practice Assessment (EPA) (24) Quality and Outcomes Framework (QOF) (25), and CIHI Pan-Canadian Primary Care Indicators] (18). The Australia and New Zealand tools were not examined as the Quality Tool was mostly a derivative of these tools (4,5). Three members of the research team with indicator expertise reviewed the indicators for face validity. All disagreements were resolved and agreement reached on indicators that were face valid.

We use the terms face validity and content validity as technical descriptions that the judgements were reasonable(26). We included an indicator in the Quality Tool as face valid if, on the face of it, it was the same or similar to an indicator in any of the other tools examined. Content validity is a judgement that the included indicators were appropriate for the intended purpose. An example of a face valid indicator in the Quality Tool is, ‘There is a system to manage patient test results and medical reports’ and the indicator in the EPA tool, ‘There is a procedure for managing patient information regarding outgoing requests (tests, referrals, and requests from third parties)’(3). Occasionally a criterion in the Quality Tool aligned with an indicator in another tool examined, in which case the Quality Tool indicator was considered face valid.

Identifying and listing unique indicators

The remaining indicators that were only found in the Quality Tool or in one of the other tools were listed as ‘unique’ indicators for the Delphi panel. For completeness, in addition to the unique indicators, we added some ‘new’ indicators for the Delphi panel consideration. These new indicators were chosen because they had recently prompted funding of special primary care programs in Canada: colorectal cancer screening (27), access to a family doctor (28) and care of older adults(29).

The Delphi method

We used a Delphi method to further validate the unique and new indicators(30). We chose a Delphi method as a pragmatic group facilitation technique to help guide the participating individuals towards concurrance(31). We used a quasi-anonymity method, in that the respondents were known to the researchers and to each other, but their survey response judgements and opinions remained anonymous(32). A similar method was used in developing the standards for the Quality and Outcomes framework(33). In our process, we did not ask open ended questions in the survey. Instead we provided pre-existing indicators for rating and invited comments. This may have limited the available options, but enabled the participants to focus on the task and kept them engaged(34).

Development of indicator inclusion principles

Prior to convening the Delphi panel, a small group of key informants, called the pre-Delphi group, developed a set of inclusion principles from a comprehensive review of indicator selection criteria used by other reporting bodies and projects that could assist the Delphi panel members in rating the value of an indicator for the Quality Tool (35,36).

Identifying key informants and Research Ethics Approval

Our team identified, invited and sought consent from key informants agreeing to participate. Our pre-Delphi group had clinical, administrator and patient expertise from those involved in the development and testing of the Quality Tool and were selected from a list developed by the investigator team(21,22). The pre-Delphi group became Delphi panelists. In addition, other family physician leaders and a pharmacist, who were academics and decision makers, with expertise in quality improvement, were selected and invited to take part in the Delphi panel. We purposefully included an over-representation of family physicians, as family physicians are by far the most representative of clinicians in family practice, work in solo, group, rural and urban locations, play a leadership role in quality improvement and own and manage family practices in most settings in Canada. Our team felt that the voice of other primary care providers was well represented in the pre-Delphi group with the addition of the pharmacist. Delphi panel participants were provided with a small honorarium for taking part in the study. Research Ethics Board Approval [08-369] was received from McMaster University.

Consensus

Although the literature provides few clear guidelines for measuring consensus, numerical consensus is commonly used(37). Our rating level cut-points for ‘included’, ‘excluded’ and ‘undecided’ indicators were loosely based on a published report of a Delphi process in primary care(38). For our purposes we chose to define consensus to participants as concurrence or agreement on which indicator should be included, excluded or undecided. An indicator was included as content valid if 80% or more agreed, excluded if 50% or less agreed and undecided if between 51% and 79% agreed.

Delphi process

The Delphi panel was convened for ~4 months (November 2008 to March 2009). The Delphi process involved two rounds of on-line surveys(39) with teleconference calls following each survey to discuss findings, and one final face-to-face meeting to resolve any outstanding issues. In the first round, panelists rated and commented on all of the unique and new indicators written as they appeared in the tool from which they originated. A report was produced of the indicators that were included, excluded and undecided. In the second round, participants were encouraged to review their ratings of the undecided indicators arising from the first round and a second report produced. The results of the second round undecided indicators were brought to the face-to-face meeting. At the face-to-face meeting, participants discussed each remaining undecided indicator and then voted in a paper survey either yes or no on whether to include this indicator in the revised Quality Tool. Paper surveys were collected at the meeting or electronically following the meeting. Delphi participants who could not attend the face-to-face meeting were invited to complete the survey electronically. At this meeting, in addition to Delphi panel participants, three experts in quality assessment from Germany, Australia and New Zealand contributed to the discussion about the undecided indicators. Following the each round and the face-to-face meeting, some of the undecided indicators were neither included nor excluded but recommended to become criteria in the revised Quality Tool, thereby contributing to reducing the number of indicators and simplifying the tool.

Analysis of inclusion principles

Participants were asked to rate each indicator, in each round of on-line surveys, against the set of four inclusion principles using a five-point Likert scale with the anchors strongly disagree to strongly agree. These inclusion principles were presented to Delphi panelists to provide context to their decision-making. After reflecting on the inclusion principles and rating them, participants were then asked to indicate whether the indicator should be included by answering yes or no to the statement, ‘This indicator should be part of the revised Quality Tool’. The yes answers were combined to form the percentage used for consensus in each round.

An analysis of the ratings of each of the inclusion principles was completed to determine if the decision to include or not include the indicator was significantly influenced by the principles. Round 1 of the Delphi was chosen for analysis, as Round 2, and the face-to-face meeting included only the undecided indicators. Using an independent sample t-test, the mean scores of the four inclusion principles were compared for included (80% voted to include) and not included (all others) indicators in the first round. A P value (two-sided) of less than 0.05 was considered statistically significant. The distributions of the ratings of the inclusion principles were also visually compared to determine if the differences were really significant.

Results

Including the same or similar indicators as face valid and identifying and listing unique and new indicators

Forty-four of the 80 indicators in the Quality Tool were included as face valid because they were in at least one of the other tools examined. (Table 1) This resulted in 63 unique and new indicators (36 from Quality Tool and 27 from other tools and newly funded programs) that were brought to the first Delphi process.

Table 1.

Indicators included by comparing the indicators and criteria in the Quality Tool with the indicators in CIHI, EPA, and QOF prior to the Delphi process (although some criteria in the Quality Tool matched indicators in the other tools, they became indicators to be validated in the Delphi process)

Indicator No. (n = 44)	(Indicator No.) Description in Quality Toola (3)	(Domain) Description in EPA (24)	(Indicator No.) Description in CIHIb (18)	(Indicator Ref.) Description in QOFb (25)
1	(A.1.3) The practice encourages patient suggestions and feedback into service planning	(QUALITY & SAFETY: Patient Perspective) The practice has a suggestion box for patients on a clearly visible place		(PE 2 Patient surveys 1) The practice will have undertaken an approved patient survey each year
2	(A.2.1) Information about practice services is available for patients	(INFORMATION: Information for patients about non-clinical issues) The practice has a practice information sheet & If the practice has an information sheet, it contains relevant information
3	(A.2.2) The practice makes provision to ensure patients are able to access 24-hour care, 7 days a week	(INFRASTRUCTURE: Accessibility and availability) The practice has a clear phone message when phoning the practice out of hours, or direct connection to the deputizing service/ own GP	(31) Average number of extended hours (beyond 9:00 a.m. to 5:00 p.m., Monday to Friday), provided by PHC organizations per month, by PHC organization.	(Records 3) The practice has a system for transferring and acting on information about patients seen by other doctors out of hours
4	(A.2.4) The practice uses a system that assists the practice team to identify and provide an appropriate response to emergencies/urgent medical conditions	(INFRASTRUCTURE: Accessibility and availability) Reception staff have been trained to recognize and respond appropriately to urgent medical matters		(Education 1) There is a record of all practice-employed clinical staff having attended training/updating in basic life support skills in the preceding 18 months
5	(A.2.5) The practice team ensures that patients are provided with information to enable them to make informed decisions about their care		(76) % of PHC clients/patients, 18 years and over, who were involved in clinical decision-making regarding their health, with their regular PHC provider, over the past 12 months.
6	(A.2.6) The practice provides educational information on health promotion and disease prevention to patients		(25) % of PHC clients/patients, 18 years and over, with a chronic health condition(s), whose PHC organization provided them with resources to support self-management or self-help groups.
7	(A.2.7) Initial and repeat prescribing in the absence of a visit is accurate, appropriate and timely			(Records 9) For repeat medicines, an indication for the drug can be identified in the records (for drugs added to the repeat prescription with effect from 1 April 2004). Minimum Standard 80%
8	(A.3.1) Patients can easily access the practice using the telephone system			(Information 7) Patients are able to access a receptionist via telephone and face to face in the practice, for at least 45 hours over 5 days, Monday to Friday, except where agreed with the primary care organization
9	(A.3.2) Patients can access the practice for advice or appointments appropriately	(INFRASTRUCTURE: Accessibility and availability) The practice has a booking system	(33) % of PHC clients/patients, 18 years and over, who are satisfied with wait time to obtain an appointment with their regular PHC provider for an emergent but minor health problem.	(PE 7 Patient experience of access) The percentage of patients who, in the appropriate national survey, indicate that they were able to obtain a consultation with a GP (in England) or appropriate health care professional (in Scotland, Wales and NI) within two working days (in Wales this will be within 24 hours).
10	(A.3.3) Patient visits are effective and efficient		(27) % of PHC clients/patients, 18 years and over, with a chronic condition(s), who had sufficient time in most visits to confide their health-related feelings, fears and concerns to their PHC provider.	(PE 1 Length of Consultations) The length of routine booked appointments with the doctors in the practice is not less than 10 minutes. (If the practice routinely sees extras during booked surgeries, then the average booked consultation length should allow for the average number of extras seen in a surgery session. If the extras are seen at the end, then it is not necessary to make this adjustment).
11	(B.5.1) The practice waiting area is comfortable and sufficient to accommodate patients and their family members who wait for services	(INFRASTRUCTURE: Premises) There is sufficient seating in the waiting room
12	(B.5.3) The practice has appropriate disinfection and sterilization facilities available for infection control and follows	(QUALITY AND SAFETY: Safety of the staff and patients) The practice has procedures for infection control
13	(B.5.7) The practice demonstrates a commitment to the Workplace Safety and Insurance Act 1997		(90) % of PHC providers who had a workplace related injury over the past 12 months, by type of PHC provider.
14	(B.6.1) Medical equipment and resources are appropriate, available and maintained	(QUALITY AND SAFETY: Quality policy) The practice has lists/ inventories of medical equipment and drugs		(Management 7) The practice has systems in place to ensure regular and appropriate inspection, calibration, maintenance and replacement of equipment including:
				• A defined responsible person
				• Clear recording
				• Systematic pre-planned schedules
				• Reporting of faults
15	(C.8.1) Smoking cessation	(INFORMATION: Clinical data) The medical record contains smoking status	(13) Tobacco use(20) % of population, 12 years and over, who are current smokers.	(Information 5) The practice supports smokers in stopping smoking by a strategy which includes providing literature and offering appropriate therapy
16	(C.8.2) Diabetes mellitus		(7) % of PHC organizations who currently have specific programs for PHC clients/patients with specific chronic conditions.	(DM 19) The practice can produce a register of all patients aged 17 years and over with diabetes mellitus, which specifies whether the patient has Type 1 or Type 2 diabetes
17	(C.8.3) Hypertension		(54) % of PHC clients/patients, 18 years and over, who had their blood pressure measured within the past 24 months.	(BP 1) The practice can produce a register of patients with established hypertension
18	(C.8.4) Stroke or transient ischaemic attacks (TIAs)			(Stroke 1) The practice can produce a register of patients with Stroke or TIA
19	(C.8.5) Secondary prevention in coronary heart disease (CHD)		(61) % of PHC clients/patients, 18 years and over, with established CAD and elevated LDL-C (i.e. >2.5 mmol/l) who were offered lifestyle advice and/or lipid lowering medication.	(CHD 1) The practice can produce a register of patients with coronary heart disease
20	(C.8.6) Anti-coagulation medication			(AF 3) The percentage of patients with atrial fibrillation who are currently treated with anti-coagulation drug therapy or an anti-platelet therapy
21	(C.8.7) Mental Health		(64) % of PHC clients/patients, 18 years and over, with depression who were offered treatment (pharmacological and/or non-pharmacological) or referral to a mental health provider.	(MH 8) The practice can produce a register of people with schizophrenia, bipolar disorder and other psychoses
22	(C.8.10) Asthma care		(37) % of PHC clients/patients, ages 6 to 55 years, with asthma who visited the emergency department in the past 12 months.	(Asthma 1) The practice can produce a register of patients with asthma, excluding patients with asthma who have been prescribed no asthma-related drugs in the previous 12 months
23	(C.8.11) Chronic obstructive pulmonary disease (COPD)			(COPD 1) Practice can produce a register of patients with COPD
24	(C.8.12) Epilepsy			(Epilepsy 5) The practice can produce a register of patients aged 18 and over receiving drug treatment for epilepsy
25	(C.8.13) Hypothyroidism			(Thyroid 1) The practice can produce a register of patients with hypothyroidism
26	(C.8.14) Cancer			(Cancer 1) The practice can produce a register of all cancer patients defined as a ‘register of patients with a diagnosis of cancer excluding non-melanotic skin cancers from 1 April 2003
27	(C.9.1) The practice provides comprehensive care for children			(CHS 1) Child development checks are offered at intervals that are consistent with national guidelines and policy
28	(C.9.4) The practice provides comprehensive care for cervical screening		(50) % of women PHC clients/patients, ages18 to 69 years, who received papanicolaou smear within the past 3 years.	(CS 7) The practice has a protocol that is in line with national guidance and practice for the management of cervical screening, which includes staff training, management of patient call/ recall, exception reporting and the regular monitoring of inadequate smear rates
29	(C.9.6) The practice provides comprehensive care for breast cancer screening		(49) % of women PHC clients/patients, ages 50 to 69, who received mammography and clinical breast examination within the past 24 months.
30	(C.9.9) The practice provides comprehensive care for adult patients		(12) Non-urgent routine care (e.g. well care (baby, child, woman and/or man)
31	(C.9.10) The practice provides palliative end-of-life care to patients		(12) End-of-life care	(PC 3) The practice has a complete register available of all patients in need of palliative care/support irrespective of age.
32	(D.11.1) Medical records and documents are stored or filed safely	(INFORMATION: Confidentiality) Patient medical records or other files containing patient information are not stored or left visible in places where patients could gain access to them or read them.
33	(D.11.2) There is a system to manage patient test results and medical reports	(INFORMATION: Management of external patient data) The practice has a procedure for managing external patient data	(83) % of PHC FPs/GPs/NPs who repeated tests because findings were unavailable over the past month.
34	(D.11.3) Registration of new patients and transfer of medical records is patient-friendly			(Information 4) If a patient is removed from a practice’s list, the practice provides an explanation of the reasons in writing to the patient and information on how to find a new practice, unless it is perceived that such an action would result in a violent response by the patient
				(Records 19) 80% of newly registered patients have had their notes summarized within 8 weeks of receipt by the practice
35	(D.12.1) The practice provides care that is integrated with other care agencies and community services to improve individual patient care		(80) % of PHC organizations who currently have collaborative care arrangements with provider organizations beyond the health care sector (e.g. housing, justice, police, education)
36	(D.12.2) The practice provides services to patients and families to meet patients with complex needs (high users, regular emergency users, patients often in crisis and patients with multiple problems)		(10) % of PHC organizations who currently provide specialized programs for vulnerable/special needs population groups.
37	(D.13.1) All members of the practice team are qualified or trained for their position	(PEOPLE: Personnel) The practice checks certificates when a new employee is appointed
38	(D.13.2) All members of the practice team have contracts and current job description, and management structures are in place	(PEOPLE: Personnel) Staff have a signed contract & Staff have a job description		(Management 10) There is a written procedures manual that includes staff employment policies including equal opportunities, bullying and harassment and sickness absence (including illegal drugs, alcohol and stress), to which staff have access
39	(E.15.1) The practice promotes continuous quality improvement (CQI)		(69) % of PHC organizations who implemented at least one or more changes in clinical practice as a result of quality improvement initiatives over the past 12 months.
40	(E.15.2) The practice promotes continuing professional development (CPD)	(PEOPLE: Education and training) Staff has been on training course related to their work in the past 12 months	(72) % of PHC providers and support staff whose PHC organization provided them with support to participate in continuing professional development within the past 12 months, by type of PHC provider and support staff.
41	(E.15.3) The practice has a morbidity and mortality management system to address serious or potentially serious practice problems (adverse incidents, near misses, etc.)		(67) % of PHC providers whose PHC organization has processes and structures in place to support a non-punitive approach to medication incident reduction.	(Education 10) The practice has undertaken a minimum of three significant event reviews within the last year
42	(E.15.4) A range of educational resources and materials are available for reference purposes to members of the practice	(INFORMATION: Information for staff) Every GP has access to the internet & Every GP has access to e-mail
43	(E.15.5) The practice is aware of the contractual obligations and makes every effort to avoid financial mishaps	(FINANCE: Financial leadership and responsibilities) Responsibilities for financial management are clearly defined
44	(E.16.1) The practice promotes a healthy balance of work and home life	(PEOPLE: Perspective of GPs on working conditions) GPs experience a positive work satisfaction	(92) % of PHC providers who were satisfied with the overall quality of work life balance over the past 12 months, by type of PHC provider.

CAD, Coronary Artery Disease; COPD, Chronic Obstructive Pulmonary Disease; GP, General Practitioner; NI, National Insurance; NPs, Nurse Practitioners; PHC, Primary Health Care; TIA, Transient Ischemic Attack.

^aFor more details about the indicators and criteria see Quality Tool (23).

^bMost appropriate indicator was used in this table, other indicators may also apply.

Delphi method

Development of indicator inclusion principles

The pre-Delphi included three family physicians and three administrators, two nurses, a social worker, a dietician and a patient. Eleven completed an on-line survey about which of nine proposed principles should be included to assist the Delphi panel in its deliberations. A teleconference was held to discuss survey findings and there was consensus that four principles should be used to provide context when considering the inclusion of each indicator (Table 2).

Table 2.

Inclusion principles and definitions for contextual rating of indicators in Round 1and Round 2 of the Delphi process

Inclusion principle	Definition
Value added	The indicator is value added: it reflects an area of assessment that is not covered by any other process.
Measurable	The indicator is measurable at the patient, practice or population level and changes in the indicator can be clearly identified and compared over time.
Standard	The indicator’s criteria would be considered a standard for family practice, including what is formally required by law.
Important	The indicator reflects an important or emerging issue that impacts on primary health care or primary health care delivery and provides information that can be used to inform policy decisions or change the behaviour of health service providers.

Each assessed using a 5-point Likert scale: strongly disagree = 1; disagree = 2; neutral = 3; agree = 4; strongly agree = 5.

Delphi panel member characteristics

The Delphi panel consisted of 23 stakeholders, 11 pre-Delphi panel members, 11 additional family physicians and a pharmacist. All Delphi members completed each on-line survey. In total, 18/23 (78%) participants attended the first and 16/23 (70%) the second round conference calls. Eighteen (78%) attended the final face-to-face meeting and completed the final ratings on paper or sent in their final ratings electronically. Of the inter-professional group participating, a nurse missed the first round teleconference and a practice manager, a nurse and social worker missed the second round teleconference, but all four attended the face-to-face meeting. One practice manager attended each teleconference but missed the face-to-face meeting and sent her final ratings electronically. The patient expert participated in each Delphi round and the face-to-face meeting.

Delphi Indicators

Table 3 summarizes the number included, excluded, recommended to be criteria and undecided indicators from each round of the Delphi process. Overall, 37/63 indicators were included, 15 were excluded and 11 were recommended to be criteria.

Table 3.

Summary of included/excluded/criteria/undecided indicators in Delphi process

Delphi Round	n	Included	Excluded	Criteria	Undecided
Round 1	63	30	5		28
Round 2	28		4	7	17
Face-to-face meeting	17	7	6	4	0
Total	63	37	15	11	0

Tables 4, 5 and 6 list the included, recommended to be criteria and excluded indicators. These tables also include the indicator origin, and the round in which inclusion/exclusion occurred. Of the final 37/63 unique and new indicators included, 25 of 36 indicators originated from the Quality Tool, 0 of 1 from EPA, 1 of 7 from CIHI, 6 of 12 from QOF and 4 of 6 from new indicators. An example of an indicator becoming a criterion is the indicator from the QOF(10), ‘[Percentage of] Patients on the diabetes register and/or the CHD register for whom case finding for depression has been undertaken on one occasion during the previous 15 months using two standard screening questions’ became a criterion under the indicator ‘Diabetes Mellitus, …the percentage of patients with diabetes mellitus who in the past 15 months have a record of …depression screening’ (40).

Table 4.

Delphi Panel’s included indicator, description, indicator origin, number and percentage ratings Yes, and round or face-to-face meeting in which they were held

Indicator (n = 37)	Description	Indicator origin	n/N (%) Yes	Round or face to face included
1	The practice demonstrates its commitment to respecting the needs and rights of its practice population	Quality Tool	19/23 (82.6)	Round 1
2	Fire risk is minimized by demonstrating a commitment to relevant legislation and Codes of Practice relating to fire safety, disasters or other emergencies	Quality Tool	19/23 (82.6)	Round 1
3	There is an audit of medication reviews	Quality Tool	19/23 (82.6)	Round 1
4	Screening for TB status, and immunization status for measles, rubella, polio, influenza, tetanus, diphtheria and pneumococcus is reviewed on a regular basis, according to CPSO Infection Control Guidelines section 1.4	Quality Tool	19/23 (82.6)	Round 1
5	The practice can produce a register of patients with heart failure.	QOF	19/23 (82.6)	Round 1
6	Patients who have a current diagnosis of heart failure due to LVD are treated with ACE inhibitors or ARBs if there is no contraindication.	QOF	19/23 (82.6)	Round 1
7	Patients with prescription or illicit drug use problems are offered, provided, or directed to treatment by their PHC provider	CIHI	19/23 (82.6)	Round 1
8	The practice provides comprehensive care for older adults	New indicator	19/23 (82.6)	Round 1
9	Patients on the CKD register have had their blood pressure recorded in the previous 15 months.	QOF	19/23 (82.6)	Round 1
10	Patients on the CKD register with hypertension and proteinuria are treated with an angiotensin converting enzyme inhibitor (ACE-I) or angiotensin receptor blocker (ARB) (unless a contraindication or side effects are recorded)	QOF	19/23 (82.6)	Round 1
11	Mandatory reporting occurs in accordance with legislation in the family practice	Quality Tool	20/23 (86.9)	Round 1
12	The practice has an effective system to monitor waiting times for investigations and referrals to ensure that patients are receiving them in a timely manner	Quality Tool	19/23 (82.6)	Round 1
13	The practice continuously improves team functioning	Quality Tool	20/23 (86.96)	Round 1
14	Patients with alcohol problems have received specific help or information to manage their alcohol consumption	CIHI	20/23 (86.96)	Round 1
15	The practice can produce a register of patients aged 18 years and over with CKD	QOF	20/23 (87.0)	Round 1
16	Examination areas for assessment and management ensure patient comfort	Quality Tool	20/23 (87.0)	Round 1
17	The practice has a protocol/policy and procedures in case of an accidental needle-stab or other type of exposure	Quality Tool	21/23 (91.3)	Round 1
18	The practice has an effective system to identify and record adolescent immunizations	Quality Tool	21/23 (91.3)	Round 1
19	Patients with dementia have had their care reviewed in the last 15 months.	QOF	21/23 (91.3)	Round 1
20	Colorectal cancer screening for patients 50–74 years and over has taken place via FOBT in the last 2 years or completed colonoscopy in the last 5 years.	New indicator	21/23 (91.3)	Round 1
21	The practice premises are clearly signposted and physically accessible	Quality Tool	22/23 (95.7)	Round 1
22	Biomedical waste (includes anatomical waste, blood, non-anatomical waste, and other waste) is safety disposed of in accordance with local regulations	Quality Tool	22/23 (95.7)	Round 1
23	Office procedures are only performed after suitable training and in accordance with accepted guidelines	Quality Tool	22/23 (95.7)	Round 1
24	The practice uses evidence-based clinical practice guidelines to ensure consistent high quality health care	Quality Tool	22/23 (95.7)	Round 1
25	The practice has a policy for how it provides maternity services to its patients	Quality Tool	22/23 (95.7)	Round 1
26	Continuity of care is promoted by the practice	Quality Tool	22/23 (95.7)	Round 1
27	Screened adults with positive stool for occult blood are notified of results and referral for colonoscopy within 2 weeks of the test results.	New indicator	22/23 (95.7)	Round 1
28	The practice maintains the privacy of patient information in accordance with Bill 31	Quality Tool	23/23 (100)	Round 1
29	Examination areas for assessment and management ensure patient safety	Quality Tool	23/23 (100)	Round 1
30	The practice has appropriate vaccine storage and maintains the Cold Chain in line with provincial guidelines	Quality Tool	23/23 (100)	Round 1
31	The practice has a shared care model in psychiatry, obstetrics and chronic disease management that promotes continuity of care	Quality Tool	15/18 (83.3)	Face to face
32	The practice honours its commitment to recognizing the diversity of its patients	Quality Tool	15/18 (83.3)	Face to face
33	The practice has a system to keep track of and manage patients that are hospitalized, in rehabilitation, and following discharge	Quality Tool	16/18 (88.9)	Face to face
34	Patients are seen by the third next available appointment by provider	New indicator	15/18 (83.3)	Face to face
35	The practice ensures that only authorized people have access to prescription medication kept within the practice	Quality Tool	17/18 (94.4)	Face to face
36	The practice has a policy for prevention, investigation, management, and referral for sexually transmitted diseases	Quality Tool	15/18 (83.3)	Face to face
37	The practice respects patients’ rights to formally complain	Quality Tool	16/18 (88.9)	Face to face

ACE, angiotensin-converting enzyme; ARBs, angiotensin receptor blockers; CKD, chronic kidney disease; CPSO, College of Physicians and Surgeons of Ontario; FOBT, Fecal Occult Blood Test; LVD, Left Ventricular Dilatation; n, number responding; N, of total group; PHC, Primary Health Care.

Table 5.

Delphi Panel’s excluded indicators, description, indicator origin, number and percentage ratings Yes, and round or face-to-face meeting in which they were included

Indicator (n = 15)	Description	Indicator origin	n/N (%) Yes	Round or face to face included
1	The practice has a system in place for undertaking research	Quality Tool	11/23 (47.8)	Round 1
2	The practice has organized a social event to which the whole staff was invited in the past 12 months.	EPA	11/23 (47.8)	Round 1
3	Team effectiveness is facilitated by strong leadership.	CIHI	10/23 (43.5)	Round 1
4	There is a register of patients who have reported having duplicate medical tests completed in the last 12 months.	CIHI	8/23 (34.8)	Round 1
5	Doctor’s bags are not accessible to unauthorized persons	Quality Tool	9/23 (39.1)	Round 1
6	Regular patients of the practice are able to obtain visits from a provider in long-term care facilities	Quality Tool	9/23 (39.1)	Round 2
7	The practice can produce a register of patients with learning disabilities.	QOF	10/23 (43.5)	Round 2
8	Team effectiveness is facilitated by a focus on quality care	CIHI	11/23 (47.8)	Round 2
9	The practice promotes quality provider–patient relationships	Quality Tool	11/23 (47.8)	Round 2
10	The practice demonstrates a commitment to an eco-efficient or green office.	Quality Tool	7/18 (38.9)	Face to face
11	There are active contacts with local family physician organizations; e.g. PCNS, FHTs, LHINS, Hospital Family Medicine Departments	Quality Tool	8/18 (44.4)	Face to face
12	Post-menopausal Bleeding	Quality Tool	4/18 (22.2)	Face to face
13	The practice can produce a register of patients aged 16 and over with a BMI greater than or equal to 30 in the previous 15 months	QOF	7/18 (38.9)	Face to face
14	Benzodiazepines for older adults	Quality Tool	6/18 (33.3)	Face to face
15	Sore Throat	Quality Tool	8/18 (44.4)

BMI, body mass index; FHTs, Family Health Teams; LHINS, Local Health Integrated Networks; n, number responding; N, of total group; PCNs, primary care networks.

Table 6.

Delphi Panel included indicators that were recommended to become criteria: description, indicator origin, number and percentage ratings Yes, and round or face-to-face meeting in which the decision was made

Indicator (n = 11)	Description	Indicator Origin	n/N(%) Yes	Round or face to face
1	Patients with a diagnosis of heart failure have had this confirmed by an echocardiogram or by a specialist assessment	QOF	14/23 (60.9)	Round 1
2	[Percentage of] Patients on the diabetes register and/or the CHD register for whom case finding for depression has been undertaken on one occasion during the previous 15 months using two standard screening questions	QOF	14/23 (60.9)	Round 1
3	There is an auditing of the number of patients with CHF who have visited the emergency department in the last 12 months.	CIHI	15/23 (65.2)	Round 1
4	Patients spend no more than 45 minutes between arrival and departure from physician visit	New indicator	15/23 (65.2)	Round 1
5	There is a register of patients with CHF who are treated with ACE inhibitors or ARBs	CIHI	17/23 (73.9)	Round 1
6	[Percentage of] Patients on the CKD register whose blood pressure has been recorded in the previous 15 months and was 140/85 or less	QOF	17/23 (73.9)	Round 1
7	The practice completes assessments of functional status for older adults	New indicator	18/23 (78.3)	Round 1
8	The practice provides comprehensive care for chlamydia	Quality Tool	3/18 (16.7)	Face to face
9	There is a register of patients whose 1st language is not English who have reported having language barriers	CIHI	11/18 (61.1)	Face to face
10	The practice can produce a register of patients diagnosed with dementia	QOF	9/18 (50.0)	Face to face
11	The percentage of women of childbearing age where rubella immune status is recorded	Quality Tool	7/18 (38.9)	Face to face

ACE, angiotensin-converting enzyme; ARBs, angiotensin receptor blockers; CHD, coronary heart disease; CHF, congestive heart failure; n, number responding; N, of total group.

Contribution of Inclusion Principles to Overall Inclusion

Mean ratings for indicators that were included compared to indicators that were not included in the first round were statistically significantly higher for the measurable (3.95, 0.29 versus 3.71, 0.29; P < 0.01) and standard (3.89, 0.54 versus 3.52, 0.37; P = 0.01) principles although the magnitude of the differences was less than half a point on the five-point Likert scale. Mean ratings for indicators that were included compared to indicators that were not included in the first round for important and value-added principles were not statistically significantly different. Therefore the inclusion principles did not appear to influence the inclusion decision.

Discussion

Summary of main findings

This study lists, compares and regards as face valid indicators in the Quality Tool that are the same or similar to indicators in international and Canadian tools that have been previously tested or validated (6,10,18). The remaining unique indicators and new indicators were content validated using a Delphi panel of Canadian experts. This has resulted in a list of validated indicators that have been further refined in the next version of the Quality Tool, the Quality Book of Tools, applicable to the current Canadian and international primary care context (40). These indicators represent a comprehensive range of primary care domains, are useful for family physicians, primary care providers, managers and decision makers and could be used for measuring quality improvement in primary care in Canada and elsewhere.

The purpose of the study was to create a valid set of primary care indicators and simplify the Quality Tool. Of the final list of 81 included indicators, 44 were face valid through comparative analysis by members of our research team and 37 were content valid through the Delphi process. On the face of it, the number of indicators at the end were more. However, 107 indicators were reviewed, 80 from the Quality Tool and 27 indicators were added. Only 69 (44 plus 25) of the original 80 indicators in Quality Tool were included. Twelve new indicators were added to the set following the Delphi. A total of 81/107 (69 plus 12) indicators were validated. Criteria allowed further simplification, as some indicators could be grouped together and the detail measured as a criterion. Furthermore, the Quality Book of Tools (40) has 70 indicators and 198 criteria, as there were further redundancies removed in the final revisions. The process also enabled a refinement of the categories of groups of indicators, informed by the international tool comparison (19). These categories emphasize domains of key attributes of primary care such as continuity of care and equity.

Voluntary accreditation has been gaining acceptance as a model to measure and improve quality in health care since the 1970s, and in primary care since the 1990s. A number of tools have been developed, validated and tested to support these accreditation programs in different countries and regions (4–6). The Quality Tool is rooted in these other tools. Although the tools are designed for similar primary care settings, to our knowledge, no study has reported an international comparison of similar and unique indicators in primary care and from them created a validated set of indicators.

Limitations and strengths

Some key indicators excluded by the Delphi panel are very common such as sore throat and post-menopausal bleeding, or have emerged as new areas to assess such as the ‘green’ office. However, given the broad spectrum of primary care and its ongoing evolutions, no tool is ever complete. Any tool will likely result in the inclusion of additional indicators, and the exclusion of others and will require regular revisions and updates.

The high participation and response rate was generally a strength of this study: 100% for survey completion and 70–78% participation in the teleconference calls and 78% for the face-to-face meeting. The international expert participation at the face-to-face meeting brought additional experience from established programs in Europe, Australia and New Zealand and added further context to the discussion about maintaining, modifying or excluding indicators.

This study used an interdisciplinary multi-stakeholder expert panel to reach consensus. The experts came from key health care professionals (physicians, nurses, dieticians, social workers and pharmacists), staff (practice managers, receptionists) and patient representation who would be involved in quality assessment of indicators in family practices. As such, the included indicators are well positioned for use in interdisciplinary health care teams in Canada and elsewhere. The interdisciplinary group was well represented in all rounds. At no time were there any complaints about the lack of inclusion of the voice of any of the groups in the Delphi panel. Our feedback from participants was generally very positive. Our patient expert participated in all rounds of the Delphi bringing the patient perspective to the expert panel and we do not claim that this process fully represents the interdisciplinary or patient voice.

We developed the inclusion principles with the expectation that their ratings would inform the final decision about whether to include the indicator as valid, and provide additional information to inform the discussion. We were surprised that there was little statistical significance found between the inclusion principles and the ratings; however, we believe it did inform the discussion and was a useful exercise. In future, we would use these principles as a preamble to reflect on prior to making a decision and would not rate them.

Some might criticize the equivalency of the indicators and criteria from the Quality tool with indicators from other tools (Table 1). Our research team content experts reviewed all the matched indicators and reached agreement that they were similar enough to be included as face validated indicators. Our international experts helped clarify major differences in the indicators at the face-to-face meeting. However there may still be some misunderstandings of the wording used in the tools.

Conclusion

This study validated 81 primary care indicators by comparing indicators in the Quality Tool with those that were the same or similar to indicators in other tools and by subjecting the remaining unique indicators and new indicators to a Delphi panel process. These indicators formed the foundation for the next phase of the Quality Tool (40).

Declarations

Funding: Ontario Ministry of Health and Long-Term Care .

Ethical approval: Research Ethics Board Approval [08-369], McMaster University.

Conflict of interest: none.

Acknowledgements

The authors would like to acknowledge Carol Lane for her research assistance, Michelle Howard PhD for assistance with analysis of the inclusion principles and Annie Zhu for assistance in preparation of the manuscript.