Multiple case study of processes used by hospitals to select performance indicators: do they align with best practices?

Abstract Several health policy institutes recommend reducing the number of indicators monitored by hospitals to better focus on indicators most relevant to local contexts. To determine which indicators are the most appropriate to eliminate, one must understand how indicator selection processes are undertaken. This study classifies hospital indicator selection processes and analyzes how they align with practices outlined in the 5-P Indicator Selection Process Framework. This qualitative, multiple case study examined indicator selection processes used by four large acute care hospitals in Ontario, Canada. Data were collected through 13 semistructured interviews and document analysis. A thematic analysis compared processes to the 5-P Indicator Selection Process Framework. Two types of hospital indicator selection processes were identified. Hospitals deployed most elements found within the 5-P Indicator Selection Process Framework including setting clear aims, having governance structures, considering indicators required by health agencies, and categorizing indicators into strategic themes. Framework elements largely absent included: adopting evidence-based selection criteria; incorporating finance and human resources indicators; considering if indicators measure structures, processes, or outcomes; and engaging a broader set of end users in the selection process. Hospitals have difficulty in balancing how to monitor government-mandated indicators with indicators more relevant to local operations. Hospitals often do not involve frontline managers in indicator selection processes. Not engaging frontline managers in selecting indicators may risk hospitals only choosing government-mandated indicators that are not reflective of frontline operations or valued by those managers accountable for improving unit-level performance.


Introduction
Given the financial resources dedicated to patient care, governments and private insurers around the world have mandated healthcare organizations to monitor hundreds of indicators to help improve outcomes and enhance accountability [1][2][3][4][5][6].An unintended consequence of these well-intended mandates is an overabundance of indicators that can paralyze decision-making and create distrust between providers, government agencies, and insurance companies [1][2][3][6][7][8][9][10]. Consequently, several agencies recommend reducing the number of indicators monitored by healthcare organizations [10][11][12][13][14].However, to identify which indicators should be eliminated, one must understand indicator selection processes.
Current literature focuses on how individual clinical disciplines select indicators.Little research examines the methods that provider agencies, such as hospitals, use to select critical operational indicators to measure organizational performance, while complying with regulatory reporting that may not match operational needs [15].A recent scoping review resulted in the development of the 5-P Indicator Selection Process Framework which offers an evidence-based structure to design indicator selection processes [15].As illustrated in Table 1, the framework's five domains (Purpose, Polity, Prepare, Procedure, and Prove) and 17 elements represent a best practice checklist that helps organization determine why they are selecting indicators; who should govern and select indicators; and how to compile, assess, and validate a final list of indicators [15].

Methods
This study researched the question: "What processes do hospitals in Ontario, Canada use to select performance indicators and how do they align with the 5-P Indicator Selection Process Framework?"The study's exploratory qualitative design included two data collection methods: semistructured interviews and document analysis [16].
In Ontario, Canada, there are 147 hospitals publicly funded through a single government agency.Fifty-seven operate 48% of the province's beds and are classified as large community acute care hospitals [17].Purposeful sampling was used.Four hospitals operating at least 400 beds with annual revenues greater than $400 Million Canadian Dollars were studied [17].These parameters ensured that hospital cases had comparable services and data reporting requirements.Across four hospitals, 13 executives responsible for reporting on indicators were interviewed with three respondents from Cases A, B, and D and four respondents from Case C. University Table 1.The 5-P Indicator Selection Process Framework [15].

Domain Elements Element description
Purpose Clarify Aim Articulate the rationale for conducting an indicator and target selection exercise.By stating the process aim, whether it is to align indicators to an operational process, a strategic plan, a regulatory requirement, or public reporting, the work can be scoped properly.

Develop Guiding Principles
Establish principles to ensure participants understand the values by which the process is being conducted.Principles may include openness, transparency, scientific soundness, relevance, accountability, scope, and span of control.

Identify Level of Use
Identify the organizational unit that will use the indicators to ensure relevancy to end users.As an example, indicators used by a Board to monitor quality outcomes may be different from indicators selected by a clinical unit focused on process improvement.

Build Governance Structures
Identify a structure that will manage indicator and target selection to ensure it is completed.These structures may include a steering committee, a project management team, a data quality advisory group, and an expert panel that will assess potential indicators and targets.

Recruit Participants
Select and recruit expert panel members.Panels should be diverse and multidisciplinary to ensure equity and a broad view of how indicators and targets will be used.Identify and use a recognized consensus-building method such as the Delphi, modified Delphi, or Normative Group Technique.This is particularly important when indicators are being identified to measure a new strategy compared to a quality improvement project.

Identify a Facilitator
Select an independent facilitator so as to not bias the process.The facilitator should be a third party or a neutral party from an organization's performance measurement department.

Indicator Selection Criteria
Set criteria by which the assessment of indicators will be based.Respondents received a consent form, copy of the 5-P Indicator Selection Process Framework, and interview protocol.Respondents were interviewed about their hospital's most recent indicator selection process.Interviews were conducted by the first author between June and August 2021, were 45-60 min in length, audio-recorded, and transcribed.Interview data were analyzed using inductive techniques, and responses were aggregated into thematic findings to ensure that contributions were anonymous, protected from social risks, and reflected a common perspective [16].Document analysis of data presentations and briefing notes were triangulated with interview answers [16].Data coding was completed by the first author and validated by remaining authors.Member checking was completed by sending aggregated, anonymized summaries, from the first author to respondents individually to validate data [16].Thus, participants' identities were known only by the first author.

Results: case findings
Illustrated in Table 2, two indicator selection processes were identified from analysis of the four cases.Table 3 maps each case process to the framework.

Case A-Process 1
Case A selected indicators at the beginning of its fiscal year.
The process this study explored was conducted in 2020.Case A did not document its selection methods.Case A described improving quality, accountability, and funding requirements as process aims.Executives led the process with support from the finance and quality departments.The process did not include a formal consensus methodology.While a draft indicator list was shared with clinical directors and physician leaders, Case A did not involve either group in final indicator selection.Executives set annual targets based on reaching the top 25th percentile performance in their peer group.
Frontline managers did not provide input on the indicators being monitored.Case A-Executive 2 noted this opportunity, "We've successfully hardwired a meeting that brings leaders together to go over data, but those metrics are picked by executives.Moving forward we need to seek feedback from clinical units to see if they would change anything."Case A's process aligns to only a few domains of the framework (Table 4).

Case B-Process 2
Case B completed an indicator selection process following its 2019 strategic plan, which aimed to select indicators that would measure business performance, enable benchmarking, meet reporting guidelines, and ensure organizational alignment.Facilitated by the strategy department, a working group identified initial indicators mandated by government agencies and peers.Directors and physician leaders recommended a final list to executives and the board who cogoverned the process.Approved indicators were shared with frontline managers for information only and not to seek validation.Unique aspects of Case B's process included having data automation as a selection criterion and that they sought to balance the number of process, outcome, and structural indicators.Case B-Executive 2 shared that one of their goals was to "use our process to get directors, physicians and executives to use the same nomenclature, so we knew how to work together." In summary, Case B's process aligns to several elements of the framework (Table 4).

Case C-Process 2
Case C conducted an indicator selection process following its 2019 strategic plan.The aim of the process was to support improvement, measure business performance, and enhance accountability.Case C-Executive 1 stated, "We've put an emphasis on picking indicators that make our leaders accountable for our entire business performance.That means focusing on quality, but also paying attention to finance and human resource metrics."Case C's executives asked clinical directors and physician leaders to draft a list of indicators before they made recommendations to their board.Initial indicators were generated from government and peer scorecards.A formal consensus methodology was not used.Participants were oriented on the process.The final indicator list was not validated by frontline managers.A unique aspect of Case C's process was its aim to make the process simple by encouraging participants to select a manageable number of indicators.Case C-Executive 2 noted that the process was facilitated by an external consultant, stating "We needed an outside expert to challenge previous biases to measurement."In summary, Case C's process aligned with most of the framework (Table 4).

Case D-Process 1
Case D reviewed indicators at the beginning of each fiscal year.The process this study explored was conducted in 2020.Case D documented the process, which aimed to measure its strategic performance.The process was governed by its executives and board with openness and transparency as guiding principles.Indicator selection criteria included data quality, data availability, and ability to benchmark.Case D was in the process of reviewing these criteria.As Case D-Executive 3 shared,  • Produces indicator definitions and data sources.
• Does not include target rationales.
• Produces indicator definitions, data sources, and targets.
• Does not include target rationales.
• Produces indicator definitions, data sources, and target justifications when selecting indicators.

Categorize Potential indicators into Strategic Themes
• Categorizes indicators into themes that match their strategic plan.
• Does not consider if an indicator is a process or outcome indicator.
• Categorizes indicators into the quadrants of their balanced scorecard.
• Considers if an indicator is a process or outcome indicator.

Categorizes indicators into the quadrants of their balanced scorecard
• Does not consider if an indicator is a process or outcome indicator.
• Categorizes indicators into the quadrants of their balanced scorecard.
• Does not consider if an indicator is a process or outcome indicator.
(continued) • Data quality, data availability, gap in performance, and clinical relevance.
• Data quality and availability, strategic alignment, benchmarking, and quality improvement.

Analytically Assess indicators
• Do not vote on an indicator list.
• Indicators selected by informal agreement.
• Do not vote on an indicator list.
• Indicators selected by informal agreement.
• Do not vote on an indicator list.
• Indicators selected by informal agreement.
• Do not vote on an indicator list.
• Indicators selected by informal agreement.

Set indicator Targets
• Selects targets by first meeting provincial benchmarks.

•
No target setting philosophy on finance or human resource (HR) indicators.
• Selects targets by benchmarking against peer performance then incrementally reaching top 25th percentile.
• Finance and HR indicator targets align to fiscal plan.deaths and then use risk adjustments so clinicians can make direct practice changes.
Unique aspects of Case D's process included the documentation of indicator selection criteria, definitions, and target justifications.In summary, Case D's process matched some elements of the framework (Table 4).

Results: multicase analysis
The following section describes the common alignment and deficiencies case processes had compared to the framework.

Domain 1: Purpose
All hospitals articulated the aim of their processes, with Cases B, C, and D documenting them.Aim statements included selecting indicators that measured business performance, supported quality improvement, and met reporting requirements.Case A did not identify guiding principles related to their process, whereas Case B articulated principles, but they were not documented.Cases C and D documented guiding principles.Openness, transparency, and accountability were common guiding principles.

Domain 2: Polity
Executives led their respective processes.Cases B, C, and D required board approval of final indicator lists, while Case A did not report having to do so.Process participants included executives, directors, and physician leaders.Absent from all governance structures were end users such as frontline managers.

Domain 3: Prepare
Government-mandated indicators were prioritized given they are often essential to obtain funding.All hospitals noted that the maintenance of indicator lists and definitions was inconsistent and an area for improvement.Cases B and C oriented participants on indicator selection given their processes followed new strategic plans.Cases A and D did not orientate participants to their annual processes.

Domain 4: Procedure
No case used a consensus-building methodology such as a Delphi process [18], arguing that the absence of a consensus method was influenced by their culture.Case B-Executive 1 stated, "We would never put a list of indicators in front of our leaders to vote on.Given we need to work together every day, it is just not in our culture.If there is a disagreement, that would be up to executives to decide." Cases A, B, and D had internal staff facilitate their processes, while Case C used external facilitation.Selection criteria did not vary significantly across cases.Data quality, availability, and comparability were common criteria.All hospitals considered an indicator if it was required to obtain funding.
Case B considered whether an indicator was a process, structure, or outcome indicator.
In setting targets, hospitals relied on benchmarks published by government and policy agencies.Cases B and C's target setting philosophies were based on achieving the top 25th percentile of their peer group.In setting targets, Case D analyzed annual performance and then considered peer and government benchmarks.Target setting philosophies were evident for quality indicators, but not financial or human resources indicators.

Domain 5: Prove
No case quantitatively validated their indicators for data quality, reasoning that they generally selected governmentmandated indicators.Hospitals did not seek input from frontline managers or patients on the face validity of indicators, opting to share a list of indicators for information only.Case B-Executive 3 explained, "We present our indicators to our patient council.But it is more of a report to them than active engagement."

Statement of principal findings
This study identified two types of indicator selection processes: the renewal of indicators to measure annual operational performance and selection of indicators following the completion of a new strategic plan (Table 2).Table 4 provides a comparison of case processes and how they align with the 5-P Indicator Selection Process Framework.No process mirrored the framework completely.Cases A and D, who completed processes as part of their annual planning cycle, were less aligned with the framework, whereas Cases B and C, who completed processes following strategic plans, were more aligned.Common findings are categorized into three themes: the structure and mechanics of indicator selection processes, the engagement of end users, and documentation.
Structure and mechanics of indicator selection processes is the first area of findings in which gaps existed.These gaps are found in the following framework elements: Clarify Aim, Identify a Facilitator, Indicator Selection Criteria, Set Indicator Targets, and Assess Data.
All hospitals confused setting process aims and guiding principles.The aim of a process may be to select indicators that measure business performance or quality, whereas the principles guiding the process may be openness, transparency, and accountability.Setting clear aims and guiding principles allows participants to understand process's goals and how they may contribute [19].Hospitals generally relied on internal staff to facilitate the process.Internal facilitation may bias processes, whereas external facilitators can manage discussions that might otherwise be difficult [20].Given indicator selection processes are linked to measuring goals and accountabilities, processes would be improved with external facilitation.
Hospitals focused on selecting quality and patient experience indicators.However, hospitals are not only medical agencies, but business units with economic impact.Hospitals would benefit from having finance and human resources indicators as part of their selection processes.The Donabedian framework advocates that quality is measured using three types of indicators: structural, process, and outcome [21].Only Case B used the Donabedian framework as a criterion for selecting indicators.Hospitals should consider the Donabedian framework when selecting indicators to explore the cause and effect relationships that influence quality.
All four cases stated that government funding requirements were criteria they strongly considered.This led hospitals to question how to balance monitoring indicators that measure local needs while also monitoring mandated indicators that may be less relevant.This finding confirms earlier literature that found government-mandated indicators that are not regularly renewed are less relevant to operations and create confusion [2,3,5,6,22].To streamline indicator selection, hospitals and funding agencies should align criteria to instruments such as the Appraisal of Indicators through Research and Evaluation tool [23].Of course, not all hospitals operate in a single-payer system; we note that in multipayer systems, as in the USA, payers may have different priorities, which may add further complexity to indicator selection.
Study participants identified that target setting needed improvement.They attributed this need to their reliance on government-mandated metrics, overuse of benchmarks, and an inability to attribute numerical gains to improvement activities.
No case quantitatively validated their list of indicators for data quality.Hospitals reasoned that they did not need to complete this validation given most indicators they chose were from funding agencies.This is a false assumption.While an indicator's technical formula may be validated by an external agency, data generated from local information systems may be of different quality.Confidence in hospital processes may be improved if they validate local data quality.
The most glaring deficiency of the processes studied was in the engagement of end users in indicator selection.This gap is associated with the following framework elements: Recruit Participants, Seek End User Input, Orient and Train Participants, Validate with End Users, and Utilize a Consensus-Building Method [15].
Hospital board involvement seemed perfunctory.Boards appeared to only approve a recommended list, rather than actively participating in indicator selection.While cases oriented participants on measurement if a process followed a strategic plan (Process 2), hospitals did not orient participants on annual processes (Process 1).These are important reflection points.Involving boards in indicator selection helps improve outcomes [24,25].Processes that have broader participation have a deeper understanding of frontline operations [15].Participants who understand why a project is undertaken are more likely to positively contribute to the initiative [26,27].Excluding individuals who are accountable for service delivery can lead to selecting indicators that do not match operational realities [6,9,28].
All cases stated that time demands prohibited them from using consensus-building methodologies.Delphi techniques allow transparent debate while reducing bias and groupthink [18].Given all cases desired to build collaborative cultures, Delphi techniques should be seen as helpful and not a hindrance.Documentation generally took the form of briefing notes and presentations.To ensure indicator selection processes are defined and understood, hospitals should document their approaches in the form of policies and procedures.Utilizing the framework as a basis for documentation will ensure that key elements of the process are considered.

Strengths and limitations
This paper represents the first real-world assessment of indicator selection processes using the 5-P Indicator Selection Process Framework.A qualitative multiple case study approach enabled a systematic exploration of participant experiences across different hospitals.However, this study also has several limitations.Its focus on large community acute care hospitals in Ontario, Canada, may limit the generalizability of findings to other hospital types.This study examined hospitals funded by a single public agency; many others are subject to multiple public and private reimbursement agencies.The study was completed during the corona virus disease (COVID-19) pandemic: this may have impacted participant recall given hospitals may have stopped reviewing indicators during that period.This study researched executive perceptions of indicator selection and not those who participated in the described processes, who may have had different perceptions.While we were fortunate to gain access to executives, the sample (n = 13) is relatively small.Finally, respondents received an advance copy of the framework that may have influenced their responses.

Implications for policy, practice, and research
This study identified two types of hospital indicator selection processes that are heavily influenced by indicators mandated by regulatory and funding agencies.For policymakers, this study may inform how they might standardize the methods by which indicators are selected and retired.For practitioners, this study emphasizes the need to include frontline managers in indicator selection processes, so they can understand how they can contribute to improving indicator performance.An objective of the framework is to guide better collaboration and decision-making.Future research may therefore study how the framework informs proactive planning of indicator selection and how indicators are used for operational decision-making.

Interpretation within the context of the wider literature
There is much research on consensus-building exercises to select indicators for specific clinical areas compared to studying how hospital organizations select indicators [15].This study helps address that gap by being the first real-world assessment of four hospital indicator selection processes using the 5-P framework.

Conclusion
Reducing the number of indicators required by both regulatory and payment bodies is a major challenge for healthcare systems that will take time to achieve.In the meantime, hospital executives must help their organizations select an appropriate number of critical indicators to help their frontline managers achieve local strategic and operational goals.This study identified two indicator selection processes used by hospitals: annual renewal of indicators to measure operational performance and selection of indicators following the completion of a new strategic plan.Gaps in these processes compared to the 5-P Indicator Selection Process Framework included the lack of guiding principles; consensus-building methodologies; evidence-based selection criteria; business-based indicators; balance between structural, process, and outcome indicators; and validation.The most glaring gap was the lack of engagement of frontline unit managers in these processes.Frontline managers should be active participants in indicator selection given their accountability for implementing changes designed to improve outcomes.
We've made indicator selection too simple by only accepting administrative outcome data.Patient conditions are very complex.If I had to re-select the indicators we monitor, I would focus on clinical process indicators like how many specimens are lost, or number of cancers misdiagnosed, or outcome indicators that address unnecessary Composition of panels should consider the process aim and level of use when selecting participants.
Identify the range of indicators used in their area or that are required by regulation.A search of literature and evidence-based guidelines and government-mandated indicators will help organizations identify a comprehensive set of indicators to assess.Build an Inventory of Potential IndicatorsCompile a comprehensive list of indicators with definitions and data sources, so participants understand each indicator to be evaluated.If the process addresses target selection, the nature of the target (e.g.past performance, benchmark, and best practice) should be explained.Categorize Potential Indicators into Strategic ThemesCategorize indicators into themes aligned with the organization's strategy, quadrants of the balanced scorecard, or the Donabedian framework of outcomes, process, and structure.By creating categories, process participants and end users will better understand the linkage an indicator has with the identified purpose.Orient and Train ParticipantsProvide participants with orientation materials on the process aim, definition and purpose of each indicator, potential targets, and methods they will use to recommend indicators and targets.

Table 2 .
Ontario hospital indicator selection process types.Annual process to select indicators that will measure yearly goals and objectives.•Governed by the senior executive team and corporate data support departments such as business analytics, finance, and quality recommend indicators to the senior team.• Senior executive reports indicator results to their board, but the board is not formally involved in indicator selection.Clinical directors and physician leaders are informally consulted but are aware of the indicators being recommended.• Process may or may not be documented.Process is governed by the board and senior executive team and involves clinical director and physician leader participation in the selection of indicators and targets.• Process is documented and outlines aim, guiding principles, indicator selection criteria, and target setting rationales.

Table 3 .
Case findings compared to the 5-P Indicator Selection Process Framework.

Table 4 .
Case Alignment with 5-P Indicator Selection Process Framework Summary.