Validation of the Antifungal National Antimicrobial Prescribing Survey (AF-NAPS) quality assessment tool

Abstract

Background

The Antifungal National Antimicrobial Prescribing Survey (AF-NAPS) was developed to undertake streamlined quality audits of antifungal prescribing. The validity and reliability of such tools is not characterized.

Objectives

To assess the validity and reliability of the AF-NAPS quality assessment tool.

Methods

Case vignettes describing antifungal prescribing were prepared. A steering group was assembled to determine gold-standard classifications for appropriateness and guideline compliance. Infectious diseases physicians, antimicrobial stewardship (AMS) and specialist pharmacists undertook a survey to classify appropriateness and guideline compliance of prescriptions utilizing the AF-NAPS tool. Validity was measured as accuracy, sensitivity and specificity compared with gold standard. Inter-rater reliability was measured using Fleiss’ kappa statistics. Assessors’ responses and comments were thematically analysed to determine reasons for incorrect classification.

Results

Twenty-eight clinicians assessed 59 antifungal prescriptions. Overall accuracy of appropriateness assessment was 77.0% (sensitivity 85.3%, specificity 68.0%). Highest accuracy was seen amongst specialist (81%) and AMS pharmacists (79%). Prescriptions with lowest accuracy were in the haematology setting (69%), use of echinocandins (73%), mould-active azoles (75%) and for prophylaxis (71%). Inter-rater reliability was fair overall (0.3906), with moderate reliability amongst specialist pharmacists (0.5304). Barriers to accurate classification were incorrect use of the appropriateness matrix, knowledge gaps and lack of guidelines for some indications.

Conclusions

The AF-NAPS is a valid tool, assisting assessors to correctly classify appropriate prescriptions more accurately than inappropriate prescriptions. Specialist and AMS pharmacists had similar performance, providing confidence that both can undertake AF-NAPS audits to a high standard. Identified reasons for incorrect classification will be targeted in the online tool and educational materials.

Introduction

Invasive fungal infections (IFIs) most commonly affect immunocompromised and critically ill patients and recently have been recognized as an important complication of COVID-19.^1,2 IFIs are associated with high morbidity, mortality^3–5 and costly management.⁶ Studies of antifungal prescribing quality have found that 33%–71%^7–11 of prescriptions are considered inappropriate, increasing the risk of treatment failure, drug toxicity, drug–drug interactions (DDIs), and development of antifungal resistance.¹² As such, there is a need for antifungal stewardship (AFS) focusing on optimizing antifungal prescribing, IFI management and improving patient outcomes.

The Mycoses Study Group Education and Research Consortium published core recommendations for AFS in 2020.¹³ It was recommended that facilities evaluate the quality of antifungal prescribing on a systematic basis and use data-driven strategies to optimize AFS interventions. Several tools have been developed to assist facilities in evaluating the quality of antifungal prescribing^14–19 and studies of AFS interventions have employed a wide variety of quality metrics.^7,20–24 These metrics are broad and may not meet the needs of all healthcare facilities. Furthermore, it is not known if these metrics and audit methodologies are valid and reliable.

The Antifungal National Antimicrobial Prescribing Survey (AF-NAPS) was developed as a ‘deep-dive audit’ within the Australian Hospital National Antimicrobial Prescribing Survey (NAPS), an established platform to monitor appropriateness and guideline compliance via an annual point prevalence methodology.²⁵ The context for antifungal prescribing recognizes that IFIs are rare, and many patient and disease factors need to be considered in assessing antifungal prescribing quality. The AF-NAPS was co-designed and developed to address a lack of consensus AFS auditing tools. The metrics used in the AF-NAPS are based on the results of a Delphi consensus survey of international experts in the field of AFS to identify a core set of important and feasible metrics to assess antifungal prescribing practices.²⁶ The AF-NAPS assesses both compliance with guidelines and overall appropriateness. Appropriateness assessment allows for evaluation in terms of indication, drug, dose and route selection, allergy or drug–bug mismatch, and duration, as well as whether appropriate streamlining has occurred in terms of switching to a narrower spectrum agent or oral conversion where possible.²⁵ This is an important adjunct to assessment of compliance with guidelines as guidelines are not available for all indications and may be insufficient when taking into consideration the whole treatment episode, patient risk factors and the phases of IFI management. To support clinicians to utilize the AF-NAPS accurately and consistently, a prescribing appropriateness assessment matrix (Figure 1) and user guide were developed; however, the validity and reliability of the AF-NAPS tool and its supporting documents has not yet been characterized.

Figure 1.

Antifungal NAPS appropriateness assessment matrix. *Takes into account acceptable changes due to the patient’s weight, allergy status, renal or hepatic function, or relevant drug interactions (if this information is available).

The objective of this study was to assess the validity and reliability of the AF-NAPS used by clinicians to assess appropriateness of antifungal prescribing and compliance with guidelines. This study will direct the development of learning materials to ensure consistent and reliable use of the AF-NAPS for quality improvement initiatives as well as to guide the selection of suitable types of clinicians to undertake antifungal prescribing quality audits.

Materials and methods

Case vignette development

Authors A.K., M.S. and J.A. prepared case vignettes describing antifungal prescribing in four domains: haematology; intensive care; liver and lung solid organ transplant (SOT); and paediatrics. The cases included patient demographics, hospital admission, progress, discharge and outpatient notes, procedures as well as pathology, and imaging results. Details of antifungal prescribing including drug name, dose, route and duration were provided. A single case vignette contained one or more prescriptions for an individual patient over the course of their management. The vignettes were based on cases at the authors’ sites of practice; however, they were modified to ensure de-identification (i.e. change to patient gender, age) and, where required, realistic adjustments were made to case contents to allow variety in case type and prescribing appropriateness.

Definitions

Appropriateness of prescribing was defined as optimal, adequate, suboptimal, inadequate or not assessable according to the AF-NAPS appropriateness assessment matrix (Figure 1).

Appropriateness classifications are grouped into broader categories of appropriate (optimal or adequate) and inappropriate (suboptimal or inadequate), where an appropriate classification reflects acceptable management that maintains patient safety, while an inappropriate classification indicates unreasonable management that may compromise patient safety. Not assessable may be selected where the assessor has insufficient detail or feels the case is too complex to assess without the support of a multidisciplinary AFS team. Reasons for inappropriate prescribing were based on the AF-NAPS User Guide definitions (Appendix S1, available as Supplementary data at JAC Online). Compliance with guidelines was rated as compliant with specified 2021 Australasian consensus guidelines^27–35 and/or international guidelines,^36,37 non-compliant, no guideline available or not assessable (Appendix S2). Antifungal course type was defined as prophylaxis, empirical or directed therapy (Table 1).

Table 1.

Antifungal course type definition

Antifungal course type	Definition
Prophylaxis	Antifungal prescribed for the prevention of fungal infection, typically in an immunocompromised patient
Empirical therapy	Antifungal prescribed for a suspected IFI where there has not yet been confirmation of an IFI diagnosis based on microbiology, non-culture-based tests, microscopy or imaging investigation
Directed therapy	Antifungal prescribed for the treatment of a confirmed IFI based on the results of microbiology, non-culture-based tests, microscopy and/or imaging investigations conducted

Steering group

Four steering groups were assembled in each of the clinical domains. Members consisted of an infectious diseases (ID) physician, an AMS pharmacist, an additional ID or specialist physician and a clinical pharmacist specializing in the clinical area. Each steering group met via videoconference to review the cases. Vignettes were reviewed and modified based on steering group suggestions to permit a variety of case types and appropriateness classifications. The steering groups ensured that the cases remained within the realm of reality and thus retained complexity and may have fallen within the grey zone of guidelines, allowing for an accurate simulation of antifungal prescribing appropriateness assessment. The steering groups deliberated on the appropriateness, guideline compliance and reason for inappropriateness classification for each prescription based on the resources provided. The groups agreed on correct classifications for each prescription, which formed the gold standard against which participant responses were evaluated.

Participant recruitment

We aimed to recruit 28 participants, 4 of each assessor type: AMS pharmacist; adult ID physician; paediatric ID physician; haematology pharmacist; ICU pharmacist; SOT pharmacist; and paediatric pharmacist. Participants were identified via purposive sampling. An e-mail invitation was sent describing the study and requirements; invitees were asked to partake or suggest suitable colleagues. Up to two reminder e-mails were sent if invitees had not responded within 2 weeks. Invitations continued to be sent until the target number of completed surveys was achieved. Participants were offered a gift voucher on completion of the survey as a token of appreciation for their time and participation. Participants were not provided written or verbal training regarding use of AF-NAPS resources prior to undertaking the survey.

Prescription classification procedure

Participants who agreed to take part in the study were sent a link to a Redcap^® survey. The survey included modules for consent, demographics and case vignettes in each of the clinical domains. AMS pharmacists were assigned to assess prescriptions across all four domains. Adult ID physicians assessed prescriptions in all domains excluding paediatrics. Paediatric ID physicians assessed only the prescriptions in the paediatric domain and specialist pharmacists were asked to classify prescriptions within their area of specialty only. As such, each case was assessed by 12 participants (4 ID physicians, 4 AMS pharmacists and 4 specialist pharmacists). The case vignette modules contained the case description, AF-NAPS appropriateness assessment matrix, definitions for reasons for inappropriateness and links to relevant guidelines. Participants were required to independently rate each prescription for appropriateness and compliance with guidelines. Where a prescription was classified as inappropriate, participants were required to select one or more reasons for inappropriate prescribing. Participants were able to provide comments regarding each case vignette. Where a participant only partially completed the survey, their responses were excluded from the analysis.

Sample size determination

This study was a priori designed to be an entirely exploratory study. As such, it was not subject to a formal sample size estimate. A sample of 60 prescriptions and four of each assessor type was selected for calculation of accuracy, sensitivity, specificity, inter-rater reliability and associated CIs. This was consistent with sample size used in a comparable study assessing agreement between multiple observers.³⁸

Statistical analysis

Categorical variables were summarized using frequency and percentage. Categorical variables for appropriateness were grouped into broader categories of appropriate (optimal or adequate) and inappropriate (suboptimal or inadequate) (Figure 1). This grouping was made in line with standard national reporting for the Hospital NAPS. To calculate accuracy, sensitivity (correct identification of appropriate and guideline compliant prescriptions) and specificity (correct identification of inappropriate and guideline non-compliant prescriptions), the assessor’s classifications were compared with gold standard classifications made by the steering groups. Accuracy, sensitivity and specificity were summarized using point estimates and associated 95% CIs. Overlap of CIs was used to assess statistical significance. For analysis of appropriateness in prescribing, responses were excluded where a participant selected ‘not assessable’. For analysis of compliance with guideline, responses were excluded when either the steering group determined there was ‘no guideline available’ or the participant selected ‘no guideline available’ or ‘not assessable’.

Inter-rater agreement of categorical appropriateness and guideline compliance were assessed using Fleiss’ kappa. Fleiss’ kappa scores were interpreted as follows: 0.01–0.2 as slight agreement, 0.21–0.4 as fair agreement, 0.41–0.6 as moderate agreement, 0.61–0.8 as substantial agreement, and 0.81–1.0 as almost perfect agreement.³⁹ All classifications including ‘not assessable’ and ‘no guideline available’ were included in the inter-rater agreement analysis.

Qualitative analysis

Participant’s classification for appropriateness and reason for inappropriateness classifications were analysed in conjunction with free-text written responses for each case vignette. Author A.K. identified barriers and enablers to accurate classification and categorized these into key themes. The number of instances of each barrier or enabler were tallied.

All analyses were undertaken using Stata version 16.1 (StataCorp^®, College Station, TX, USA).

Results

Participant characteristics

Invitations were sent to 44 individuals, of whom 33 (75%) accepted. Twenty-eight participants (85%) completed the REDCap^® survey. Characteristics of participants are described in Table 2. We recruited participants from across Australia and New Zealand. Participants were primarily from the Australian state of Victoria (39.3%), followed by New South Wales (28.6%). Participants were primarily from principal referral (67.9%) or specialist women’s and children’s hospitals (21.4%) and all but one were from hospitals located in major cities (96.4%). One AMS pharmacist participant did not assess the paediatric case vignettes due to lack of experience in managing complex paediatric patients.

Table 2.

Participant characteristics

	ID physician n (%)	AMS pharmacist n (%)	Specialist pharmacist n (%)
Number of participants	8	4	16
Areas of specialtya
ȃAMS	5 (62.5)	4 (100.0)	6 (37.5)
ȃGeneral medicine	3 (37.5)	0 (0.0)	2 (12.5)
ȃHaematology	4 (50.0)	0 (0.0)	4 (25.0)
ȃID	8 (100.0)	2 (50.0)	5 (32.3)
ȃIntensive care	2 (25.0)	1 (25.0)	5 (32.3)
ȃMicrobiology	3 (37.5)	1 (25.0)	0 (0.0)
ȃOncology	4 (50.0)	0 (0.0)	3 (18.8)
ȃSOT	2 (25.0)	1 (25.0)	4 (25.0)
ȃSurgery	3 (37.5)	1 (25.0)	0 (0.0)
SOT specialtya
ȃHeart	1 (12.5)	1 (25.0)	2 (12.5)
ȃKidney	2 (25.0)	0 (0.0)	2 (12.5)
ȃLiver	1 (12.5)	0 (0.0)	4 (25.0)
ȃLung	0 (0.0)	1 (25.0)	5 (32.3)
Adult versus paediatric settinga
ȃAdult	4 (50.0)	4 (100.0)	12 (75.0)
ȃPaediatric	4 (50.0)	0 (0.0)	5 (32.3)
Years of professional practice
ȃ<5	5 (62.5)	1 (25.0)	3 (18.8)
ȃ5–10	3 (37.5)	2 (50.0)	3 (18.8)
ȃ11–19	0 (0.0)	0 (0.0)	8 (50.0)
ȃ>20	0 (0.0)	1 (25.0)	2 (12.5)
Confidence level in advising antifungal prescribing
ȃNot at all confident	0 (0.0)	0 (0.0)	0 (0.0)
ȃSlightly confident	2 (25.0)	0 (0.0)	3 (18.8)
ȃSomewhat confident	3 (37.5)	3 (75.0)	7 (43.8)
ȃQuite confident	2 (25.0)	1 (25.0)	5 (31.3)
ȃExtremely confident	1 (12.5)	0 (0.0)	1 (6.2)

AMS, antimicrobial stewardship.

Participants could select one or more responses to this question.

Antifungal prescription characteristics

Thirty case vignettes were prepared, incorporating 59 antifungal prescriptions. Prescriptions were for directed therapy (45.8%), empirical therapy (30.5%) or prophylaxis (23.7%). The antifungals and indications for use are described in Table 3.

Table 3.

Prescription characteristics by clinical domain

	Haematology n (%)	Intensive care n (%)	SOT n (%)	Paediatrics n (%)
Number of prescriptions	16	15	12	16
Course type
ȃDirected therapy	6 (37.5)	7 (46.7)	7 (58.3)	7 (43.8)
ȃEmpirical therapy	4 (25.0)	7 (46.7)	4 (33.3)	3 (18.8)
ȃProphylaxis	6 (37.5)	1 (6.7)	1 (8.3)	6 (37.5)
Antifungal
ȃEchinocandin	2 (12.5)	4 (26.7)	2 (16.7)	4 (25.0)
ȃFluconazole	1 (6.3)	5 (33.3)	1 (8.3)	4 (25.0)
ȃLiposomal amphotericin B	2 (12.5)	2 (13.3)	2 (16.7)	4 (25.0)
ȃPosaconazole	4 (25.0)	1 (6.7)	—	1 (6.3)
ȃVoriconazole	4 (25.0)	2 (13.3)	5 (41.7)	2 (12.5)
ȃOthera	3 (18.8)	1 (6.7)	2 (16.7)	1 (6.3)
Indication for antifungal prescribing
ȃCandida and Candida-like infection	2 (12.5)	3 (20.0)	4 (33.3)	3 (18.8)
ȃCryptococcus infection	—	2 (13.3)	—	3 (18.8)
ȃIntra-abdominal infection	—	5 (33.3)	—	—
ȃMould infection	8 (50.0)	4 (26.7)	7 (58.3)	4 (25.0)
ȃProphylaxis	6 (37.5)	1 (6.7)	1 (8.3)	6 (37.5)
Gold standard appropriateness classification
ȃAppropriate	9 (56.3)	7 (46.7)	5 (41.7)	9 (56.3)
ȃInappropriate	7 (43.8)	8 (53.3)	7 (58.3)	7 (43.8)
Gold standard guideline compliance classification
ȃCompliant with guidelines	10 (62.5)	7 (46.7)	5 (41.7)	8 (50.0)
ȃNon-compliant with guidelines	4 (25.0)	4 (26.7)	6 (50.0)	8 (50.0)
ȃNo guideline available	2 (12.5)	4 (26.7)	1 (8.3)	—

Other = 5-flucytosine, isavuconazole, terbinafine or no antifungal prescribed.

Accuracy of classification

Six hundred and seventy-seven assessments were included in the analysis of the validity of prescribing appropriateness assessment (Figure 2, Table 4). Overall accuracy was 77% (95% CI 73%–80%) with a sensitivity of 85.3% (95% CI 81.1%–88.8%) and specificity of 68% (95% CI 62.7%–73.0%). Prescriptions with the highest accuracy were those in the paediatric clinical domain [84% (95% CI 78%–89%)], for directed therapy [83% (95% CI 78%–88%)], for liposomal amphotericin B [82% (95% CI 73%–91%)] and for Cryptococcus spp. infection [98% (95% CI 95%–100%)]. Prescriptions with the lowest levels of accuracy were those in the haematology clinical domain [69% (95% CI 63%–75%)], prescribed for antifungal prophylaxis [71% (95% CI (64%–78%)] and for echinocandins [73% (95% CI 66%–80%)].

Figure 2.

Assessments included in validity analysis.

Table 4.

Accuracy of appropriateness classification

	Number of assessment	Accuracy % (95% CI)	Sensitivity % (95% CI)	Specificity % (95% CI)
Overall	677	77 (73–80)	85.3 (81.1–88.8)	68.0 (62.7–73.0)
By clinical domain
ȃPaediatrics	174	84 (78–89)	88.7 (80.6–94.2)	79.2 (68.5–87.6)
ȃSOT	140	77 (70–84)	81.4 (69.1–90.3)	72.8 (61.8–82.1)
ȃIntensive care	175	74 (68–81)	75.9 (65.3–84.6)	72.8 (62.6–81.6)
ȃHaematology	188	69 (63–75)	91.6 (84.6–96.1)	46.9 (35.7–58.3)
By antifungal course type
ȃDirected	312	83 (78–88)	88.7 (83.8–92.5)	76.9 (66.9–85.1)
ȃEmpirical	206	73 (66–79)	79.7 (68.3–88.4)	65.7 (57.1–73.6)
ȃProphylaxis	159	71 (64–78)	78.6 (65.6–88.4)	63.1 (53.0–72.4)
By drug class
ȃOthera	47	94 (86–100)	97.1 (85.1–99.9)	91.7 (61.5–99.8)
ȃAmphotericin B	114	82 (73–91)	82.6 (73.3–89.7)	81.8 (59.7–94.8)
ȃFluconazole	123	79 (72–86)	71.9 (58.5–83.0)	86.4 (75.7–93.6)
ȃMould-active azole	255	75 (71–80)	92.3 (85.4–96.6)	58.3 (50.0–66.2)
ȃEchinocandin	138	73 (66–80)	82.8 (70.6–91.4)	63.7 (52.2–74.2)
By indication group
ȃCryptococcus	57	98 (95–100)	95.7 (85.2–99.5)	100 (71.5–100)
ȃIntra-abdominal	58	85 (78–92)	100 (73.5–100)	69.6 (54.2–82.3)
ȃMould	267	77 (72–83)	86.5 (80.3–91.3)	68.3 (58.4–77.1)
ȃCandida and Candida-like	136	73 (66–81)	78.3 (66.7–87.3)	68.7 (56.2–79.4)
ȃProphylaxis	159	71 (64–78)	78.6 (65.6–88.4)	63.1 (53.0–72.4)
By assessor type
ȃSpecialist pharmacist	217	81 (75–86)	90.9 (83.9–95.6)	70.1 (60.5–78.6)
ȃAMS pharmacist	225	79 (74–84)	84.6 (76.8–90.6)	73.1 (63.8–81.2)
ȃID physician	235	71 (65–77)	80.7 (72.4–87.3)	61.2 (51.7–70.1)
By years of practice
ȃ>10	187	81 (75–86)	89.2 (81.1–94.7)	72.3 (62.2–81.1)
ȃ≤10	490	75 (71–79)	83.8 (78.7–88.1)	66.2 (59.8–72.2)
By confidence level
ȃQuite to extremely	228	77 (72–83)	88.5 (81.1–93.7)	66.1 (56.7–74.7)
ȃSlightly to somewhat	449	76 (72–80)	83.7 (78.3–88.2)	69.0 (62.4–75.1)

AMS, antimicrobial stewardship.

Other = 5-flucytosine, isavuconazole, terbinafine or no antifungal prescribed.

Five hundred and forty-four assessments were included in the validity analysis of compliance with guideline (Figure 2, Table 5). Overall accuracy was 76% (95% CI 73%–80%) with a sensitivity of 85.6% (95% CI 81.2%–89.2%) and specificity of 67.1% (95% CI 60.6%–73.2%). Patterns of accuracy closely followed those for appropriateness assessment. Prescriptions with the highest accuracy were those in the paediatric [78% (95% CI 72%–85%)] and SOT domains [78% (95% CI 71%–86%)], for directed therapy [83% (77%–89%)], for liposomal amphotericin B [85% (95% CI 76%–94%)] and for Cryptococcus infection [94% (95% CI 85%–100%)]. Prescriptions with the lowest levels of accuracy were those in the haematology clinical domain [71% (95% CI 63%–79%)], prescribed for antifungal prophylaxis [74% (95% CI (67%–81%)] and for mould-active azoles [69% (95% CI 63%–75%)].

Table 5.

Accuracy of compliance with guidelines classification

	Number of assessments	Accuracy % (95% CI)	Sensitivity % (95% CI)	Specificity % (95% CI)
Overall	544	76 (73–80)	85.6 (81.2–89.2)	67.1 (60.6–73.2)
By clinical domain
ȃPaediatrics	167	78 (72–85)	86.9 (77.8–93.3)	69.9 (58.8–79.5)
ȃSOT	123	78 (71–86)	84.5 (72.6–92.7)	72.3 (59.8–82.7)
ȃIntensive care	99	72 (63–82)	73.4 (60.9 -83.7)	71.4 (53.7–85.4)
ȃHaematology	155	71 (63–79)	92.0 (85.4 -96.3)	50.0 (34.2–65.8)
By antifungal course type
ȃDirected	243	83 (77–89)	92.0 (87.2–95.5)	74.5 (61.0–85.3)
ȃProphylaxis	147	74 (67–81)	83.3 (70.7–92.1)	65.6 (55.0–75.1)
ȃEmpirical	154	68 (60–75)	71.4 (60.0–81.2)	63.6 (51.9–74.3)
By drug class
ȃOthera	45	100 (100–100)	100 (89.7–100)	100 (71.5–100)
ȃAmphotericin B	92	85 (76–94)	84.5 (74.0–92.0)	85.7 (63.7–97.0)
ȃFluconazole	101	83 (75–90)	77.1 (62.7–88.0)	88.7 (77.0–95.7)
ȃEchinocandin	97	70 (61–79)	80.0 (65.4–90.4)	59.6 (45.1–73.0)
ȃMould-active azole	209	69 (63–75)	87.6 (80.4–92.9)	50.0 (39.1–60.9)
By indication group
ȃCryptococcus	57	94 (85–100)	97.8 (88.5–99.9)	90.9 (58.7–99.8)
ȃMould	202	75 (67–82)	84.6 (78.0–89.9)	65.2 (49.8–78.6)
ȃProphylaxis	147	74 (67–81)	83.3 (70.7–92.1)	65.6 (55.0–75.1)
ȃCandida and Candida-like	127	73 (66–81)	81.0 (69.1–89.8)	65.6 (52.7–77.1)
ȃIntra-abdominal	11	N/A	N/A	N/A
By assessor type
ȃSpecialist pharmacist	169	77 (70–83)	85.7 (77.2–92.0)	67.6 (55.5–78.2)
ȃAMS pharmacist	185	81 (75–86)	86.2 (78.3–92.1)	75.0 (63.7–84.2)
ȃID physician	190	72 (65–78)	84.8 (76.8–90.9)	59.0 (47.3–70.0)
By years of practice
ȃ>10	142	80 (74–87)	84.1 (74.4–91.3)	76.7 (64.0–86.6)
ȃ≤10	402	75 (71–79)	86.1 (81.0–90.2)	63.6 (55.8–71.0)
By confidence level
ȃQuite to extremely	188	78 (72–84)	88.6 (80.9–94.0)	67.5 (56.3–77.4)
ȃSlightly to somewhat	356	76 (71–80)	84.1 (78.5–88.7)	66.9 (58.5–74.6)

AMS, antimicrobial stewardship; N/A, not assessable.

Other = 5-flucytosine, isavuconazole, terbinafine or no antifungal prescribed.

Specialist and AMS pharmacists had the highest accuracy, followed by ID physicians. Participants with greater than 10 years of professional practice had higher accuracy compared with those with less than 10 years practice, although these differences were not statistically significant. Self-reported confidence in advising on the prescribing of antifungals for management and prevention of IFI was not associated with improved accuracy of classification.

Inter-rater reliability

All 692 assessments were included in the analysis of inter-rater reliability (Table 6). Overall inter-rater reliability for appropriateness and compliance with guidelines classification was rated as fair, with a kappa score of 0.3906 and 0.3409, respectively. In terms of appropriateness classification, specialist pharmacists achieved moderate reliability (0.5304) compared with fair reliability amongst ID physicians and AMS pharmacists (0.3060–0.3671). Reliability was moderate for prescriptions in the paediatric domain (0.4715) and fair for the other specialist domains (0.2845–0.3921). Prescriptions for prophylaxis (0.4483) and directed therapy (0.4324) had higher reliability than empiric courses (0.2300). Reliability was almost perfect for management of Cryptococcus spp. infection (0.8170) and fair across common antifungal drug classes (0.2745–0.3732), as well as prescriptions for mould and Candida and Candida-like infections (0.3070–0.3202). Reliability for compliance with guidelines classification followed similar patterns to those of appropriateness classification (Table 6).

Table 6.

Inter-rater reliability of appropriateness and guideline compliance classifications

	Appropriateness		Guideline compliance
	Kappa	Interpretation	Kappa	Interpretation
Overall	0.3906	Fair	0.3409	Fair
By clinical domain
ȃPaediatrics	0.4715	Moderate	0.4154	Moderate
ȃHaematology	0.3921	Fair	0.3158	Fair
ȃSOT	0.3143	Fair	0.2625	Fair
ȃIntensive care	0.2845	Fair	0.2093	Fair
By antifungal course type
ȃProphylaxis	0.4483	Moderate	0.4312	Moderate
ȃDirected	0.4324	Moderate	0.3798	Fair
ȃEmpirical	0.2300	Fair	0.1633	Slight
By drug class
ȃOthera	0.7758	Substantial	0.7970	Substantial
ȃMould-active azole	0.3732	Fair	0.2427	Fair
ȃFluconazole	0.3732	Fair	0.3916	Fair
ȃAmphotericin B	0.3146	Fair	0.2946	Fair
ȃEchinocandin	0.2745	Fair	0.2679	Fair
By indication group
ȃCryptococcus	0.8170	Almost perfect	0.7746	Substantial
ȃProphylaxis	0.4483	Moderate	0.4312	Moderate
ȃMould	0.3202	Fair	0.1834	Slight
ȃCandida and Candida-like	0.3070	Fair	0.3414	Fair
ȃIntra-abdominal	0.2520	Fair	0.1176	Slight
By assessor type
ȃSpecialist pharmacist	0.5304	Moderate	0.4354	Moderate
ȃAMS pharmacist	0.3671	Fair	0.3751	Fair
ȃID physician	0.3060	Fair	0.2586	Fair

AMS, antimicrobial stewardship; N/A, not assessable.

Other = 5-flucytosine, isavuconazole, terbinafine or no antifungal prescribed.

Barriers to accurate prescription classification

Key barriers to the accurate classification of prescription appropriateness were identified (Table 7). The most common barrier was limitations to a participant’s knowledge in specific areas, e.g. drug dosing in hepatic or renal impairment, fluconazole dosing for the treatment of sensitive dose-dependent yeast organisms, DDIs, requirement for loading dose and drug penetration to infection sites. Incorrect use of the AF-NAPS’ appropriateness matrix, user guide or selection of accurate reason for inappropriateness were common impediments, with participants commonly selecting the ‘appropriate’ classification, despite comments correctly identifying one or more reasons for inappropriate prescribing. The lack of national or international guidelines for several indications, or a participant’s difficulty in interpretating guidelines were noted as barriers to classification. The availability of clear guidelines was noted as an enabler, which assisted in prescription classification.

Table 7.

Barriers to accurate prescription classification

	Number of instances noted
Barrier	Total	ID physician	AMS pharmacist	Specialist pharmacist
Assessor knowledge in specific area(s)	37	14	8	15
Incorrect use of AF-NAPS appropriateness assessment matrix and/or user guide	19	6	5	8
Specific case detail(s) not identified	17	6	6	5
More than one appropriate management strategy	9	5	2	2
No guideline available	7	—	2	5
Familiarity with prescribing area	6	1	5	—
High case complexity	6	3	3	—
Interpretation of reason for inappropriateness	4	2	—	2
Difficulty in interpreting guidelines	4	1	1	2
Assessment based on local practice	3	2	—	1
Incorrect selection or interpretation of guidelines	2	1	1	—
Incorrect case interpretation	2	1	1	—
Total	116	42	34	40

AMS, antimicrobial stewardship.

It was noted, in 15 instances, that the participant’s prescription appropriateness classification accounted for additional valid factors not considered by the steering group in their assessment. Examples included voriconazole dosing in obesity and several DDIs.

Discussion

To the authors’ knowledge, this is the first study to undertake a comprehensive analysis of a tool to systematically assess appropriateness of antifungal prescribing and compliance with guidelines, validating against gold standard assessments and measuring inter-rater reliability.

Validity and reliability of the AF-NAPS tool

Results of this study indicate that clinicians using the AF-NAPS tool have reasonable accuracy in classifying prescription appropriateness. Sensitivity was higher than specificity, indicating that clinicians were more easily able to correctly identify appropriate rather than inappropriate prescriptions. In many circumstances, clinicians identified a reason for inappropriate prescribing such as incorrect dose, duration, or route within their written comments, but failed to apply the AF-NAPS appropriateness matrix correctly where any such reason(s) should have resulted in an ‘inappropriate’ classification.

In this study we found very similar rates of accuracy for compliance with guidelines when compared with appropriateness classifications. This was contrary to our hypothesis that classification of compliance to guidelines may show higher accuracy due to the objective nature of such classifications, given that participants were provided with links to relevant guidelines. Results may have been influenced by the need to exclude classifications of ‘not assessable’ and ‘no guideline available’ from the analysis, as discussed in the limitations.

Specialist and AMS pharmacists appeared to have the highest accuracy and reliability of classifications, despite AMS pharmacist participants infrequently identifying the specialist clinical domains as their areas of speciality. Additionally, AMS pharmacists expressed lower confidence in providing advice regarding prescribing of antifungal agents. However, AMS pharmacists likely have greater familiarity with the Hospital NAPS tool to support antimicrobial prescribing quality audits, hence increasing accuracy of classification. These findings provide confidence that both specialist and/or AMS pharmacists will be able to undertake AF-NAPS audits effectively. ID physicians appeared to have lower accuracy due to incorrect use of the provided AF-NAPS supporting documents and knowledge gaps in areas of antifungal DDIs and drug pharmacokinetics, domains which are typically overseen by pharmacists.

Directed antifungal therapy, particularly for treatment of Cryptococcus spp. Infection, achieved high accuracy and almost perfect inter-rater agreement. This was an expected result due to the availability of clear management guidelines for Cryptococcus infection, which reduce the subjectivity of classification by assessors. Conversely, prescriptions for antifungal prophylaxis achieved lower accuracy, likely due to perceived ambiguity of guidelines³⁰ and a lack of strong data, in some areas, for the assessment of IFI risk and optimal choice of agents. This was supported by the qualitative analysis of participant responses, noting difficulty in interpreting guidelines and lack of familiarity with the clinical domain as barriers to correct classification.

Echinocandin prescriptions achieved the lowest accuracy of all antifungal classes. Noted reasons for incorrect classification included knowledge gaps in drug pharmacokinetics, namely caspofungin use and dosing in hepatic impairment, as well as drug penetration into the urinary tract and CNS. Prescriptions in the haematology domain achieved the lowest accuracy; this is thought to be partly due to specific case details not being identified by the majority of participants in two of the haematology prescriptions. One of these missed case details was the requirement for IV therapy for antifungal prophylaxis for a high-risk patient who was receiving parenteral nutrition due to grade III mucositis. It is also possible that clinicians may have had a preference to perform therapeutic drug monitoring prior to switching to an IV route, hence they considered this prescription appropriate based on provided case details. The large number of incorrect classifications for these two prescriptions may have skewed the overall accuracy within this clinical domain.

Comparison with the literature

A 2016 study evaluated the reliability of antimicrobial prescribing quality assessments made by remote hospital assessors with different levels of clinical ID expertise using the Hospital NAPS.⁴⁰ While this study did not focus solely on antifungal agents, these results were consistent with our findings, reflecting either fair or slight agreement, with none achieving moderate agreement or higher. Contrary to our findings, guideline compliance assessments did achieve higher inter-rater reliability than appropriateness classification, with moderate agreement (0.41–0.6) amongst several assessor groups.

A Dutch study investigated how ID physicians’ assessments of antibacterial prescribing quality compared with a reference standard.³⁸ ID physicians agreed with the reference standard in 80% of cases, with a sensitivity and specificity of 75% and 84%, respectively. These results are also consistent with our findings, although with a higher specificity, indicating that experts appeared to find it easier to decide whether a prescription was inappropriate than whether it was appropriate. This may be due to greater clarity in guidelines for antibacterial prescribing, less need for risk stratification and greater confidence in antibacterial compared with antifungal prescribing.

Improving accuracy of classification using the AF-NAPS tool

The current study has identified common barriers to correct classification, many of which can be addressed to optimize clinicians’ use of the AF-NAPS. Firstly, we will limit the selection of appropriateness classification in the online tool, such that when one or more reason(s) for inappropriate prescribing are selected, users will not be permitted to select ‘optimal’ and will receive a warning if they intend to select ‘adequate’. For example, where an assessor has identified that a prescription dosage is incorrect but wishes to classify the prescription as ‘optimal’, the online tool will prevent this selection by prompting reassessment and selection of a more suitable classification (i.e. ‘suboptimal’ or ‘inadequate’). Secondly, participants in the study were not provided with any education prior to undertaking the assessments. The AF-NAPS tool will be accompanied by educational materials including a mandatory eLearning module, video demonstrations and worked case examples. Common knowledge gaps identified in this study will be incorporated in the educational materials. Lastly, we believe that a multidisciplinary AFS team incorporating AMS and/or specialist pharmacists, as well as ID physicians, will ensure that experts with knowledge around infection risk, diagnostics, drug pharmacokinetics and DDIs work together to comprehensively assess each prescription and determine the most suitable classifications. The study by Cotta et al.⁴⁰ found that AMS teams had the greatest reliability in performing appropriateness assessments compared with clinical pharmacists or infection control practitioners alone.

Limitations

Our study had some limitations. Firstly, we included an option to select ‘not assessable’ for both appropriateness and guideline compliance, meaning not all prescriptions were included in the analysis, possibly impacting on findings. Notably though, only a small proportion of assessments (2%) were excluded for this reason. Secondly, we assessed individual clinicians’ assessment and did not assess the benefit of a multidisciplinary AMS team approach to assessment. This is a potential area for future study as has been done in other settings.⁴⁰ Thirdly, it was not mandatory for participants to include written remarks regarding each prescription classification, limiting qualitative assessments only to those prescriptions where interpretable comments were documented. Additionally, the gold standard classifications as determined by the steering groups may not represent the single true classification. As was identified by the qualitative analysis, there were instances in which the participants identified valid additional reasons for inappropriate prescribing, not accounted for by the steering group. This highlights the complexity and nuance of assessing quality of prescribing for antifungal agents.

We acknowledge that bias may have been introduced by the purposive sampling methodology and gifting of a voucher to participants. A small number of clinicians participated in the study, many of whom had less than 5–10 years professional practice. These participants were selected as such physicians are commonly tasked with undertaking quality improvement activities; however, this may limit generalizability to settings where more experienced clinicians routinely undertake such work. Lastly, the study included a greater number of pharmacists than physicians, which reflects the practice in Australian principal referral and specialist hospitals, in which AMS and specialist clinical pharmacists are commonly employed and frequently participate in quality improvement activities. Similar clinicians may not be available as standard at all healthcare facilities, hence a multidisciplinary model of AFS may not be feasible in all settings, potentially further limiting generalizability.

Conclusions

We have found that the AF-NAPS is a valid tool, with assessors able to correctly classify appropriate prescriptions more easily than they are able to classify inappropriate prescriptions. Inter-rater reliability was only fair across the majority of prescription types and domains, highlighting the inherent challenges in assessing antifungal prescribing. The knowledge gaps observed provide additional weight to the need for AFS programmes that include antifungal restrictions and approvals, post-prescription review and safety monitoring. Specialist and AMS pharmacists had similar performance, providing support for both to perform the AF-NAPS to a high standard, despite AMS pharmacists’ lower confidence and experience in speciality areas. We recommend that the AF-NAPS is undertaken in a multidisciplinary team incorporating both physicians and pharmacists to ensure adequate expertise in all aspects of antifungal prescribing. Identified reasons for incorrect classification and knowledge gaps will be targeted in the AF-NAPS online tool and educational materials to ensure consistent and reliable use of the AF-NAPS, which has been available to all Australian hospitals since October 2022.

Acknowledgements

We gratefully acknowledge the following steering group members for their valuable contribution: Dr Gabrielle Haeusler, Ms Emily Harding, Dr Ashley Ng, Ms Janelle Penno, Dr Olivia Smibert and Dr Michelle Yong. We thankfully acknowledge the following survey participants for their contribution: Belinda Badman, Alison Boast, Jessica Bui, Olivia Bupha Intr, Coen Butters, Maggie Chau, Kaman Cheung, Zoy Goff, Gabrielle Grosfeld, Juliet Ho, Karlee Johnston, Jessica Joseph, Caitlin Keighley, Tony Lai, Julian Lindsay, Brendan McMullan, Josh Osowicki, Matt Rawlins, Gemma Reynolds, Lucy Sharrock, Grace Shaw, Nikhil Sing, Amanda Tey, Rachel Thorson, Karen Urbancic, Sanchia Warren, Heather Weerdenburg and Paul Williams.

Funding

This work was funded by the National Health and Medical Research Council (NHMRC) project grant (Grant number: 1156426).

Transparency declarations

A.P.D. has received an educational grant from Gilead, with payment to their institution. D.C.M.K. is a recipient of a research grant from F2G that is unrelated to the current work. D.K.Y. has acted as a consultant for an education workshop organized by Gilead Sciences. M.A.S. has received educational grants from Merck, Gilead and F2G and sat on advisory boards for Roche, F2G, Cidara, Pfizer, Gilead and Takeda, with all payments made to their institution. All other authors: none to declare.

Supplementary data

Appendices S1 and S2 are available as Supplementary data at JAC Online.