Pediatric Infectious Diseases Milestones: A Step in the Right Direction to Evaluate Subspecialty Learners

Abstract

We share the work of the ACGME Pediatric Infectious Diseases Working Group in creating the Pediatric Infectious Diseases-Specific Milestones and discuss key considerations that lead to the reformation of competencies to better assess learners in Pediatric Infectious Diseases.

BACKGROUND ON COMPETENCY-BASED MEDICAL EDUCATION

Over the course of the last century, graduate medical education has largely moved from a time-based prescriptive model to one emphasizing the acquisition of foundational knowledge, skills, and values by trainees, in line with Benjamin Bloom’s domains of learning [1]. This focus on outcomes (competency), rather than time in training, aims to ensure that all learners achieve a minimum level of competency in preidentified areas before graduating to independent practice in their chosen field.

The Accreditation Council for Graduate Medical Education (ACGME), a private, nonprofit organization that accredits residency and fellowship training programs, collaborated with the American Board of Medical Specialties (ABMS) to introduce a competency-based medical education (CBME) structure to the U.S. training system in 1999, identifying six core domains of competency for physicians [2, 3]. These core competencies included patient care (PC), medical knowledge (MK), practice-based learning and improvement (PBLI), interpersonal and communication skills (ICS), professionalism (PROF), and systems-based practice (SBP).

In 2013, with the introduction of the first iteration of the Milestones, each competency was further divided into subcompetencies reflecting key areas within each competency that could be distinctly evaluated (Figure 1). Some subcompetencies were identified as necessary skills for all physicians, irrespective of specialty or subspecialty, referred to as “harmonized” and used in all graduate medical training programs irrespective of specialty. The use of other subcompetencies varied according to the medical specialty being accredited. Within each subcompetency, Milestones were identified as the measurable performance outcomes of the learner; these were rooted in Dreyfus’ five-stage developmental model, where each learner begins as a novice and eventually grows into an expert in each individual skill (Figure 1) [2, 3]. In this way, learners’ progress across various themes could be tracked and interventions or remediation be provided in particular areas as needed to ensure adequate achievement in all areas by the time of completion of training.

Figure 1.

Anatomy of the Pediatric Infectious Diseases Milestones 2.0 Document (adapted from ACGME, 2023) [4].

In 2009, the ACGME and the American Board of Pediatrics (ABP) began working on the Pediatric Milestones project [5]. In 2014, the first iteration of the Pediatric Subspecialty Milestones (PSM), version 1.0, was shared between various subspecialties within the field of pediatrics, ranging from child abuse and adolescent medicine to pediatric critical care and pediatric transplant hepatology [6].

PSM 1.0 retained the original six core competencies, as well as harmonized subcompetencies that were shared with general pediatrics (Table 1) [2]. Subcomptencies that were developed exclusively for the pediatric subspecialty learner included role modeling (PC4), appraisal and assimilation of scientific studies to inform patient management (MK1), risk-benefit analysis (SBP3), identifying system errors and implementing solutions (SBP5), use of information technology (PBLI3), education of the medical team and patients (PBLI4), leadership skills (PROF3), and working as a team member (ICS2) and in a consultant role (ICS3) (Table 1) [6, 7]. These subcompetencies, while important for general subspecialty care, did not assess proficiencies unique to the Pediatric Infectious Diseases (PID) learner, such as infection prevention; nuances in management related to antibiotic stewardship; care of individuals with dysregulated immune function; and role in public health. Furthermore, designated institutional officials shared concerns regarding the utility of Milestones 1.0 when used by educators and learners as an assessment and feedback tool.

Table 1.

Pediatric Milestones and Pediatric Subspecialty Milestones 1.0 (Adapted From ACGME) [4, 6, 7]

Competency Domain	Pediatric Milestone Subcompetency Description (2012)	Pediatric Subspecialty Milestone Subcompetency Description (2014)	Pediatric Infectious Diseases Milestones (2023)
Patient care	PC1. Gather essential and accurate information about the patient	PC1. Provide transfer of care that ensures seamless transitions	PC1: History and physical examination
	PC2. Organize and prioritize responsibilities to provide patient care that is safe, effective, and efficient	PC2. Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment	PC2: Organization and prioritization of patient care
	PC3. Provide transfer of care that ensures seamless transitions	PC3. Develop and carry out management plans	PC3: Diagnostic reasoning
	PC4. Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment	PC4. Provide appropriate role modeling	PC4: Consultative care
	PC5. Develop and carry out management plans		PC5: Management of patients with possible and proven infectious diseases
Medical knowledge	MK1. Critically evaluate and apply current medical information and scientific evidence for patient care	MK1. Locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems	MK1: Pathophysiology and foundational science
			MK2: Diagnostic evaluation and stewardship (consideration of priorities, risks, benefits, costs, and consequences)
			MK3: Treatments including source control, anti-infectives, immunoprophylaxis, and adjunctive therapies
			MK4: Infection control/prevention and epidemiology
			MK5: Public health
			MK6: Antimicrobial stewardship
Systems-based practice	SBP1. Coordinate patient care within the health care system relevant to their clinical specialty	SBP1. Work effectively in various health care delivery settings and systems relevant to their clinical specialty	SBP1: Patient safety
	SBP2. Advocate for quality patient care and optimal patient care systems	SBP2. Coordinate patient care within the health care system relevant to their clinical specialty	SBP2: Quality improvement
	SBP3. Work in interprofessional teams to enhance patient safety and improve patient care quality	SBP3. Incorporate considerations of cost awareness and risk-benefit analysis in patient and/or population-based care as appropriate	SBP3: System navigation for patient centered care—coordination of care
		SBP4. Work in interprofessional teams to enhance patient safety and improve patient care quality	SBP4: System navigation for patient-centered care—transitions in care
		SBP5. Participate in identifying system errors and implementing potential systems solutions	SBP5: Population and community health
			SBP6: Physician role in health care systems
Practice-based learning and improvement	PBLI1. Identify strengths, deficiencies, and limits in one’s knowledge and expertise	PBLI1. Identify strengths, deficiencies, and limits in one’s knowledge and expertise	PBLI1: Evidence-based and informed practice
	PBLI2. Identify and perform appropriate learning activities to guide personal and professional development	PBLI2. Systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement	PBLI2: Reflective practice and commitment to personal growth
	PBLI3. Systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement	PBLI3. Use information technology to optimize learning and care delivery
	PBLI4. Incorporate formative evaluation feedback into daily practice	PBLI4. Participate in the education of patients, families, students, residents, fellows, and other health professionals
Professionalism	PROF1. Humanism: Compassion, integrity, and respect for others; based on the characteristics of an empathetic practitioner	PROF1. Professional Conduct: High standards of ethical behavior which includes maintaining appropriate professional boundaries	PROF1: Professional behavior
	PROF2. Professionalization: A sense of duty and accountability to patients, society, and the profession	PROF2. Trustworthiness that makes colleagues feel secure when one is responsible for the care of patients	PROF2: Ethical principles
	PROF3. Professional Conduct: High standards of ethical behavior which includes maintaining appropriate professional boundaries	PROF3. Provide leadership skills that enhance team functioning, the learning environment, and/or the health care delivery system/environment with the ultimate intent of improving care of patients	PROF3: Accountability/conscientiousness
	PROF4. Self-awareness of one’s own knowledge, skill, and emotional limitations that leads to appropriate help-seeking behaviors	PROF4. The capacity to accept that ambiguity is part of clinical medicine and to recognize the need for and to utilize appropriate resources in dealing with uncertainty	PROF4: Well-being
	PROF5. Trustworthiness that makes colleagues feel secure when one is responsible for the care of patients
	PROF6. Recognize that ambiguity is part of clinical medicine and to recognize the need for and to utilize appropriate resources in dealing with uncertainty
Interpersonal skills and communication	ICS1. Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds	ICS1. Communicate effectively with physicians, other health professionals, and health-related agencies	Interpersonal and communication skills 1: Patient- and family-centered communication
	ICS2. Demonstrate the insight and understanding into emotion and human response to emotion that allows one to appropriately develop and manage human interactions	ICS2. Work effectively as a member or leader of a health care team or other professional group	Interpersonal and communication skills 2: Interprofessional and team communication
		ICS3. Act in a consultative role to other physicians and health professionals	Interpersonal and communication skills 3: Communication within health care systems

Review of the ACGME Milestones 1.0 by specialty and subspecialty leaders over the first years after implementation revealed issues across several common themes, including the complexity of language for each Milestone, the number of subcompetencies and Milestones, and the inclusion of multiple variations of the same Milestone, resulting in redundancy [8, 9]. This was in part addressed by the harmonization of Milestones across four competencies (ICS, PBLI, PROF, and SBP), revised in 2017. Other goals that were identified as part of the iterative process that began in 2016 to build Milestones 2.0 included the need to limit each subcompetency to three Milestones, to ensure a complete developmental process is followed by each Milestone across the five levels for a given subcompetency, and to create (sub)specialty-specific Milestones [8, 9]. In response to the call for subspecialty-specific Milestones, the ACGME invited PID members, including program directors, ABP representatives, and faculty and fellow volunteers, to form the PID Working Group who would reform the PSM 1.0 to create tailored PID Milestones. Here, we share the process behind the development of, highlight the notable changes to, and review the initial public response to the PID Milestones.

CREATION OF THE PID MILESTONES WORK GROUP

An independent Working Group of subject matter experts was convened by the ACGME in April of 2022 to develop the Pediatric Infectious Diseases Milestones (Milestones 2.0). This included medical educators, fellowship program directors, and three current fellows, some serving in a voluntary capacity and others appointed as representatives of important stakeholders, including the American Board of Pediatrics, the Association of Pediatric Program Directors, the Council of Pediatric Subspecialties, the Pediatric Infectious Diseases Society Training Program Committee (PIDS TPC), and the American Society for Transplantation. Together, the group represented diverse levels of experience, areas of career focus, and geographic locations. Work was facilitated by the ACGME Vice President for Milestones Development and coordinated by a dedicated Project Manager. The Working Group met in a series of virtual meetings and a 2-day hybrid in-person or virtual forum over the course of 11 months, completing additional tasks independently or in small groups between meetings using a shared OneDrive (Microsoft OneDrive, Redmond, WA). Group members received comprehensive resources for review prior to meetings.

PRINCIPLES GUIDING THE DEVELOPMENT OF MILESTONES 2.0

The goal of competency-based medical education is to improve the patient and clinician experience and provide better outcomes at lower health care costs. The Milestones 2.0 developmental process required the group to identify and understand developers’ knowledge and assumptions about the role of the Milestones in this effort, identify critical subcompetencies within the core competencies, and standardize expectations for fellows’ performance across each subcompetency (a “theme” or developmental trajectory). The recently revised language of the harmonized Milestones for ICS, PBLI, PROF, and SBP was not changed in order to provide greater consistency in evaluations across specialties, facilitate sharing and validation of assessment tools, and simplify faculty development [10]. The PID Milestones Working Group did have the option to add new subcompetencies to these, however, to reflect unique aspects of the subspecialty’s practice. The group considered existing Milestones (Pediatric, Pediatric Subspecialty, and Internal Medicine Infectious Diseases Milestones), entrustable professional activities (EPAs), board certification and licensing requirements, best practices, models of skills acquisition, and the potential for future use in assessment tools when developing the PC and MK subcompetencies [11, 12].

The Working Group first ensured there was collective agreement on themes within each subcompetency and expectations for skills at the start, middle, and end of training. An important goal was simplifying and clarifying the language and content, including removing educational jargon that was confusing to some users. The number of subcompetencies and the included themes were reviewed to ensure that each Milestone, when used collectively, accurately reflected the breadth of the specialty and was specific enough to permit users to understand the competency in the context of subspecialty practice. Within each subcompetency, skills that are learned over the course of fellowship were developed as themes. Language for levels 1-4 focused on this developmental trajectory, rather than time in training. Where feasible, a fifth level was included; as is the case in all Milestones, this highest level was intended to be aspirational, to promote lifelong learning which is an expectation of all pediatricians and pediatric subspecialists.

A subspecialty-specific Supplemental Guide was created simultaneously with the Milestones. This was intended to provide more detailed information to facilitate the understanding and consistency of assessments across individual evaluators and Clinical Competency Committees. Included in the Supplemental Guide were subspecialty-specific level examples, suggested assessment tools, and other resources. In some cases, several examples for how to assess an individual subcompetency were provided (Figure 2). This was done to recognize the differences in common patient diagnoses, illness severity, patient complexity, and presence of specialty clinics or units (eg, HIV clinic, bone marrow transplant unit) at the various institutions in which PID fellows train.

Figure 2.

Utilizing the supplemental guide in the subcompetency assessment for a learner, example using PC5.

CREATION OF NOVEL PID-SPECIFIC SUBCOMPETENCIES

In the PC and MK domains, all subcompetencies from the General Pediatrics Milestones 2.0 (2021) were revised through the lens of PID practice and, where deficits were identified, new subcompetencies were created. Within the PC domain, history and physical examination were combined (PC1) into one subcompetency and diagnostic reasoning (PC3) was emphasized. Consultative care was added (PC4), and learners’ educational progression regarding management of patients with infectious concerns was clarified (PC5). Within the MK domain, completely new subcompetencies were created to assess the learner’s comprehension of basic pathophysiology (MK1); diagnostic evaluation and stewardship (MK2); treatment strategies, including atypical pathogens and unique hosts (MK3); infection prevention practices (MK4); public health practices (MK5); and antimicrobial stewardship practices (MK6).

PUBLIC COMMENT ON THE PID MILESTONES 2.0

After completion of the Working Group’s revisions, public review and comment on the proposed Milestones were solicited through an electronic survey created by the ACGME. A link to the survey was included in an ACGME e-communication to the GME community and sent to designated institutional officials, institutional coordinators, program directors, and program coordinators by the PIDS TPC. These groups were encouraged to forward the request to other stakeholders. The survey, which was available for 3 weeks, asked respondents to consider whether the Milestone language for each subcompetency represented a realistic trajectory of knowledge, skills, and behaviors; whether the Milestones discriminated between meaningful levels of ability; whether the subcompetency could be effectively assessed; and if the Supplemental Guide was a useful resource. Respondents indicated their agreement or disagreement via a 4-point Likert Scale (Strongly Agree, Agree, Disagree, Strongly Disagree). An additional question asked if respondents believed the Milestone set could be used effectively in their educational setting. Free text comments were also collected.

The majority of the feedback from a total of 14 respondents was positive (Table 2), with most of the survey participants stating that the PID Milestones were a marked improvement to the prior iteration, had been tailored to PID learners, and would likely be useful in assessing trainees. Of the 26 subcompetencies that respondents considered over the 4 domains (104 Likert scale aggregate responses), there were two instances where the percentage of respondents who “agreed or strongly agreed” was less than 80%. One was with respect to Patent Care 3, Diagnostic Reasoning, where there was 75% agreement that the Supplemental Guide was a useful resource in understanding this Milestone set. The other involved Systems-Based Practice 6, Physician Role in Health Care Systems, but this was one of the harmonized Milestones common to all physician specialties, and thus was not eligible for revision by our Working Group.

Table 2.

Public Comment Feedback on Pediatric Infectious Diseases Milestones 2.0

		Subcompetency Responses from Survey Respondents (n = 14) in 4 Domains; % Strongly Agree + Agreea
Competency	Subcompetency	Realistic Progression of Knowledge (%)	Discriminates Level (%)	Can Assess Effectively (%)	Useful Supplemental Guide (%)
Patient care	History and Physical Exam	100	100	100	100
	Organization and Prioritization of Patient	100	100	100	100
	Diagnostic Reasoning	100	100	92.3	75
	Consultative Care	100	100	100	100
	Management of Patients	100	92.3	92.3	100
Medical knowledge	Pathophysiology and Foundational Science	91.7	91.7	91.7	100
	Diagnostic Evaluation and Stewardship	100	100	100	100
	Treatments	100	100	83.34	100
	Infection Control/Prevention and Epidemiology	100	100	100	100
	Public Health	100	91.7	91.7	90.9
	Antimicrobial Stewardship	100	91.7	100	100
Systems-based practice	Patient Safety	100	100	100	100
	Quality Improvement	100	100	100	100
	System Navigation-Patient Centered Care Coordination of Care	80	100	100	100
	System Navigation-Patient Centered Care Transitions	90	100	100	100
	Population and Community Health	100	100	100	100
	Physician Role in Health Care Systems	80	77.8	88.9	100
Practice-based learning improvement	Evidence-Based & Informed Practice	100	100	100	100
	Reflective Practice and Commitment to Personal Growth	100	100	100	100
Professionalism	Professional Behavior	100	80	100	100
	Ethical Principles	100	100	100	100
	Accountability/ Conscientiousness	100	100	100	100
	Well-Being	90	80	80	100
Interpersonal and communication skills	Patient and Family-Centered Communication	100	100	100	100
	Interprofessional and Team Communication	90	90	90	100
	Communication within Health Care Systems	100	100	100	100

^aNot all respondents provided responses to all questions (range 8-14).

It is difficult to draw themes from the free text comments since only 14 people responded to the survey and not all left comments. However, two respondents indicated that there were too many milestones, one commented that the same example should be used across all five levels in the supplemental guide to depict learner growth, and another commented that the supplementary guide should be limited to have less examples. The responses were reviewed by the PID Milestones 2.0 Working Group, the Milestones 2.0 draft was edited to reflect this input, and the final document was published for implementation in July 2023.

APPLICATION AND RELEVANCE OF MILESTONES 2.0 IN THE ASSESSMENT OF PID LEARNERS

The development of the PID Milestones presents a significant improvement in the opportunity to operationalize assessment of PID fellows nationally, using a standardized tool specifically developed to evaluate the skills needed for their successful future practice. The previous subspecialty Milestones sought to find common-ground skills that could be assessed across multiple pediatric subspecialties. In contrast, the new PID Milestones 2.0 contains both harmonized Milestones (common to all subspecialties) and Milestones developed to assess unique aspects of the PID profession.

To accurately assess the full scope of PID work, some expansion in the number of subcompetencies needed to occur. This was important for two reasons: first, a successful career in PID requires competency in several domains not common to all physician practices. These domains include antimicrobial and diagnostic stewardship, epidemiology and infection control, care for immunocompromised populations, and collaboration with public health. The need to evaluate skills in these important dimensions of PID physician practice prompted the addition of several subcompetencies. Second, assessing each of these domains in separate subcompetencies provides learners with a more authentic description of their skill level in each of these areas. Lumping skills into a single subcompetency for assessment creates problems for the assessor as well as the learner. For the assessor, what score should be given for a fellow who performs highly in one skill, but below average in a second skill that are both contained within the same assessment category? Should the skill levels be averaged to determine the score? Should the lowest score serve as the foundation for assessment? Regardless of how assessment is handled in categories where multiple skills are combined, the assessment becomes less reflective of reality. For the learner, the resultant summated assessments lack clarity on discrete areas of improvement and therefore less helpful to serve the learner as a guide on how to better advance in their training. Understanding where improvement is needed is difficult for the learner when several skills are combined together. For fellowship directors, who are faced with having to create remediation plans for fellows who are not achieving competencies in one or more areas, having well-defined and specific subcompetency assessment categories will more clearly describe what the fellow is struggling with, which will allow for the creation of a targeted and specific remediation plan. Hence, although some public comments suggested consolidating some Milestones, the need to reliably assess each of these subcompetencies separately and at different skill levels resulted in the shared consensus by the Working Group to retain the subcompetencies as proposed in Milestones 2.0.

Though the public commentary expressed some difficulty with discriminating between levels for one of the harmonized Milestones originally created with contributions from nonphysician, expert members of the Milestone Working Groups and carried forward from the core pediatric competencies, these remained unchanged in Milestones 2.0. These harmonized Milestones ensure that the learners develop certain core skills deemed necessary across all specialty and subspecialty learners, and that Milestone structure and expectations remain consistent across levels of training.

Use of the PID Milestones began on July 1, 2023. Fellowship Directors and Clinical Competency Committee Members should review the new Milestones and the Supplemental Guide developed for the Pediatric Infectious Diseases Milestones. This guide was developed as a tool to provide additional direction on the intent of each subcompetency and provide level-specific examples on learner demonstration of achievement. Additionally, the guide provides examples of ways each subcompetency could be assessed and lists additional resources pertaining to each subcompetency. Each program will also need to review its current assessment tools for fellows to decide if modifications or additional assessments are needed to allow for appropriate evaluation and assessment using the new Milestones. In particular, additional data may need to be obtained to evaluate fellows’ skills in antimicrobial and diagnostic stewardship, immunocompromised populations, epidemiology and infection control, and public health. Programs that require fellows to spend some time with faculty experts in antimicrobial stewardship or infection control, for example, could embed a relevant question in the fellow evaluation for that experience, providing assessment data for that particular subcompetency. Another way to operationalize obtaining data on these new subcompetencies is to embed a list of the subcompetency topics around which faculty comments are needed in the existing attending evaluation of the fellow. To limit the burden on faculty, faculty may choose to address two specific subcompetency areas. Across several months, this strategy should provide data on all subcompetencies. A third option would be to add two specific questions to the fellow evaluations about the new subcompetency topics but to rotate them (on even months ask about public health and infection control, and on odd months ask about antimicrobial/diagnostic stewardship and care of immunocompromised populations). While there is no singular method that may work uniformly for all educators or fellowship programs, each fellowship program should think strategically to identify best practices considering both the importance of collecting accurate learner assessments and the logistics and faculty time required to obtain it.

Milestones are not intended for use as the only tool in high-stakes decisions such as graduation readiness. Rather, the Milestones are intended to be a formative description of a fellow’s trajectory in attaining core competencies. As such, they are subject to the skill and potential biases of the evaluator in interpreting each trainee’s performance and assigning a Milestone level. In utilizing the Milestones for evaluation of fellows, it is important to realize that level 4 was designed as a potential goal for a graduating fellow, but there is no requirement to achieve this level in order to successfully complete a fellowship. Decisions regarding successful completion of the fellowship and the ability to successfully transition into unsupervised practice are made by the fellowship program director in collaboration with the clinical competency committee using EPAs as a summative assessment tool. EPAs are further elaborated upon by Pitts et al [12, 13]. The inclusion of level 5 Milestones reflects the need to provide goals for learners who have achieved level 4 prior to fellowship completion. Achievements at level 5 identify fellows whose attributes include advanced knowledge, attitudes, and/or skills more characteristic of experienced faculty [13].

As is true with any new assessment tool, future research will need to be done to establish inter-assessor reliability and validity using the PID Milestones.

While the development of the Pediatric Milestones 1.0 and the Subspecialty Subcompetencies represented important steps in the evaluation and directed formation of pediatric learners, the need for these Milestones to apply to learners across multiple stages of training and within various subspecialties, each with different approaches to care, meant that they were frequently inappropriate or irrelevant for the evaluation of learners pursuing subspecialty training. The development of the PID-specific Milestones is the next step in the directed assessment of fellows in PID educational programs. In addition to allowing evaluation of learners with respect to skills integral to the practice of all physicians, these Milestones highlight the unique aspects of PID practice which may not have been assessed in previous iterations of the tools, and allow fellowship directors to more directly identify and address concerns as learners move through their training programs. Initial feedback regarding the Milestones and the associated Supplemental Guide was generally positive, with some minor adjustments made based on comments; however, we anticipate that the Milestones will continue to be refined based on programs’ experience of use, as well as future developments in educational theory and the evolution of subspecialty needs over time.

Acknowledgments

We thank Ida Haynes (ACGME) and Sydney McLean (ACGME) for their assistance in facilitating and coordinating the meetings of the Work Group.

Funding

No funding was received by any of the authors for performance of the work reported in this manuscript.

Conflicts of Interest

My coauthors and I do not have any conflicts of interests to disclose.