A Systematic Approach to the Design and Implementation of Clinical Informatics Fellowship Programs

Abstract

Clinical Informatics (CI), a medical subspecialty since 2011, has grown from the initial four fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) in 2014 to more than 50 and counting in the present day. In parallel, the literature guiding Clinical Informatics Fellowship training and the curriculum evolved from the original core content published in 2009 to the more recent CI Subspecialty Delineation of Practice and the updated ACGME Milestones 2.0 for CI. In this paper, we outline this evolution and its impact on CIF Curricula. We then propose a framework, specific processes, and tools to standardize the design and optimize the implementation of CIF programs.

Keywords

Clinical Informatics - Professional training - Education - Curriculum Development - Training requirements

Background and Significance

Biomedical and Health Informatics is, as defined by the American Medical Informatics Association (AMIA), an “interdisciplinary field that studies and pursues the effective uses of biomedical data, information, and knowledge for scientific inquiry, problem-solving, and decision making, motivated by efforts to improve human health.”[1] Clinical Informatics (CI) is the application of informatics and information technology to deliver health care services. It is also referred to as applied CI and operational informatics.[2] While this field has been evolving since the 1960s, in 2009, the American Recovery and Reinvestment Act and the Health Information Technology for Economic and Clinical Health Act injected significant funding and regulations promoting the widespread adoption of electronic health records. Since then, there has been a recognized need for a well-trained CI workforce.[3] [4]

CI, as a board-certified medical subspecialty, received final approval from the American Board of Medical Specialties in September 2011. This approval was based on the development in 2009 of (1) the core content[5] and (2) the Clinical Informatics Fellowship (CIF) Program Requirements.[6] While these provided the building blocks for creating new CIFs, there was significant variability in the level of training and experience among CIF faculty in developing, implementing, and managing graduate medical education (GME) programs.[7] This variability in the training experience is documented in the recently published survey of CIF graduates.[8]

As CIF program directors gained experience in training and assessing fellows, it became clear that the Clinical Informatics Subspecialty (CIS) core content knowledge outline was insufficient for developing competencies on which fellows could be taught and evaluated.[9] Furthermore, this outline did not encompass all informatics-related expertise required by graduating CI fellows in postfellowship practice. Therefore, a comprehensive practice analysis of CI diplomates was conducted and resulted in the creation of the CIS delineation of practice (DoP).

Objectives

1. This paper summarizes the continuing process of incorporating the DoP into the existing structure for training fellows provided by the Accreditation Council for Graduate Medical Education (ACGME).

2. We provide recommendations from this process for optimizing CIF programs, focusing on designing and implementing a curriculum and fellow assessment.

Evolution of the Clinical Informatics Fellowship Curriculum

Creation and Maintenance of Clinical Informatics Core Content and Clinical Informatics Fellowship Program Requirements

The initial CI core content was created by a working group of 11 experts and published in 2009.[5] Subsequently, a group of 12 experts leveraged this core content document to determine how CI should be taught in a 2-year fellowship training program resulting in the proposed CI program requirements.[6] The mission of the ACGME is to improve health care and population health by assessing and enhancing the quality of resident and fellow physicians' education through advancements in accreditation and education.[10] Based on these documents, CI Program Requirements were initially issued by the ACGME on February 3, 2014. Section IV.A.5. of that document specified 59 CI subcompetencies grouped under ACGME competency categories (see [Fig. 1] and [Table 1]).

Fig. 1 Evolution of clinical informatics subcompetencies, knowledge statements, milestones, program requirements, and certification examination blueprint.

Table 1 CI Fellowship ACGME Competencies Specified in Program Requirements In 2014, following the 2020 revision, and proposed DoP-derived subcompetencies

ACGME Competency	2014–2017[a]		2020[b]		DoP-derived
	No. of Subcompetencies	%	No. of Subcompetencies	%	No. of Subcompetencies	%
Patient care and procedural skills	12	20	12	44	5	13
Medical knowledge	11	19	11	41	3	8
Practice-based learning and improvement	8	14	1	4	8	21
Interpersonal and communication skills	6	10	1	4	5	13
Professionalism	7	12	1	4	5	13
Systems-based practice	15	25	1	4	12	32
Total	59	100	27	100	38	100

Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; DoP, delineation of practice.

^a These subcompetencies remained unchanged in the Revised Common Program Requirements effective on July 1, 2017.

^b Current Program Requirements incorporated into the 2020 Common Program Requirements effective on July 1, 2020.

Creation of the Clinical Informatics Subspecialty Delineation of Practice

In 2018, AMIA and the American Board of Preventive Medicine (ABPM), in collaboration with the American Board of Pathology (ABP), agreed to update and expand the core content using a formal practice analysis methodology. Collaboration between these entities was important since both the ABPM and ABP certify candidates to take this examination (the ABP certifying pathologists only) and the ABPM administers the CIS certification examination.

This practice analysis resulted in a validated, comprehensive, contemporary CIS DoP comprising five domains, 42 tasks, and 139 knowledge and skill statements.[11] This framework is a unique and “best practice” approach to creating and applying a data-driven definition of CI across the CIF Program Requirements, curriculum, fellow assessment, and board certification.[12]

Creating Clinical Informatics Subspecialty Delineation of Practice-Derived Subcompetencies

The 59 CI subcompetencies in the 2014 CIF Program Requirements represented a “snapshot” created by a small group of experts nearly 13 years ago and no longer adequately reflect the evolving field of CI.

In September 2018, the Community of Clinical Informatics Program Directors (CIPD) Executive Council, a member of the AMIA Academic Forum, created the CIF criteria for excellence workgroup. This workgroup reviewed the CIS DoP to delineate both knowledge and tasks the fellows should acquire during CIF training (see [Supplementary Appendix A]—How-to Guide, Clinical Informatics Fellowship Tasks, Knowledge, and Criteria for Excellence, pages 5–6 [available in the online version]). In addition, a detailed map of CIS DoP knowledge statements supporting each subcompetency was also finalized (see [Table 2] for definitions). During this process, four tasks (i.e., subcompetencies) and nine knowledge statements were eliminated as these specific items were felt to apply to experienced CI practitioners rather than fellows. However, the original DoP numbering was retained to preserve coherence between the original CIS DoP and the DoP-derived subcompetencies.

Table 2 Definitions

Term	Definition
ACGME competency	The six ACGME core competencies are:  • Medical knowledge  • Patient care and procedural skills  • Interpersonal and communication skills  • Practice-based learning and improvement  • Professionalism  • Systems-based practice
DoP derived subcompetency	A task statement derived from the CIS DoP mapped to one of the six ACGME core competencies. In the context of this article, these statements are also synonymous with learning objectives.
Knowledge statement	Definition of what knowledge is required to develop competency in a specific subcompetency
Milestone grid	Milestones provide a framework for the assessment of the development of the fellow in key dimensions of the elements of physician competency. Each milestone grid is assigned to one of the six ACGME core competencies.

Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; DoP, delineation of practice.

Incorporation of Delineation of Practice-Derived Subcompetencies into the Accreditation Council for Graduate Medical Education Clinical Informatics Fellowship Program Requirements

Building on the work of the CIF criteria for excellence workgroup, in 2020, the CIPD Curriculum Subcommittee mapped each DoP-derived subcompetency to one of the six ACGME competency categories using a robust process involving primary and secondary reviews (see [Table 1]). As a result, the majority (79%) of the 38 DoP-derived subcompetencies fall within the ACGME competencies of practice-based learning and improvement (PBLI; n = 8, 21%), interpersonal and communication skills (ICS; n = 5, 13%), professionalism (PROF; n = 5, 13%), and systems-based practice (SBP; n = 12, 32%). However, these are the four competencies that the ACGME 2020 Common Program Requirements removed from all fellowships, assuming that these competencies are mastered during primary residency training. This assumption is invalid for CI as the competencies required differ significantly from those of clinical specialty training, as clearly shown by the mapping process above.

While CIFs can currently be accredited by four ACGME Review Committees (family medicine, internal medicine, pathology, or pediatrics),[13] the Internal Medicine Review Committee serves as the lead Review Committee as they have accredited the largest number of CI programs. Hence, members of the CIPD curriculum subcommittee brought the issue of incorporating DoP subcompetencies to the attention of the Internal Medicine Review Committee. Further discussion resulted in the ACGME Committee on Requirements recommending that the information be added to Section IV.C., Curriculum Organization, and Fellow Experiences.

Updated Clinical Informatics Fellowship Milestones Based on Clinical Informatics Subspecialty Delineation of Practice

In 2021, the ACGME Milestones Development Group conducted a process to revise the original 2015 CI Milestones. As a result, CI Milestones 2.0[14] and a Supplemental Guide[15] were published in April 2022 with an implementation date of July 2022. Of the resulting 21 milestone grids, 15 (71%) were mapped to specific DoP-derived subcompetencies.

Based on concerns regarding differences in the non-PC and non-MK milestone content across specialties, the ACGME convened four groups to develop cross-specialty “harmonized” milestones for ICS, PBLI, PROF, and SBP.[16] Six harmonized milestone grids were added to the new CI milestones as required by ACGME across all fellowships (see Section 2.B.1 below). However, these harmonized milestone grids are yet to be directly correlated with any specific DoP-derived subcompetencies.

The revised 2022 CI milestones based on the DoP-derived subcompetencies are valuable and relevant tools. However, given that the current CIF Program Requirement subcompetencies do not yet reflect the DoP-derived subcompetencies, there may be confusion about how the new milestones can and should be utilized beyond current semiannual national reporting requirements.

As of the writing of this article, the Internal Medicine Review Committee has begun the process of conducting a major revision of the CIF Program Requirements to be completed in 2024. This revision will help resolve any remaining discontinuities and confusion regarding which subcompetencies must be taught and reflected in milestone evaluations.

Proposed Approach to Creating and Implementing Clinical Informatics Fellowship Curricula

Based on the work described above, the CIPD Curriculum Subcommittee developed a “How-to Guide” consisting of supporting documents to be used as a resource for CIF programs. This guide articulates a detailed methodology for creating and implementing CIF curricula and fellow assessments, helping program directors and their teams.

• Create a curriculum based on DoP-derived subcompetencies.

• Utilize milestones and other instruments to assess trainees.

• Customize each trainee's learning experiences (individualized learning plan (ILP) and short-term and long-term project planning).

• Contribute to program evaluation.

An essential resource in this guide is the “Rosetta Stone Spreadsheet” (see [Supplementary Appendix B]—>CIF Rosetta Stone Curricular Planning Tool [available in the online version]). This spreadsheet maps each of the DoP-derived subcompetencies to:

• ACGME Core Competencies (i.e., medical knowledge, patient care, and procedural skills, ICSs, PBLI, PROF, SBP).

• ACGME CI milestone grid.

• Knowledge statements (in 2021, the ABPM Clinical Informatics Examination Blueprint[17] was significantly revised to focus on the knowledge statements of the CIS DoP) to:

  • – Support the acquisition of the skill described in the subcompetency.

  • – Include foundational knowledge statements not mapped to specific subcompetencies.

How-to Guide

The material provided in the How-to Guide, while not required for accreditation, is designed to optimize training in CI. CIF programs may select those components which are helpful. The methodology described will continue to apply if/when delineated subcompetencies and knowledge statements evolve by remapping the new subcompetencies to an ACGME competency category and milestone grid.

The following steps are recommended as a “best practice” guide for creating a CIF curriculum based on DoP-derived subcompetencies.

Identify and Articulate What is to be Taught (Skills and Knowledge)

This step is essential in aligning CIF curriculums to CI milestones and the certification exam. Therefore, despite uncertainty about how the DoP-derived sub-competencies will be incorporated into new CIF Program Requirements, we recommend focusing on the DoP-derived subcompetency and the knowledge statements mapped to each subcompetency (see [Supplementary Appendix B]—CIF Rosetta Stone Curricular Planning Tool [available in the online version]). Knowledge statements can easily be identified for most subcompetencies using the CIF Rosetta Stone Curricular Planning Tool mappings, the worksheets “DoP Curric GRID,” and “Sub-comp to Knowledge Mapping.” These knowledge statements also include foundational knowledge not mapped to specific subcompetencies.

Design Your Block Diagram

ACGME accreditation requires developing and using a block diagram representing the rotation schedule for a resident in a given postgraduate year.[18]

With the subcompetencies in mind, discuss and decide (1) which curricular elements to utilize (i.e., rotations, research, longitudinal, projects, didactics, online, etc.); and (2) the duration of each curricular element, typically in months. Record these decisions in a simple Word document (see [Supplementary Appendix A]—How-to Guide, Block Diagram Planning Template [available in the online version]). The CIS DoP domains are included in a table at the top as a high-level reference to the DoP-derived subcompetencies. The calendar is depicted monthly, typically the minimum length of a curricular element, particularly rotations.

The following steps will introduce processes and tools to help translate the block diagram into more detailed rotation descriptions. These detail which subcompetencies and knowledge are to be taught and which milestone grids should be utilized for faculty assessment of fellows.

Using the Clinical Informatics Fellowship Rosetta Stone Curricular Planning Tool, Assign Delineation of Practice-Derived Subcompetencies to Each Rotation

Once the block diagram design grid is completed, a manageable number of DoP-derived subcompetencies must be assigned to each rotation using the CIF Rosetta Stone Curricular Planning Tool. Next, enter the rotation names and durations from the block diagram design grid into the CIF Rosetta Stone Curricular Planning Tool (see detailed instructions on the top “Instructions” worksheet of the Planning Tool).

Once the block diagram has been added to the CIF Rosetta Stone Curricular Planning Tool, it may be necessary to revise the block diagram (i.e., rotations may need to be added or their focus and duration modified) if any of the following issues are found:

1. Less than or more than 24 months of rotations.

2. Rotations (columns) with 0 subcompetencies assigned—the rotation should be modified or deleted unless subcompetencies can be assigned to it.

3. Subcompetencies (rows) that are not assigned to any rotation—the subcompetency should be added and rotations may need to be added or modified.

Rotations focus on the development of skills reflected in the assigned subcompetencies. In addition to skill development, online coursework is particularly suited for knowledge acquisition necessary for a strong foundation and passing the board certification examination.

To ensure that all of the knowledge required to pass the board certification examination is provided to fellows, online course providers are encouraged to create a map of the material taught to all CIS DoP knowledge statements included in the ABPM CI examination blueprint including foundational knowledge statements which are not mapped to any subcompetency. In addition, when online coursework and/or projects result in DoP-derived subcompetency skill development, this can and should be reflected in CIF Rosetta Stone Curricular Planning Tool.

Compile Rotation Subcompetencies, Knowledge Statements, and Relevant Milestone Grids for Each Rotation to be Used for Formative Assessments

To determine which subcompetencies are assigned to each rotation, sort the table on the worksheet “DoP Curric GRID” by the column named by that rotation in Row 4. Sorting will cause all the subcompetencies designated with an “x” to be listed at the top together (see example in the CIF Rosetta Stone Curricular Planning Tool “Instructions” worksheet).

By sorting on the “ambulatory informatics” rotation (with a secondary sort on the DoP subcompetency number), the column shaded in blue comes to the top after sorting. Additionally, the subcompetencies and milestone grids (shaded in green) relevant for that rotation are specified (see [Fig. 2]).

Fig. 2 Sorting Rosetta Stone spreadsheet to identify assigned subcompetencies and milestone grids.

How to Utilize Milestones and Other Instruments to Assess Trainees

The 2020 ACGME CI Program Requirements note that “the program must provide an objective performance evaluation based on the Competencies and the subspecialty-specific Milestones, must use multiple evaluators, and provide that information to the Clinical Competency Committee (CCC) for its synthesis of progressive fellow performance and improvement toward unsupervised practice.” There is also a requirement for semiannual reviews, including performance evaluations, progress along the subspecialty-specific milestones, and the development of ILPs to capitalize on their strengths and identify areas for growth.[19]

Based on these requirements, the minimum required fellow assessment is a portfolio comprised of the following.

1. Milestone assessments relevant to the subcompetencies assigned to each rotation and completed by faculty.

2. Individualized learning plans, which include self-assessments by fellows utilizing all milestone grids semiannually, along with semiannual summative ratings determined by the CCC for all milestone grids.

3. Structured documentation of project work along with faculty evaluation of undertaken projects.

Milestone Assessments

Of the 38 DoP-derived subcompetencies, 26 (68%) were mapped directly to a specific 2022 revised milestone grid and 12 (32%) were not. The CIPD Curriculum Subcommittee reviewed and assigned a revised CI Milestone grid to each unmapped subcompetency. This was done as a two-stage process with four members of the subcommittee independently performing an individual mapping of a revised milestone grid onto each unmapped subcompetency. These individual mappings were then compared and demonstrated a very high level of agreement among the individual mappings resulting in the assignment of a specific revised milestone grid to each unmapped subcompetency. Finally, the 12 mappings were reviewed and approved by the whole CIPD Curriculum Subcommittee.

The above mapping opens the door to utilizing the new revised CI Milestones for summative assessments (i.e., reported biannually to the ACGME), formative (i.e., rotation) assessments, and ILPs.

Faculty understandably have difficulty completing milestone grids focused on subcompetencies not covered during that rotation. This issue is addressed by assigning a specific group of milestone grids to each rotation utilizing the CIF Rosetta Stone Curricular Planning Tool (see worksheet “Instructions” for detailed instructions on how to do this). Once these rotation milestone groupings are established, they can be programmed into GME software systems for completion by rotation faculty following each rotation.

In addition to the milestone grids described above, the following questions should be added to all fellow rotation evaluations using a 1 to 5 rating scale.

• Please rate the overall performance of this fellow on this rotation.

• Please comment on this fellow's areas of strength.

• Please comment on this fellow's areas for improvement.

• Please add any additional comments (comment text box only).

As mentioned above, in 2020, the ACGME introduced “harmonized” milestone grids to be utilized across all fellowship programs to assess ICS, PBLI, PROF, and SBP. These grids (listed below) were added to the 2022 CI Milestones and thus should be included in all rotation assessments. In addition, they are relatively easy for faculty to utilize and completion of these will assist the CCC in determining and submitting fellows' national milestone ratings to the ACGME semiannually.

• Practice-based learning and improvement 5: reflective practice and commitment to personal growth.

• Professionalism 3: professional behavior and ethical principles.

• Professionalism 4: accountability/conscientiousness.

• Professionalism 5: self-awareness and help-seeking.

• ICSs 3: interprofessional and team communication.

• ICSs 4: communication within health care systems.

This approach aligns each rotation's subcompetencies, knowledge statements, and milestone grids. An example of an “ambulatory informatics” rotation description utilizing this methodology is included in [Supplementary Appendix A]—How-to Guide, Ambulatory Informatics Rotation Description, and Mapped Milestone Assessment Grids (available in the online version). For fellow rotation evaluations, only the milestone grids which assess those subcompetencies are distributed for completion by rotation faculty. This creates a virtuous cycle where faculty's understanding of and attention to focusing on specific subcompetencies is reinforced.

How to Customize Each Trainee's Learning Experiences: Individualized Learning Plans

ILPs focus on learner self-assessment, goal setting, and career planning and should be updated semiannually. ILPs serve as input to CCC meetings which then provide feedback to fellows and faculty advisors.

The following process for the development and maintenance of ILPs is recommended (see [Supplementary Appendix A]—How-to Guide, Semi-Annual Self-Assessment and Individualized Learning Plan [available in the online version]).

• The Semi-Annual Self-Assessment form is provided to each fellow upon enrollment, along with specific instructions and the indicated attachments. Each fellow:

  • – does a self-assessment by rating themselves on each of the 2022 CI milestone grids upon starting the fellowship and then semiannually.

  • – responds to four questions as indicated on the semiannual self-assessment form.

• The individualized self-assessment and learning plan is generated upon enrollment and updated semiannually by the fellow in conversation with their faculty advisor based on the semiannual self-assessment and cumulative milestone self-assessment grids (see [Fig. 3] for sample).

Fig. 3 Sample cumulative milestone self-assessment grids utilizing original 2015 CI milestone grids.

Following the CCC meeting, the fellow discusses the feedback received with their faculty advisor for incorporation in the ILP update (similar to a “P-D-S-A” cycle)

Structured Documentation of Project Work Along with Faculty Evaluation of the Project

Sample project assessment and documentation forms are included in [Supplementary Appendix A]—How-to Guide, Project Assessments and Documentation (available in the online version)

• Short-term projects would typically be less than 4 months, and due to their shorter duration and narrower focus, the documentation and review required are not as extensive as that for long-term projects.

• Long-term projects would typically be at least 4 months and have intermediate deliverables.

  • – Prospectus (1/3 of the time into the project).

  • – Final report: components and format requirements (within “x” days of completion of the rotation).

• Ideally, there should be a single designated faculty member who oversees fellow projects from an educational and administrative perspective, with other faculty supervising and directing individual projects, all of whom should complete the evaluations.

How to Contribute to Program Evaluation

The processes and documentation described above have the additional benefit of creating the following materials for semiannual CCC and annual PEC meetings:

• Initial and ongoing/periodic CI milestone self-evaluations, semiannual self-assessments, and ILP.

• Project reports and faculty evaluation of these reports.

• Semiannual fellow milestone self-assessment trending (individual and all fellows).

CI fellows do not provide direct patient care services as fellows or clinical informaticians. Instead, they care for patients as independent practitioners of medicine in their clinical specialty. The clinical institution typically provides supervision during CI fellow's clinical activities. Ideally, the clinical institution should provide a relevant summary of fellow clinical activity and capability to the CCC.

Unlike clinical rotations where faculty and fellows work closely together on clinical units, CI CCC members frequently have not had significant direct observation of fellows. In this context, rotation faculty milestone evaluations using national CI Milestone grids are a valuable input to CCC discussions.

Discussion

This paper aims to articulate best practices to integrate the following:

• Tasks (subcompetencies) and knowledge that support skill acquisition for each task.

• Milestones based on those subcompetencies, which can be used for formative and/or summative fellow assessment within the CIF training program

• The development and revision of ACGME CIF Program Requirements

The definition of CI was based on the original core content derived largely from expert opinion. As more CIF programs received initial accreditation, there was understandably a lot of variability in how their curriculums were created, implemented, and evaluated. CIF programs were often led by talented and dedicated faculty who typically had little formal training in curriculum development or ACGME program and administrative processes and requirements.

The tools and processes described in this article were originally developed and deployed in 2016 at The University of Arizona College of Medicine—Phoenix based on the original core content and milestones. However, the lack of connection between these two components was limiting. As the DoP-derived subcompetencies became available in 2019 along with revised CI milestones in 2022, the same curricular design and evaluation processes could now be utilized based on these connected components. In this initial and ongoing implementation, feedback from fellows and faculty was very positive.

• Fellows appreciated that they were given a concise and relevant list of learning objectives for each rotation and that cumulatively all subcompetencies and knowledge was coherently taught across the totality of their rotations, thus preparing them well for the certification exam and subsequent CI work.

• Faculty were happy to receive more specific direction about what to teach during various rotations and felt much more comfortable completing milestone evaluations limited and linked to the instructional aims articulated for each rotation—this also increased the timeliness, quality, and completeness of the fellow evaluations.

• CCC members now had direct access to faculty milestone evaluations of fellows for each rotation which cumulatively allowed more accurate completion of national milestone ratings.

• The Program Evaluation Committee can easily update the block diagram and subcompetencies assigned to each block based on feedback from faculty and fellows.

As this approach was informally shared within the CIPD Curriculum Subcommittee, several faculty in that group and others outside that group successfully implemented these methods in their programs.

The authors recognize that the above components were developed in the context of program accreditation, training, and certification requirements in the United States. The domains, tasks, and knowledge defined in the original CIS DoP provide a robust, detailed, and transportable description of CI. Competency frameworks created outside of the United States[20] can be reflected in modifications to the CIF Rosetta Stone Curricular Planning Tool and the curriculum design and program/fellow evaluation processes could be used as described herein or relatively easily adapted as needed.

We believe the approach outlined in this paper can help other medical specialties to (1) identify the data-based components (i.e., DoP) rather than relying only on expert opinion and (2) define and design a methodology to align all three items listed above.

As new versions of the subcompetencies evolve, the information and linkages provided in the Rosetta Stone Curricular Planning Tool must be updated ideally by the CIPD Curriculum Subcommittee with broader input from CIPD members. Otherwise, the methodology should not need to be changed significantly. As pioneering CIFs adopt this systematic approach to create their curricula, fellow evaluations, and ILPs, we can compare outcomes such as ABPM CI Board pass rates, Fellow Milestone Portfolios, and ACGME program evaluation data.

Clinical Relevance Statement

CIF curricula can now reflect DoP-derived sub-competencies and knowledge derived from a rigorous analysis of what practicing clinical informaticians do and know. Utilizing best practices to define and deliver fellowship instruction is critical to the success and effectiveness of fellows and future CI practitioners.

Multiple-Choice Questions

1. The Clinical Informatics Subspecialty Delineation of Practice published in 2019 articulated which of the following.

   • Core topics related to Clinical Informatics.

   • Core skills related to Clinical Informatics.

   • Core knowledge related to Clinical Informatics.

   • Core domains, skills, and knowledge related to Clinical Informatics.

   Correct Answer: The correct answer is option d. The 2019 Clinical Informatics Subspecialty Delineation of Practice was the first time that Clinical Informatics domains, skills, and knowledge were articulated. The original 2009 publication detailing the Core Content was a list of topics which was an input to the Delineation of Practice process. The choices of skills or of knowledge do not fully describe the full scope of the Clinical Informatics Subspecialty Delineation of Practice.

2. The revised Clinical Informatics Milestones released by the ACGME in 2022 was based on which of the following:

   • The original Clinical Informatics milestones released in July 2015.

   • The ACGME “Milestones 2.0” cross-specialty harmonized milestones.

   • The delineation of practice-derived skill statements.

   • Systematic and structured input from Clinical Informatics Fellowship program. directors

   Correct Answer: The correct answer is option c. The 2022 Clinical Informatics milestones were mapped specifically to the delineation of practice-derived skill statements. The original 2015 Clinical Informatics milestones were not a significant input into the revised milestones. While there was input from some Clinical Informatics Fellowship faculty on the milestones revision workgroup into the revised Clinical Informatics milestones, there was no structured input from Clinical Informatics Fellowship Program directors. The harmonized milestones were added by the ACGME after the newly composed Clinical Informatics milestones were completed but were not input into them. The original milestones were reviewed during the revision process but did not serve as a significant underpinning for the new milestones.

Conflict of Interest

None declared.

Acknowledgments

The authors wish to acknowledge the interest and support of the members of the Clinical Informatics Program Director Curriculum Subcommittee. Special thanks to Dr. Aarti Chandawarkar, who participated actively in composing the article, and to Dr. Michael Miller for reviewing and editing it. We also wish to acknowledge and thank Dr. Soumya Panchanathan and Dr. Ben Munger who contributed significantly to the early development of these materials and processes.

Protection of Human and Animal Subjects

Human and/or animal subjects were not included in the project.

• References

• 1 Kulikowski CA, Shortliffe EH, Currie LM. et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. J Am Med Inform Assoc 2012; 19 (06) 931-938
• 2 Accessed August 4, 2023 at: https://amia.org/about-amia/why-informatics/informatics-research-and-practice
• 3 Hersh W, Wright A. What workforce is needed to implement the health information technology agenda? Analysis from the HIMSS analytics database. AMIA Annu Symp Proc 2008; 2008: 303-307
• 4 Hersh W. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform 2010; 1 (02) 197-212
• 5 Gardner RM, Overhage JM, Steen EB. et al; AMIA Board of Directors. Core content for the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 153-157
• 6 Safran C, Shabot MM, Munger BS. et al; AMIA Board of Directors. Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 158-166
• 7 Longhurst CA, Pageler NM, Palma JP. et al. Early experiences of accredited clinical informatics fellowships. J Am Med Inform Assoc 2016; 23 (04) 829-834
• 8 Kim E, Van Cain M, Hron J. Survey of clinical informatics fellows graduating 2016-2024: experiences before and during fellowship. J Am Med Inform Assoc 2023; 30 (10) 1608-1613
• 9 Silverman H, Lehmann CU, Munger B. Milestones: critical elements in clinical informatics fellowship programs. Appl Clin Inform 2016; 7 (01) 177-190
• 10 Accessed August 4, 2023 at: https://www.acgme.org/about/overview/#:~:text=The%20Accreditation%20Council%20for%20Graduate,medical%20care%20to%20all%20Americans
• 11 Silverman HD, Steen EB, Carpenito JN, Ondrula CJ, Williamson JJ, Fridsma DB. Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis. J Am Med Inform Assoc 2019; 26 (07) 586-593
• 12 Berner E. Ed. Informatics Education in Healthcare, Lessons Learned. 2nd ed.. Springer; 2020: 215-218
• 13 Accessed August 4, 2023 at: https://apps.acgme.org/ads/Public/Reports/Report/1
• 14 Accessed August 4, 2023 at: https://www.acgme.org/globalassets/pdfs/milestones/clinicalinformaticsmilestones2.0.pdf
• 15 Accessed August 4, 2023 at: https://www.acgme.org/globalassets/pdfs/milestones/clinicalinformaticssupplementalguide.pdf
• 16 Edgar L, Roberts S, Holmboe E. Milestones 2.0: a step forward. J Grad Med Educ 2018; 10 (03) 367-369
• 17 Accessed August 4, 2023 at: https://www.theabpm.org/become-certified/exam-content/clinical-informatics-content-outline/
• 18 Accessed August 4, 2023 at: https://www.acgme.org/globalassets/PDFs/Guide_to_the_Construction_of_the_Block_Diagram_3.30.17.pdf
• 19 Accessed August 4, 2023 at: https://www.acgme.org/globalassets/PFAssets/ProgramRequirements/381_ClinicalInformatics_2020.pdf?ver=2020-06-29-163724-707&ver=2020-06-29-163724-707 Section V.A.1.c, page 27
• 20 Davies A, Mueller J, Moulton G. Core competencies for clinical informaticians: a systematic review. Int J Med Inform 2020; 141: 104237

Publication History

Received: 05 June 2023

Accepted: 20 September 2023

Article published online:
06 December 2023