KOI Pond: The “Elevator Pitch” that Adapts to Describe the Breadth and Variety of Clinical Informatics Practice

Abstract

Background The practice of clinical informatics (CI) is diverse and frequently tailored to individual skills and interests or to organizational/departmental needs. Prior studies have sought to define core content areas, educational milestones, and practice patterns within the clinical informatics subspecialty (CIS). Unfortunately, no single tool or framework currently exists that can succinctly define an informatician's role regardless of setting. The diversity of informatics practice makes it difficult to have one “pitch” that describes all possible domains, tasks, knowledge, and skills available to an informatician.

Objective Using qualitative data from multiple informaticians, provide a succinct framework to describe and compartmentalize the various functions an informatician can contribute to the healthcare field.

Methods We created an iterative focus group of five CIS fellows enrolled in different fellowship programs nationwide, one CIS program director, and an MD-PhD candidate in biomedical informatics. After much discussion, iteration, and consideration of career options within a young and burgeoning subspecialty, a dual-axis model was created that describes CIS practice in terms of settings (internal, external, and policy) and focus areas (knowledge, operations, and innovation).

Results and Conclusion By combining both axes into a single “KOI pond” and then prioritizing sections by interest/resource investment, we are able to generate a unique snapshot for each informatician. These snapshots can be used (1) by informaticians to characterize their own practice succinctly as a pitch, (2) by CIS fellows who are considering career options, and (3) by those unfamiliar with CIS who want to learn more.

Keywords

academic training - clinical informatics - professional activity study - career mobility - medical informatics applications

Background and Significance

The clinical informatics subspecialty (CIS) is a newly accredited subspecialty, the scope of which expands into all areas of healthcare. Although some aspects of healthcare informatics date back as far as the 1960s,[1] the inaugural examination from 2013 serves as the time in history that nearly 500 informaticians were awarded the first CIS board certification.[2] Currently, the Accreditation Council for Graduate Medical Education (ACGME) defines CIS as “the subspecialty of all medical specialties that transforms healthcare by analyzing, designing, implementing, and evaluating information and communication systems.”[3] Diplomates from clinical informatics (CI) fellowship programs are uniquely multidisciplinary: often times a CI fellow will represent an entirely different specialty from their program director, mentors, and peers in the same program. This camaraderie necessitates a broader scope of thinking, and as such, fellows frequently harness interest-driven projects to advance their understanding of CI core content.

This variety in projects, tasks, and lessons learned has its disadvantages. For trainees who engage in project-based learning, it is nearly impossible to replicate one's educational experience, even within the same institution. Additionally, given its relative nascency, CI is often poorly understood by the medical community at large and frequently mischaracterized as “computers in medicine.”[2] The presence of similar-sounding terms such as bioinformatics, public health informatics, and imaging informatics help add to the confusion,[4] and for years even these definitions were subject to debate within the literature.[5] [6] [7] Thus, an established and agreed-upon core content[8] and a description of training program requirements[9] were integral to the creation of CIS.[2]

Unfortunately, CIS remains poorly understood. One cannot simply describe the role of CI by core content alone ([Fig. 1]), and those just learning about CIS may feel confused or overwhelmed as they read about the competencies and curricula for CI, pathology informatics, nursing informatics, dentistry informatics, medical librarianship, or related fields.[10] [11] Adding to this, no set standard or job description exists for those who have undertaken CIS training. In 2016, an American Medical Informatics Association (AMIA) task force defined a chief clinical informatics officer (CCIO) role in terms of knowledge, education, and operational skillsets.[12] This role sought to expand beyond the chief medical information officer (CMIO) position[13] in a manner that also included nursing, dental, and pharmacy informatics perspectives. Despite this, neither CCIO nor CMIO roles consider informaticians working in academic, startup/consultant/vendor, or government/public policy settings. Also, neither role is exclusive to CIS diplomates.

Fig. 1 Word cloud of clinical informatics core content areas. This word cloud was generated using core content areas defined in the 2009 Gardner publication.[8] The relative size of each term denotes its frequency of use.

In 2018, a practice analysis was sent to the 1,695 CIS diplomates who were certified by the American Board of Preventive Medicine.[14] Data collected from the 316 responses (18.8%) were compiled to create a delineation of practice (DoP) containing five domains, 42 tasks, and 139 knowledge and skill (KS) statements. The five domains from this DoP are: (1) fundamental KS, (2) improving care delivery and outcomes, (3) Enterprise Information Systems, (4) data governance and data analytics, and (5) leadership and professionalism. These domains have now supplanted the 2009 core content paper[9] as the basic roadmap for CI fellowship training programs, and they will be summarily assessed in future CIS certification exams.

In many ways, this CIS practice analysis draws parallels to the Model of the Clinical Practice of Emergency Medicine (EM Model).[15] Both models are critically important for defining their specialty's skills and core content in a manner that ultimately determines its educational curriculum and specialty board examinations. However, neither the EM Model nor the CIS DoP can stand alone as a succinct description of their respective medical positions. The breadth of CIS compounds this fact, and as such the need exists for a shorter characterization of the subspecialty.

Objectives

More than ever, informaticians are in dire need of a succinct description of what they do. As CIS continues to grow, informaticians will need to be able to clearly and accurately describe their skills and interests in a manner that is accessible to those unfamiliar with CI. Furthermore, given how this field has already expanded, the traditional CMIO/CCIO role is unlikely to serve as the sole framework that describes every career path an informatician may take.

Methods

From 2016 to 2017, a model was created using an iterative focus group[16] comprised of five CI fellows enrolled in different CI fellowship programs, one CI program director, and an MD-PhD candidate in biomedical informatics. Initial concepts were created by authors J.D.M. and M.W., who were considering career options at the time, and with some overall guidance by B.P.L. As each new member was included into the focus group (C.S., E.J.B., J.F.P., and L.V.G.), the model was continually revisited to include each member's perspectives and considerations. Discussions were held in-person at AMIA's 2016 and 2017 annual symposia, asynchronously via Slack (Slack Technologies, San Francisco, CA), and synchronously via Google Hangouts (Google LLC, Mountain View, CA). Concepts were expanded and refined into a dual-axis approach that follows the motif of a broad and adaptable subspecialty. During much of this time, the group's CI fellows were actively considering their career options in this new subspecialty, which served as a major inspiration for the model. The final version of this model has remained stable since 2017, defines each practice area loosely in terms of setting (Axis 1) and focus area (Axis 2), and is currently being used by the authors to help manage their own CI projects and employment.

Results

Model: The KOI Pond

Our model builds on the notion of a Japanese KOI pond. KOI are ornamental carp that differ widely in their colors, patterns, and scales.[17] Given their diverse morphologies, their adaptability to many pond environments, and their ease of care, unique patterns of KOI can be appreciated in decorative ponds worldwide. CIS is equally diverse, so we found this motif ideal. Consider each informatician as a single fish in an expansive, interconnected pond ([Fig. 2]).

Fig. 2 KOI pond model. Axis 1 on the left is the environment where an informatician works (internal = providing care, external = supporting care, policy = overseeing care). Axis 2 on the right is the focus area (knowledge, operations, and innovation). For a given informatician/project, some degree of overlap is expected.

• Axis 1: the pond = practice setting

The first axis is the pond. For fish to remain healthy in their environment, they need:

• Water = internal health systems that “provide the care”

• Container = external organizations that “support the care”

• Filter = policy agencies that “oversee the care”

Internal: Provides the Care

Our analogy defines internal systems (water) as any health system or setting where direct or indirect patient care is provided. This may be in the form of a hospital, a clinic, a single practitioner, a skilled nursing facility, or a telemedicine service. Indirect or ancillary care areas are also included, such as radiology, pathology, pharmacy (hospital and retail), allied health professionals, and similar areas.

External: Supports the Care

External (container) is defined as any organization that supplies a product or service to the internal system. Importantly, the organization that supports the care should not be providing direct or indirect patient care as their service. This support may be in the form of physical products, such as medical devices or medications; digital products, such as electronic medical records or apps; or a basic service, such as a survey or nonmedical follow-up call. Startups, incubators, and accelerators that focus on healthcare without providing direct or indirect patient care are also included. External products may include publications, conferences, and studies that target multiple health systems. Finally, the work that a healthcare system does when coordinating with one of these support organizations may also be considered external, albeit from a different perspective.

Policy: Oversees the Care

Policy (filter) is what defines the environment for both internal and external systems. In the same way that oxygenation and cleanliness of a pond can dictate how active KOI may be or the size to which they grow, policy creates the incentive structure in which informatics operates. Any guideline, rule, or regulation placed upon either of these systems counts as policy. This may include government organizations, accreditation/regulating bodies, and organizational committees such as data governance. Textbooks, evidence-based medicine practices, and established standards of care are also included in this category.

• Axis 2: the KOI = focus areas

The second axis is the KOI, as defined by three focus areas:

• Knowledge (K)

• Operations (O)

• Innovation (I)

While each KOI focus area will typically work with and around data, the tasks performed and skill requirements for each focus area tend to differ substantially. As shown in [Fig. 3], this axis was loosely based on Rogers' diffusion of innovations model[18] in that the target audience of something entirely new (I) may require a different approach than something that targets the majority of users (O). Some tasks, such as with K, may not require end-user adoption at all (e.g., observational research and data mining).

Fig. 3 High-level relationship among KOI pond focus areas. High-level relationship between each KOI pond focus area in a manner that also considers Rogers' diffusion of innovation curve.[18] Note that a given project may overlap multiple focus areas at once. It may also shift focus areas over time.

[Fig. 3] shows the high-level relationship between each focus area. By adding the internal, external, and policy practice settings to each focus area ([Fig. 2]), we now have a model that describes what each person does (Axis 2) and where they do it (Axis 1). Note that each informatician will likely have their own unique snapshot, and this snapshot may change over the span of their career. Additionally, a single project may progress from one focus area to another over time or may overlap different areas simultaneously. Finally, it is presumed that topics arising from the K focus area will often help define or guide topics in the O or I areas.

Below, we provide a description of each KOI focus area, followed by example snapshots of an informatician who prioritizes that focus area and a basic pitch that describes them. By no means is this meant to be an exhaustive list. Rather, it should serve as an example of how to adapt the KOI pond (snapshot and pitch) to match the informatician. While each example listed below is fictional by design, existing job descriptions and real-world career choices were considered heavily.

Knowledge: Defines the Care

The Knowledge (K) focus area obtains and analyzes healthcare data in a manner that adds value to the overall understanding of a topic or condition. Using the ACGME definition of CIS listed above,[3] K focuses most strongly on “analyzing” and “evaluating” systems. As a rule of thumb, if it requires approval from an institutional review board, chances are it is deeply rooted in K. Quality improvement and operational metrics are also a part of K, though work within these projects will typically span multiple focus areas. Insights from K focus area activities may expand beyond a single health system in the form of publications (K-external), which in turn may help guide evidence-based medicine and standards of care (K-policy). [Table 1] shows an example of an academic informatician.

Table 1 KOI pond snapshot of an academic informatician

	Knowledge	Operations	Innovation
Internal	5 randomized controlled trials	2 quality	1 new study design
External	5 publications, presentations, multicenter trials	0	0
Policy	4 committees, specialty specific guidelines/policies	2 manage projects as principal investigator	2 technology transfer of research

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Pitch 1: (academic informatician) Harnessing informatics to pursue K is where I excel. I have an R01 grant for my research on (core content topic here), and I am core faculty for our CI fellowship training program. My research is in the midst of a multicenter trial, which we hope will positively impact clinical practice. We have established a national committee to define and guide our standard of care in this area.

Operations: Implements the Care

Again considering the ACGME's definition of CIS,[3] the Operations (O) focus area ties most closely to “implementing” systems. This is in large part through leading and managing change, effective communication, and team dynamics. What makes the O focus area distinct is its targeted end-user group. If a population's majority—as defined by Rogers' diffusion of innovations model[18]—serves as the primary target, then the O focus area is likely dominant. Navigating requests for proposals and large-scale vendor implementations are typically considered O-external. Those who manage organizational policies, such as data governance, would consider this work under O-policy. [Table 2] shows an example snapshot of a CMIO/CCIO.

Table 2 KOI pond snapshot of a chief medical informatics officer or chief clinical informatics officer

	Knowledge	Operations	Innovation
Internal	3 control charts, workflow analysis	5 implement/update health record order sets	1 pilot testing prior to implementation
External	0	4 requests for proposal/information	0
Policy	3 decision support tools	5 data governance, standard workflows	0

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Pitch 2: (CMIO/CCIO) I want our clinicians to be happy today, and thus my role focuses on keeping O streamlined. We work very closely with (vendor here) to ensure electronic orders are high yield and easy to navigate. We have also sought to standardize clinical workflows across the various care areas I manage. I just started meeting with (consultant/vendor/startup here) who I found via a standard request for proposal process to make (workflow issue here) easier. We are currently planning a waterfall implementation of their product at our institution.

Innovation: Disrupts the Care

As with informatics, the concept of innovation is somewhat vague and often overused in a manner that promotes confusion.[19] [20] [21] In a 2009 literature review, Baregheh found 60 different definitions of innovation published across disciplines.[22] Within healthcare, some of the most important characteristics of innovation are: “(1) novelty, (2) an application component, and (3) an intended benefit,” noting that the application component can be “new services, new ways of working, and/or new technologies.”[23] [24]

For our purposes, the Innovation (I) focus area should match the ACGME's definition of CIS[3] about “designing” systems. If new ideas and/or intellectual property are generated that previously did not exist or if something is so new that it only targets the innovator/early adopter section of Rogers' diffusion of innovations model,[18] it is likely an I focus area. This may occur as a physical device, a digital product, or an abstract idea. Examples of I-internal activities include in-house development, process redesign, and the newer concept of “intrapreneurship.”[25] [26] [27] Someone working as an entrepreneur, as a consultant/vendor, or as a collaborator for a project that spans multiple institutions (e.g., open-source development) would often consider their work I-external. Those who are willing to undertake risks to create something disruptive and positively impactful may consider themselves intrapreneurs or entrepreneurs, depending on the context in which they work. [Table 3] provides a KOI pond snapshot of an entrepreneur who focuses on new and upcoming healthcare technologies.

Table 3 KOI pond snapshot of a healthcare technology entrepreneur

	Knowledge	Operations	Innovation
Internal	0	0	0
External	1 plan, do, study, act cycles with health system	3 navigate requests for proposal, implementation, troubleshooting	5 finding new technologies, adapting them to healthcare
Policy	0	4 manage projects at startup (agile)	4 patents, branding, navigating regulations

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Pitch 3: (Healthcare technology entrepreneur) I am most interested in how we can leverage cutting-edge technology into our healthcare system today. While I still practice clinically part time, I spend a lot of my efforts at a mid-sized startup that integrates (concepts from other fields here) and applies them to healthcare. I also serve as the liaison between my startup and its client healthcare organizations through which I help test and implement design concepts into practice.

Applications

Much like KOI, the practice of CI comes in many different shapes and colors. This model can be applied by an informatician to concisely describe how they invest their own time and resources. Note that each table above shows a snapshot of an informatician in a manner that is mostly agnostic to CI core content. Take, for example, a single core content area: cybersecurity. Within cybersecurity alone, one can conduct research (K), implement measures (O), or create entirely new security measures (I). These cybersecurity efforts can also be deployed across the full spectrum of informatics settings, such as within a hospital organization (internal), through a consulting firm managing a cybersecurity portfolio (external), or in a manner that manages national/organizational standards for cybersecurity (policy). By combining both axes, we arrive at a simple yet effective way to describe the work of an informatician. This same structure can be used by an individual describing their current position (or desired position) or by a healthcare organization interested in creating new informatics positions for employment.

The K focus area can be combined with any of the settings listed above. A hospital organization's K-internal combination would likely reflect an academic informatician at the start of their career or one who gathers metrics for operational/quality use. If that informatician focuses more on grant funding, publications/conferences, or multicenter trials, then he/she would now consider much of his/her work K-external. By comparison, an informatician who is interested in national standards (K-policy) may do this by serving on an advisory board. [Table 4] provides an example snapshot of an informatician who focuses on K-policy.

Table 4 KOI Pond snapshot of an advisory board informatician

	Knowledge	Operations	Innovation
Internal	0	0	0
External	4 conference representation, registry data quality control	3 adding new health systems to data registry	0
Policy	5 national data registry governance	3 project management and leadership	0

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Pitch 4: (Advisory board informatician) As my primary specialty continues to grow, it has been my passion to ensure that we expand in a healthy and meaningful way. I serve on the (council/board of directors/advisory board) of my specialty's nonprofit (association/college). Though my role entails several tasks, my primary work is to ensure the quality and rigor of our national data registry. This registry serves an important role in our specialty's reimbursement and in large-scale academic research.

The O focus area may be similarly stratified across the care settings, such as the CMIO and other healthcare leadership positions (internal), consultation assistance for key organization informatics areas (external) or organizational/national guidelines for hospital system use of informatics assets (policy). Note additionally that healthcare leadership positions are not strictly limited to O focus areas. The chief innovation officer (CINO),[28] chief digital officer (CDO),[29] and chief strategy officer (CSO)[30] [31] are I-focused career options those with CIS training may consider, though the exact roles and tasks should be contextualized to match the organization's needs and priorities.[32] As with the CMIO/CCIO path, these roles are not mutually exclusive to CIS diplomates. [Tables 5] and [6] provide KOI pond snapshots of an implementation consultant and a CINO, respectively.

Table 5 KOI pond snapshot of an implementation consultant

	Knowledge	Operations	Innovation
Internal	3 current/ideal state metrics for workflow analysis	3 helping users implement process improvements	2 pilot testing
External	0	5 change management, kaizen sessions	0
Policy	0	2 applying lean six Sigma	1 branding

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Table 6 KOI pond snapshot of a chief innovation officer

	Knowledge	Operations	Innovation
Internal	3 gemba walks, sprint preparations	2 transition disruptive ideas into common practice	5 design thinking, design sprints
External	0	0	4 commercialization and collaboration with startups
Policy	0	2 teaching lean and agile principles	3 technology transfer, patents, enterprise architecture

Note: Numbers included represent a subjective investment of resources (0 = no investment, 3 = moderate, 5 = major/top priority).

Pitch 5: (Implementation consultant) For each practice setting and care area, clinical processes tend to vary significantly. Our company works with large health organizations to help them understand their workflows and opportunities for improvement. After deriving current state workflows and metrics, we host kaizen and brainstorming sessions with the clinical staff. We help them build from their suggestions and navigate the path of change management. My knowledge of how clinical data ties into workflow is critical for our company's ability to understand and gather accurate and reliable metrics for our clients. The ability to distinguish practice variation/patient variety from inefficiencies/edge outliers is where I apply informatics.

Pitch 6: (CINO) If we want true impactful change, then we have to be forward-thinking. My team regularly hosts sessions where we teach employees at our health system how to apply (design thinking, design sprints, agile, and lean) methodologies in their departments. We also work with (startup/accelerator/ecosystem here) to see which products may integrate best into our health system and to pursue opportunities for collaboration.

While each snapshot may be unique to the individual, that snapshot may change or narrow as one's career path matures. Additionally, the 0 to 5 investment of resources scale used in our examples was derived using subjective assessments of fictional positions. It is possible to use objective data, such as time investments or full-time equivalents (FTEs) to help standardize this scale. However, objective data alone may not fully capture a person's activities and interests. While CI does not have relative value unit assignments like other medical specialties, qualitative estimates of effort, such as those provided in the tables above, may be useful as a whole.

Qualitative and quantitative data could also be combined to show a snapshot's maturity over time, to compare current and ideal states, or to provide a basic job description that includes skills and interests with FTE breakdown. Since one cannot bill patients directly for informatics work, these snapshots may also assist CI diplomates who work on a departmental level (e.g., housed in their primary specialty). Given the paths to financial support of CI activities may vary significantly based on focus area, it may be helpful to see how other CI diplomates with similar KOI snapshots and departmental scopes were able to fund their work.

Discussion

CIS is still a very young subspecialty. From October 2014 to March 2019, the number of nationwide ACGME-accredited CI fellowship programs increased by a factor of eight (from 4 to 33 programs).[33] With many programs accepting an average of two or more fellows per year, it is likely that each fellow will pursue a different career path than their predecessors from the same program. As such, the ability for informaticians to define their practice succinctly will be important regardless of where they are working and to whom they are reporting. For those in CI, KOI pond snapshots can serve as the lexicon we use to describe our roles and responsibilities to each other and to our employers.

While core content material should be considered heavily when deciding career interests, it is also important to consider what tasks are done (K–O–I) and the setting in which they are performed (internal, external, and policy). For CIS trainees planning their intended career path, it may be helpful to look at focus areas first, then consider core content areas that are most appealing and use this information to target job positions that can support said interests. One may also consider job options within his/her primary specialty, harnessing CI in a manner that focuses more on the department than the organization.

Although we believe this model provides a comprehensive way to describe the various pathways informaticians may follow, it does have several limitations. First, this construct may still fail to fully encapsulate every setting and focus area experienced in CI practice, especially as the subspecialty continues to mature. Conversely, in an attempt to be all-inclusive, our loosely defined construct may not be specific enough and therefore may fail to communicate the nuanced differences between some CI settings and focus areas. This may in-turn lead to inconsistencies and subjectivity among where each informatician defines their own tasks and interests within the KOI pond.

Additionally, while the method chosen for model generation (iterative focus group) has its own inherent limitations,[16] [34] the inclusion of CI fellows from various programs, interests, and primary specialties should help mitigate these limitations. Iterative focus groups have been used successfully in model generation[35] and healthcare end-user prototyping,[36] and to some extent our chosen method draws parallels to user-based usability methods[37] and collaborative participatory design.[38] Nevertheless, our focus group mainly included CI fellows at the start of their informatics careers. As such, this model should be juxtaposed with the domains, tasks, and KS statements from Silverman's 2019 DoP analysis, in which respondents were more broadly distributed across age groups (35–44 [24.7%], 45–54 [34.5%], and 55–64 [26.9%] years old).[14] While outside the scope of this paper, a more rigorous comparison of our KOI pond and the initial core content areas[8] with the recent DoP analysis may add value to our subspecialty as we continue to define and refine the breadth of CI practice.

Conclusion

Informaticians are unique in their diversity, their multidisciplinary nature, and their adaptability to many settings. We have been called the subspecialty that “transforms healthcare,”[3] but there are many paths and directions we can take to achieve this. Rather, our greatest connection is our willingness to work with and around data to achieve said goals. Physicians from any specialty of medicine may pursue CIS training, and they may undertake projects unrelated to their home specialty. As CIS continues to grow, KOI pond snapshots and pitches may help provide concrete examples of the various paths other CI diplomates have taken in a way that provides value to those just learning about CI. By creating a common lexicon for describing our activities and skills, we promote our strengths (diversity, adaptability, and problem-solving) to physicians who are considering CIS, to colleagues in other specialties, to employers, and to ourselves.

Clinical Relevance Statement

The CIS is a burgeoning subspecialty that spans all areas of medicine and has expectations to transform healthcare. Its youth and diversity come at a cost, however, as the practice remains poorly understood by the medical community at large. In this paper, we propose a dual-axis model that may serve as the lexicon to help each informatician succinctly describe their unique roles and interests.

Conflict of Interest

None declared.

Acknowledgment

The authors would like to thank Lisa V. Grossman for her assistance with model creation and insights.

Protection of Human and Animal Subjects

No human or animal subjects were included in the project.

• References

• 1 Shortliffe EH, Detmer DE, Munger BS. Clinical Informatics: Emergence of a New Profession. In: Clinical Informatics Study Guide. Springer; Cham; 2016: 3-21
• 2 Detmer DE, Shortliffe EH. Clinical informatics: prospects for a new medical subspecialty. JAMA 2014; 311 (20) 2067-2068
• 3 Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Clinical Informatics. ACGME; 2019. . Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/381_ClinicalInformatics_2019_TCC.pdf . Accessed September 13, 2019
• 4 Zhu L, Zheng WJ. Informatics, data science, and artificial intelligence. JAMA 2018; 320 (11) 1103-1104
• 5 Hersh W. A stimulus to define informatics and health information technology. BMC Med Inform Decis Mak 2009; 9 (01) 24
• 6 Bernstam EV, Smith JW, Johnson TR. What is biomedical informatics?. J Biomed Inform 2010; 43 (01) 104-110
• 7 Hasman A, Ammenwerth E, Dickhaus H. , et al. Biomedical informatics--a confluence of disciplines?. Methods Inf Med 2011; 50 (06) 508-524
• 8 Gardner RM, Overhage JM, Steen EB. , et al; AMIA Board of Directors. Core content for the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 153-157
• 9 Safran C, Shabot MM, Munger BS. , et al; AMIA Board of Directors. Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 158-166
• 10 Kulikowski CA, Shortliffe EH, Currie LM. , et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. J Am Med Inform Assoc 2012; 19 (06) 931-938
• 11 McClintock DS, Levy BP, Lane WJ. , et al. A core curriculum for clinical fellowship training in pathology informatics. J Pathol Inform 2012; 3: 31
• 12 Kannry J, Sengstack P, Thyvalikakath TP. , et al. The chief clinical informatics officer (CCIO): AMIA task force report on CCIO knowledge, education, and skillset requirements. Appl Clin Inform 2016; 7 (01) 143-176
• 13 Leviss J, Kremsdorf R, Mohaideen MF. The CMIO--a new leader for health systems. J Am Med Inform Assoc 2006; 13 (05) 573-578
• 14 Silverman HD, Steen EB, Carpenito JN, Ondrula CJ, Williamson JJ, Fridsma DB. Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis. J Am Med Inform Assoc 2019; 26 (07) 586-593
• 15 Counselman FL, Borenstein MA, Chisholm CD. , et al; EM Model Review Task Force; American Board of Emergency Medicine. The 2013 Model of the Clinical Practice of Emergency Medicine. Acad Emerg Med 2014; 21 (05) 574-598
• 16 Hennink MM. Writing Focus Group Methods. Understanding Focus Group Discussions. 2014: 95-126
• 17 Blasiola GC. Koi: Everything about Selection, Care, Nutrition, Diseases, Breeding, Pond Design and Maintenance, and Popular Aquatic Plants. Barron's Educational Series; 1995 . Available at: https://market.android.com/details?id=book-fBTNYQ4A8FcC
• 18 Rogers EM. Diffusion of Innovations, 5th Edition. Simon and Schuster; 2003. . Available at: https://play.google.com/store/books/details?id=9U1K5LjUOwEC
• 19 Page T. Notions of innovation in healthcare services and products. Int J Innov Sustain Dev 2014; 8 (03) 217-231 . Available at: https://core.ac.uk/download/pdf/42481501.pdf
• 20 Pol E, Ville S. Social innovation: Buzz word or enduring term?. J Socio Econ 2009; 38 (06) 878-885
• 21 Kahn KB. Understanding innovation. Bus Horiz 2018; 61 (03) 453-460
• 22 Baregheh A, Rowley J, Sambrook S. Towards a multidisciplinary definition of innovation. Manage Decis 2009; 47 (08) 1323-1339
• 23 Omachonu VK, Einspruch NG. Innovation in healthcare delivery systems: a conceptual framework. Innov J 2010; 15 (01) 1-20 . Available at: http://www.dphu.org/uploads/attachements/books/books_1028_0.pdf
• 24 Länsisalmi H, Kivimäki M, Aalto P, Ruoranen R. Innovation in healthcare: a systematic review of recent research. Nurs Sci Q 2006; 19 (01) 66-72
• 25 Heinze KL, Weber K. Toward organizational pluralism: institutional intrapreneurship in integrative medicine. Organ Sci 2016; 27 (01) 157-172
• 26 Lega F. Strategic, organisational and managerial issues related to innovation, entrepreneurship and intrapreneurship in the hospital context: Remarks from the Italian experience. J Manag Mark Healthc 2009; 2 (01) 77-93
• 27 Parker SC. Intrapreneurship or entrepreneurship?. J Bus Venturing 2011; 26 (01) 19-34
• 28 Stevenson JE, Euchner J. The role of the chief innovation officer. Res Technol Manag 2013; 56 (02) 13-17
• 29 Tumbas S, Berente N, Brocke JV. Digital innovation and institutional entrepreneurship: chief digital officer perspectives of their emerging role. J Inf Technol Impact 2018; 33 (03) 188-202
• 30 Powell TH, Angwin DN. The role of the chief strategy officer. MIT Sloan Manag Rev 2012; 54 (01) 15 . Available at: https://pure.roehampton.ac.uk/portal/files/1157315/Powell_2012_Sloan.pdf
• 31 Breene RTS, Nunes PF, Shill WE. The chief strategy officer. Harv Bus Rev 2007; 85 (10) 84-93
• 32 Wolf JA. CPXP. Technology, innovation and transforming healthcare faster, smarter and together: A conversation with Dr. Rasu Shrestha. Patient Exp J 2019; 6 (02) 9-11
• 33 AMIA. Clinical Informatics Fellowship Programs | AMIA. Available at: https://www.amia.org/membership/academic-forum/clinical-informatics-fellowships . Accessed March 5, 2019
• 34 Kidd PS, Parshall MB. Getting the focus and the group: enhancing analytical rigor in focus group research. Qual Health Res 2000; 10 (03) 293-308
• 35 Shernoff ES, Maríñez-Lora AM, Frazier SL, Jakobsons LJ, Atkins MS, Bonner D. Teachers supporting teachers in urban schools: what iterative research designs can teach us. School Psych Rev 2011; 40 (04) 465-485
• 36 Perrier L, Kealey MR, Straus SE. An iterative evaluation of two shortened systematic review formats for clinicians: a focus group study. J Am Med Inform Assoc 2014; 21 (e2): e341-e346
• 37 Jaspers MWM. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
• 38 Scariot CA, Heemann A, Padovani S. Understanding the collaborative-participatory design. Work 2012; 41 (Suppl. 01) 2701-2705