Five Years of Clinical Informatics Board Certification for Physicians in the United States of America

Summary

Objectives: To review the highlights of the new Clinical Informatics subspecialty including its history, certification requirements, development of and performance on the certification examination in the United States.

Methods: We reviewed processes for the development of a subspecialty. Data from board certification examinations were collated and analyzed. We discussed eligibility requirements in the fellowship as well as practice pathways.

Results: Lessons learned from the development of the Clinical Informatics subspecialty, opportunities, challenges, and future directions for the field are discussed.

Conclusions: There remains a need for fellowship programs and creation and maintenance of a professional home for the subspecialty with the American Medical Informatics Association. Ongoing attention to the currency of the core content is required to maintain an examination designed to test the key concepts within the field of Clinical Informatics.

Keywords

Clinical Informatics - board certification - fellowship training - examination - American Board of Medical Specialties - physicians

Introduction and Background

In the United States (US), the use of electronic health records (EHRs) has drastically increased in the last decade[1], resulting in a need for a specialized workforce in Clinical Informatics. Recognizing physicians’ expertise in Clinical Informatics and providing them with certification fosters Clinical Informatics workforce capacity building to meet the growing needs in the healthcare industry. Although certification in Clinical Informatics is uncommon worldwide[2], the American Board of Preventive Medicine (ABPM) and the American Board of Pathology (ABPath) offered the first board certification examination in Clinical Informatics to eligible physicians in the US in 2013[3]. The first cohort of successful examinees on the inaugural examination were awarded ABPM board certification in Clinical Informatics in 2014, thereby solidifying Clinical Informatics as an official subspecialty under the umbrella of the American Board of Medical Specialties (ABMS)[4].

The Argument for Board Certification

Board certification recognizes a physician's exceptional expertise in a particular specialty and/or subspecialty of medical practice. In the US, board certification for physicians is voluntary. After completing medical school, additional residency training, and successful passing of United States Medical Licensing Examinations, many states will grant a physician a license to practice medicine. Licensure indicates a minimum skill set and knowledge and is not specialty-specific[5]. Successful post-medical school training in a specialty or subspecialty allows eligible physicians to apply for board certification in their specialty or subspecialty and demonstrate the physician's expertise in that domain. Board certification is important as it allows stakeholders including patients to identify physicians with a specific domain expertise, it allows physicians to demonstrate exceptional expertise in a particular specialty and/or subspecialty of medical practice, and benefits society by encouraging physicians to seek, maintain, and demonstrate a higher level of skills, knowledge, and expertise than is required of non-board certified physicians.

Developing a Subspecialty

In order to be considered for new medical subspecialty status, certain requirements must be met. A new subspecialty needs a professional home in a specialty society, educational and training programs, and sustained research. In the US, the American Medical Informatics Association (AMIA) had been established as the home for informaticians across the spectrum of clinical informatics. While nurses in the US established a certification process in 1992[6], the same was not true for physicians. In 2005, under the leadership of then AMIA CEO and President Don Detmer, AMIA members pursued a subspecialty assignment for physicians for Clinical Informatics with the ABMS and AMIA was elected to membership in the Council of Medical Specialty Societies (CMSS) in 2006.

A new subspecialty further requires commitment to advancing the field through peer-reviewed scholarly engagement and research. Thus, opportunities for researchers to present and disseminate their research in the field are needed. AMIA - recognizing the need for applied clinical informatics research - established two tracks for its annual symposium: foundational and applied research, with Clinical Informatics comprising a key component in the applied track.

While the Journal of the American Medical Informatics Association publishes papers in the field of applied clinical informatics, the increasing adoption of EHRs[7], a growing health informatics workforce[8], the emergence of new roles including Chief Clinical Informatics Officers[9], and increasing dialogue and discourse on the future of the field[10] prompted the creation of the inaugural journal dedicated to Clinical Informatics, Applied Clinical Informatics (ACI) in 2009[11]. ACI's core editorial subject matters include clinical information systems, administrative and management systems, eHealth systems, information technology development, deployment, and evaluation, socio-technical aspects of information technology (IT) and health IT training[12].

AMIA created a code of ethics for this new field[13] [14] and in 2007 began the development of the core content[15] that defines the domain knowledge, key competencies, and skills in Clinical Informatics[16]. After the creation of the core content and the training requirements[17] for Clinical Informatics, AMIA sought an administrative board sponsor for an application to the ABMS[3].

In 2009, the ABPM sponsored the application for the Clinical Informatics Subspecialty to ABMS. In 2010, after a rigorous vetting process, ABMS approved Clinical Informatics as a subspecialty available to diplomates of all 24 ABMS Member Boards. Subsequently, ABPM organized a sub-board in Clinical Informatics including twenty experts in the field and chaired by one of the authors (CUL), which developed the initial item pool for the ABPM Clinical Informatics certification examination. Prior to the administration of the examination in 2013, its development incorporated vetting of the exam and other industry best practices to ensure reliability and validity and to assure that the examination met or exceeded industry standards. This process included a standard setting exercise, which established the passing threshold for the examination.

Board Examination

Content

The core content of the Clinical Informatics subspecialty was created through an AMIA initiative[13] and endorsed by the ABPM[18]. The four major content areas are Fundamentals (10%), Clinical Decision Making and Care Process Improvement (30%), Health Information Systems (40%), and Leading and Managing Change (20%). The examination includes 200 question items. The examination lasts 4.5 hours starting when the examinee is seated at his/ her station. It is broken up into a 15-minute tutorial, four 60-minute blocks (50 items each, for a total of 200), and 15 minutes of break time. Examinees may end any block early at their discretion but cannot return to previous examination blocks. Any remaining block time is added to the allotted break time, which may be taken between blocks.

Consistent with industry best practices, ABPM via the Clinical Informatics subboard reviews core content items annually, develops new items, and periodically reviews the core content outline to ensure relevance and currency of concepts.

Exam Design

Certification examinations must be valid, reliable, and objective as outlined in [Table 1]. ABPM - based on the recommendation of individuals from AMIA and the ABPath - convened a committee of 20 domain experts (17 nominated by AMIA, three by ABPath), who were charged with designing an examination. The charge to this committee, the Sub-board for Clinical Informatics, included the caveat not to create an examination that would assess all aspects of the domain nor to test a candidate's knowledge of the latest-breaking medical news of the day but, instead, create an examination that is specifically designed to test the basic lasting concepts within the field of Clinical Informatics. With this charge in mind, all committee members received formal training in item writing and following well-established best practices created questions congruent with the Clinical Informatics core content. The result of this rigorous process was an examination that included recall questions, interpretive questions, and problem-solving questions, which reflect key concepts that are important and pertinent to the field of Clinical Informatics. The full committee reviewed all questions in the item pool for accuracy and relevance. On an annual basis, the committee augments the item bank by creating new relevant questions based on the core content with the goal of maintaining a robust item pool for future examinations.

Table 1 Attributes required in examination items[19]

Attribute	Definition
Validity	The question measures what it is designed to measure and covers the domain
Reliability	Applying the same test in the future will generate similar scores (Repeatable with the same results)
Objectivity	The performance on the test reflects how well the examinee understands and applies the skill (and not some outside influence)

After each examination, the committee reviews the performance of the individual items (i.e., validity, reliability, and objectivity). Only those items performing within acceptable standards are retained for the purposes of scoring the examination. Items for which statistical analysis confirms they were too difficult, too easy, or did not differentiate well between successful examinees and others may be removed from consideration when determining passing scores for the examination. Additionally, the committee may select items to be retired or sent for re-writing and review to maintain the highest quality of the item bank for future examinations.

Eligibility

To achieve general eligibility for the Clinical Informatics certification, the physician must have graduated from a medical school meeting ABPM standards, hold an active board certification from an ABMS Member Board, hold an unrestricted license to practice medicine in every state or territory in which the physician has a license to practice medicine, and provide a letter of reference from an ABMS-certified physician. Physicians, who are board certified in pathology, must apply for the Clinical Informatics certification through ABPath while applicants from all other ABMS Member Boards seeking Clinical Informatics certification apply through ABPM. The initial ABMS approval of Clinical Informatics allowed for two pathways for certification: A Fellowship Pathway and a Practice Pathway.

Fellowship Pathway

In addition to the general criteria, eligibility criteria for the Clinical Informatics Fellowship Pathway include successful completion of a 24-month full-time Accreditation Council for Graduate Medical Education accredited Clinical Informatics fellowship[21].

The Accreditation Council for Graduate Medical Education (ACGME) accredited the first fellowship programs (Stanford University, University of Illinois at Chicago, and Oregon Health & Science University) in 2014. The first applicants, who qualified via the Fellowship Pathway, were admitted to the examination in 2016.

The AMIA Community of Clinical Informatics Program Directors (CCIPD) provides leadership and supports the continued growth of ACGME accredited fellowships in Clinical Informatics[22].

[Table 2] shows the number of applications and their approval rates in 2017 for the Fellowship Pathway. Applicants applying through the Fellowship Pathway have experienced higher approval rates compared to the Practice Pathway. Only one Fellowship Pathway application was rejected because the applicant completed a program that was not ACGME accredited at the time of application.

Table 2 2017 Application approval rates

	Applications Processed (N)	Applications Approved (N)	Percent Approved
Practice Pathway	290	222	77%
Fellowship Pathway	19	18	95%

Practice Pathway

At the time of the preparation of this manuscript, to be eligible for the board examination under the Practice Pathway, the candidate must fulfill the general eligibility criteria and must demonstrate the completion of a two-year biomedical informatics master's program, two years fellowships sponsored by the National Library of Medicine or the US Department of Veterans Affairs, or must demonstrate 36 months of substantial broad-based professional activity with significant Clinical Informatics responsibility (at least 25% effort) in the five years preceding the application. Candidates may receive partial credit for fellowships of duration less than 24 months, AMIA 10x10 courses (virtual courses utilizing curricular content from existing informatics training programs), or masters-level courses in health informatics, or ABPM-approved research and educational activities in Clinical Informatics[18]. Clinical Informatics diplomates, who live and work outside the US, mainly qualified via the Practice Pathway.

Consistent with ABMS-approved practices for new subspecialties, the Practice Pathway was initially approved for five years. This time interval was extended for an additional five-year period after an ABPM petition to extend the timeline for the Practice Pathway was approved by ABMS. Therefore, applications for board certification in Clinical Informatics via the Practice Pathway will be accepted through the 2022 application cycle.

Over the initial five years that the examination has been given, the percentage of applicants in the Practice Pathway, who did not meet the eligibility criteria, increased annually. While the eligibility criteria for the Practice Pathway and the committee members reviewing the applications have not changed since the first application cycle in 2013, the candidate pool appears to have changed over the five years the examination has been administered with increasing number of candidates, who have not provided sufficient evidence of Clinical Informatics activity (e.g., clinical domain experts, who collaborated with IT departments to generate health IT artifacts such as order sets, decision support, and documentation).

Beyond insufficient Clinical Informatics experience, other reasons applicants do not meet eligibility criteria include insufficient time (lees than 36 months) in Clinical Informatics practice, less than 25% effort during the 36 months, double counting of training (e.g., the applicant is requesting credit for time spent in another ACGME-accredited residency or fellowship program), and incomplete applications. [Table 2] shows the number of applicants and their approval rate in 2017 for the Practice Pathway.

Exam Results

Each year, the Clinical Informatics subboard examination is developed using a unique combination of questions from the item pool. While the questions in the examination vary from year to year, the examination difficulty, by design, remains consistent across examination cycles. [Table 3] shows the pass rates since the first examination.

Table 3 Exam results 2013-2017

	Examination (N)	Certification (N)	Pass Rate (%)
2013	488	445	91
2014	367	329	90
2015	400	320	80
2016	472	401	85
2017	249	192	77
Total	1,976	1,687	85

Fellowships

Beginning in 2023, only candidates trained in an ACGME-accredited fellowship program will be eligible for the Clinical Informatics board examination. As of December 13, 2017, 24 ACGME-accredited Clinical Informatics fellowship programs were participating in the match for fellows in the US. Of interest to an international audience is the fact that ACGME-International accredited programs exist[22], offering the opportunity to establish ACGME-accredited programs internationally but keeping in mind that candidates must meet all current ABPM requirements including, but not limited to licensure and primary certification via an ABMS Member Board.

Establishing and Accrediting a Fellowship

ACGME restricts the primary specialties that may host a Clinical Informatics program to Anesthesiology, Diagnostic Radiology, Emergency Medicine, Family Medicine, Internal Medicine, Medical Genetics and Genomics, Pathology, Pediatrics, or Preventive Medicine. An eligible institution interested in creating a Clinical Informatics fellowship needs to complete an application with ACGME. The application will be reviewed and if approved, the new program can recruit fellows. Usually after the first year, ACGME reviewers will evaluate the program on site. The specific program requirements can be found on the ACGME website under Preventive Medicine[23].

Fellows require opportunities to acquire a comprehensive knowledge set. Most programs solve this need by subscribing to Clinical Informatics certification programs that fellows can participate in remotely. Some programs have on site master's programs in Clinical Informatics that fellows complete during the fellowship [Vanderbilt University Clinical Informatics Fellowship Program. Available online at https://www.vumc.org/dbmi/clinical-informatics-fellowship-pro-gram. Last accessed 3/30/2018].

Fellows must further have ample opportunity to gain practical experience. Many programs embed their fellows in health information technology operations and create a rotation schedule (block diagram) that permits fellows to observe and experience various aspects of health information technology.

Milestones

ACGME develops Milestones for all of its training specialties including Clinical Informatics[24]. Milestones are points of achievement along the path of a fellow's education from novice to expert[25]. Twice annually, program directors must compare their fellows’ performance to the milestones. The program's Clinical Competency Committee reviews the assessments and reports them to ACGME. [Figure 1] shows a sample milestone.

Fig. 1 Sample milestone for Clinical Informatics

Financial Challenges

With salary and benefits (including the cost of a master's degree), the cost of Clinical Informatics fellowship training can range from $100,000 to $150,000 per fellow annually not including administrative support. Unlike other clinical specialties and subspecialties, Clinical Informatics in the US does not have any billing codes and does not generate revenue for the “practice of clinical informatics”. These are required to bill health care services to a payer. Unfortunately, a clinical informatician, who performs services that reduce cost[26] [27], improve safety[28] [29], reduce length of stay[30] for patients and payers, may not be able to bill for the work provided. In other clinical specialties, the presence of a fellow increases the number of patients a physician can see, resulting in more billable events, which can justify the employment of a fellow. In Clinical Informatics (and other specialties like Public Health and General Preventive Medicine), this incentive does not exist and as a result programs have to be innovative in how they fund their training program[31].

Roles of ABPM and AMIA

AMIA is the professional home of the Clinical Informatics subspecialty in the US and provides education, networking (like CCIPD), and research opportunities. ABPM is the administrative home of the Clinical Informatics board certification, while ABPM and ABPath are the sponsoring boards. Both organizations work closely to align their activities. For example, AMIA provides educational activities for the maintenance of certification program. ABPM reviews and approves these activities.

Updating of the Core Content

Few specialties or subspecialties have seen their domains change as rapidly in the last five year as Clinical Informatics. New applications of informatics like precision medicine[32], quantified self[33], and wearable sensors[34], are starting to gain momentum, traction, and importance in the application of Clinical Informatics to patient care and may have to be included in the core content of the subspecialty in the future once they have reached mainstream status[15].

In this dynamic field of Clinical Informatics, ABPM is dedicated to the integrity and relevance of the examination and is therefore committed to ongoing evaluation of the core content. Therefore, when advancements in the industry rise to the level of becoming core to the practice, ABPM will revise the core content as necessary to accommodate those advances and simultaneously update the Clinical Informatics sub-board item bank. Consistent with this goal, in 2017, ABPM and AMIA pledged to partner on a comprehensive review of the core content for the Clinical Informatics subspecialty, which in turn informs the annual update to the examination's question pool. Efforts such as the framework developed by AMIA for the Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM) will inform this collaboration[35]. ABPM and AMIA anticipate this effort to begin in 2018 with an estimated duration of 24 to 36 months.

Future Opportunities

For ABPM, there are a number of opportunities in the context of the Clinical Informatics board certification. In 2017, administration of the examination was transferred to the National Board of Medical Examiners (NBME). This transition has strengthened the metrics and technical processes involved in the exam preparation process.

AMIA is currently formalizing a leadership structure within its Clinical Informatics Community of Practice (CICOP) for improved communications between AMIA and ABPM. As the professional home of the Clinical Informatics subspecialty, AMIA is developing a Fellow designation for Clinical Informatics diplomates, who make an ongoing commitment to lifelong learning practicing the medical subspecialty of Clinical Informatics. Both efforts will enable diplomates to surface key issues and communicate the value of the Clinical Informatics subspecialty certification.

Additional opportunities include the development of new fellowships in Clinical Informatics to foster the pipeline for physicians certified in Clinical Informatics once the Practice Pathway expires in 2022. Working with ABMS to evaluate and appropriately design the program for maintenance of certification that meets stakeholder needs will also be of importance. In this regard, ABMS is spearheading the Continuing Certification Visioning Initiative, which is designed to solicit meaningful input from all stakeholders and which will inform the next generation of standards as they relate to programs for maintenance of certification.

The demand on applicants by the sub-specialty certification requirements to meet eligibility criteria fosters high quality fellowship training, primary board certification, and licensure and will naturally entail that not every interested person will be able to meet the eligibility criteria. However, without strict requirements and without the termination of the practice pathway in 2022, the incentives for obtaining fellowship training would be undermined and fellowship programs would suffer, ultimately jeopardizing the subspecialty itself. A potential solution to this dilemma would be a collaboration between ABPM and ACGME to propose an alternative approach to fellowship training that would recognize the challenges of mid-career physicians and would ensure that relevant high quality experiences in Clinical Informatics are defined and made available to interested clinicians. A viable proposal would require a specific plan and ideally a pilot agreement from one or more accredited fellowship programs. The ACGME has not addressed this issue directly but similar problems exist for many newly developed sub-specialties and a solution may be applicable to a variety of other disciplines. The authors are supportive of further collaboration with various stakeholders in an effort to develop innovative approaches to high quality training and experiential learning that facilitate and ensure attainment of core knowledge in the field while maintaining the integrity of the standards that form the foundation of this most important subspecialty.

As the Clinical Informatics field continues to grow and as more organizations and offices implement EHRs, there will be an increasing need for physicians with demonstrated Clinical Informatics expertise to address the challenges in implementation, clinical decision support, workflow, documentation, and many other areas. Few data exist on the career opportunities of newly certified Clinical Informaticians. Tracking their career paths and their employment options will be an important task for AMIA.

Conclusions

In the US, Clinical Informatics became a board-certified subspecialty in 2013. Since its inception in 2013, the number of ABPM board-certified Clinical Informatics diplomates has risen and is currently over 1,400. Creating a new subspecialty including examination, fellowships, and professional infrastructure is challenging and requires collaboration of professionals from across the specialty.

The US model for Clinical Informatics board certification has been successfully implemented but its long-term viability and sustainability will be determined by the success of Clinical Informatics fellowships. It is our anticipation that increased demand for clinicians with Clinical Informatics skills will drive fellowship training in the future.

• References

• 1 Lehmann CU, O'Connor KG, Shorte VA, Johnson TD. Use of electronic health record systems by office-based pediatricians. Pediatrics 2015; Jan; 135 (01) e7-15
• 2 Gundlapalli AV, Gundlapalli AV, Greaves WW, Kesler D, Murray P, Safran C, Lehmann CU. Clinical Informatics Board Specialty Certification for Physicians: A Global View. Stud Health Technol Inform 2015; 216: 501-5 . PubMed PMID: 26262101
• 3 American Medical Informatics Association. History of the Clinical Informatics Subspecialty. Available online at https://www.amia.org/clini-cal-informatics-board-review-course/history . Last accessed 12/10/2017
• 4 Lehmann CU, Shorte V. Gundlapalli AV Clinical informatics sub-specialty board certification. Pediatr Rev 2013; Nov; 34 (11) 525-30
• 5 American Board of Medical Specialties. Board Certification and Maintenance of Certification. Available online at http://www.abms.org/board-certification/ . Last accessed 12/13/2017
• 6 Cummins MR, Gundlapalli AV, Murray P, Park HA, Lehmann CU. Nursing Informatics Certification Worldwide: History, Pathway, Roles, and Motivation Yearb Med Inform 2016; 264-71
• 7 Jha AK, Doolan D, Grandt D, Scott T, Bates DW. The use of health information technology in seven nations. Int J Med Inform 2008; Dec; 77 (12) 848-54
• 8 Fenton SH, Gongora-Ferraez MJ, Joost E. Health information technology knowledge and skills needed by HIT employers. Appl Clin Inform 2012; Dec 5; 3 (04) 448-61
• 9 Kannry J, Sengstack P, Thyvalikakath TP, Poikonen J, Middleton B, Payne T. , et al. The Chief Clinical Informatics Oficer (CCIO): AMIA Task Force Report on CCIO Knowledge, Education, and Skillset Requirements. Appl Clin Inform 2016; Mar 16; 7 (01) 143-76
• 10 Kim GR, Lehmann CU. In search of dialogue and discourse in applied clinical informatics. Appl Clin Inform 2009; Oct 14; 0 (01) 1-7
• 11 Haux R, Geissbuhler A, Holmes J, Jaulent MC, Koch S, Kulikowski CA. , et al. On Contributing to the Progress of Medical Informatics as Publisher. Yearb Med Inform 2017; Aug; 26 01: 9-15
• 12 Thieme Verlag, Lehmann CU. Applied Clinical Informatics. Available online at https://www.thieme.com/books-main/clinical-informatics/product/4433-aci-applied-clinical-informatics . Last accessed 3/30/2018
• 13 Detmer DE, Munger BS, Lehmann CU. Clinical informatics board certification: history, current status, and predicted impact on the clinical informatics workforce. Appl Clin Inform 2010; Feb 10; 1 (01) 11-8
• 14 Goodman KW, Adams S, Berner ES, Embi PJ, Hsi-ung R, Hurdle J, Jones DA, Lehmann CU, Maulden S, Petersen C, Terrazas E, Winkelstein P. AMIA's Code of Professional and Ethical Conduct. J Am Med Inform Assoc. 2013; Jan-Feb; 20 (01) 141-143
• 15 Gardner RM, Overhage JM, Steen EB, Munger BS, Holmes JH, Williamson JJ. , et al; AMIA Board of Directors. Core content for the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; Mar-Apr; 16 (02) 153-7
• 16 Detmer DE. Capacity building in e-health and health informatics: a review of the global vision and informatics educational initiatives of the American Medical Informatics Association. Yearb Med Inform 2010; 101-5
• 17 Safran C, Shabot MM, Munger BS, Holmes JH, Steen EB, Lumpkin JR. , et al; AMIA Board of Directors. Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; Mar-Apr; 16 (02) 158-66
• 18 American Board of Preventive Medicine. Clinical Informatics 2017 Examination Blueprint. Available online at https://www.theabpm.org/wp-content/uploads/2017/09/2017CI-Content-Outline.pdf . Last accessed 12/13/2017
• 19 Boeder N, Holzer M, Schelling J. Realisation ofthe guidelines for faculty-internal exams at the Department of General Medicine at the University of Munich: Pushing medical exams one step ahead with IMSm. GMS Z Med Ausbild 2012; 29 (03) Doc42
• 20 American Board of Preventive Medicine. Sub-Specialty Requirements Clinical Informatics. Available online at 20. https://www.theabpm.org/theabpm.org/wp-content/uploads/2017/05/ClinicalInformatics.pdf . Last accessed 12/11/2017
• 21 AMIA. Clinical Informatics Fellowship Programs. Available online at https://www.amia.org/member-ship/academic-forum/clinical-informatics-fellowships . Last accessed 12/15/2017
• 22 ACGME-International. . Available online at www.acgme-i.org/ . Last accessed 12/13/2017
• 23 Accreditation Council for Graduate Medical Education. Preventive Medicine. Available online at http://www.acgme.org/Specialties/Overview/pfcatid/20 . Last accessed 12/28/2017.
• 24 Accreditation Council for Graduate Medical Education. The Clinical Informatics Milestone Project. Available online at https://www.acgme.org/Portals/0/PDFs/Milestones/ClinicalInformat-icsMilestones.pdf . Last accessed 12/13/2017
• 25 Silverman H, Lehmann CU, Munger B. Milestones: Critical Elements in Clinical Informatics Fellowship Programs Appl Clin Inform 2016; Mar 23; 7 (01) 177-90
• 26 Stenner SP, Chakravarthy R, Johnson KB, Miller WL, Olson J, Wickizer M. , et al. ePrescribing: Reducing Costs through In-Class Therapeutic Interchange. Appl Clin Inform 2016; Dec 14; 7 (04) 1168-1181
• 27 Sick AC, Lehmann CU, Tamma PD, Lee CK, Agwu AL. Sustained savings from a longitudinal cost analysis of an internet-based preapproval antimicrobial stewardship program. Infect Control Hosp Epidemiol 2013; Jun; 34 (06) 573-80
• 28 Lehmann CU, Conner KG, Cox JM. Preventing provider errors: online total parenteral nutrition calculator. Pediatrics 2004; Apr; 113 (04) 748-53
• 29 Kim GR, Chen AR, Arceci RJ, Mitchell SH, Kokoszka KM, Daniel D. , et al. Error reduction in pediatric chemotherapy: computerized order entry and failure modes and effects analysis. Arch Pediatr Adolesc Med 2006; May; 160 (05) 495-8
• 30 Westbrook JI, Li L, Raban MZ, Baysari MT, Mum-ford V, Prgomet M. , et al. Stepped-wedge cluster randomised controlled trial to assess the effectiveness of an electronic medication management system to reduce medication errors, adverse drug events and average length of stay at two paediatric hospitals: a study protocol. BMJ Open 2016; Oct 21; 6 (10) e011811
• 31 Lehmann CU, Longhurst CA, Hersh W, Mohan V, Levy BP, Embi PJ. , et al. Clinical Informatics Fellowship Programs: In Search of a Viable Financial Model: An open letter to the Centers for Medicare and Medicaid Services. Appl Clin Inform 2015; Apr 15; 6 (02) 267-70
• 32 Sandhu C, Qureshi A, Emili A. Panomics for Precision Medicine. Trends Mol Med. 2017; Dec 4. pii: S1471-4914 (17)30201-0. [Epub ahead of print]
• 33 Almalki M, Gray K, Martin-Sanchez FJ. Refining the Concepts of Self-quantification Needed for Health Self-management. A Thematic Literature Review. Methods Inf Med 2017; Jan 9; 56 (01) 46-54
• 34 Kekade S, Hseieh CH, Islam MM, Atique S, Mohammed Khalfan A, Li YC. , et al. The usefulness and actual use of wearable devices among the elderly population. Comput Methods Programs Biomed 2018; Jan; 153: 137-159
• 35 AMIA Accreditation Committee. Knowledgeframework. Available online at http://www.cahiim.org/documents/FINAL%20AMIA%20Health%20Informatics%20Core%20Competencies%20for%20CAHIIM.pdf . Last accessed 12/14/2017