Evaluation of a Training Program to Improve Organizational Capacity for Health Systems AnalyticsFunding None.
19 April 2019
06 July 2019
28 August 2019 (online)
Objective The Leadership in Analytics and Data Science (LEADS) course was evaluated for effectiveness. LEADS was a 6-month program for working biomedical and health informatics (BMHI) professionals designed to improve analytics skills, knowledge of enterprise applications, data stewardship, and to foster an analytics community of practice through lectures, hands-on skill building workshops, networking events, and small group projects.
Methods The effectiveness of the LEADS course was evaluated using the Kirkpatrick Model by assessing pre- and postcourse knowledge, analytics capabilities, goals, practice, class lecture reaction, and change in the size of participant professional networks. Differences in pre- and postcourse responses were analyzed with a Wilcoxon signed rank test to determine significance, and effect sizes were computed using a z-statistic.
Results Twenty-nine students completed the course with 96% of respondents reporting that they were “very” or “extremely” likely to recommend the course. Participants reported improvement in several analytics capabilities including Epic data warehousing (p = 0.017), institutional review board policy (p = 0.005), and data stewardship (p = 0.007). Changes in practice patterns mirrored those in self-reported capability. On average, the participant professional network doubled.
Conclusion LEADS was the first course targeted to working BMHI professional at a large academic medical center to have a formal effectiveness evaluation be published in the literature. The course achieved the goals of expansion of BMHI knowledge, skills, and professional networks. The LEADS course provides a template for continuing education of working BMHI professionals.
Protection of Human and Animal Subjects
This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the JHM Institutional Review Board.
- 1 Davenport TH, Harris JG. Competing on Analytics: The New Science of Winning. Boston, MA: Harvard Business Review Press; 2017
- 2 Wachter RM, Howell MD. Resolving the productivity paradox of health information technology: a time for optimism. JAMA 2018; 320 (01) 25-26
- 3 Burns WA. Healthcare data's perfect storm: Why healthcare organizations are drowning in the data they are creating and why they need even more data to weather the storm. Updated 2013. Available at: https://s3.amazonaws.com/rdcms-himss/files/production/public/HIMSSorg/Content/files/HitachiDataSystems_BR-458_perfectStorm.pdf . Accessed February 2, 2019
- 4 Violino B. The secrets of highly successful data analytics teams. Updated 2017. Available at: https://www.cio.com/article/3234353/the-secrets-of-highly-successful-data-analytics-teams.html?upd=1552589464590 . Accessed March 27, 2019
- 5 Mantas J, Ammenwerth E, Demiris G. , et al; IMIA Recommendations on Education Task Force. Recommendations of the international medical informatics association (IMIA) on education in biomedical and health informatics. First revision. Methods Inf Med 2010; 49 (02) 105-120
- 6 Hersh W. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform 2010; 1 (02) 197-212
- 7 Berner ES, Dorsey AD, Garrie RL, Qu H. Assessment-based health informatics curriculum improvement. J Am Med Inform Assoc 2016; 23 (04) 813-818
- 8 Breeden EA, Clauson KA. Development and implementation of a multitiered health informatics curriculum in a college of pharmacy. J Am Med Inform Assoc 2016; 23 (04) 844-847
- 9 Tremblay MC, Deckard GJ, Klein R. Health informatics and analytics - building a program to integrate business analytics across clinical and administrative disciplines. J Am Med Inform Assoc 2016; 23 (04) 824-828
- 10 Jaspers MW, Mantas J, Borycki E, Hasman A. IMIA accreditation of biomedical and health informatics education: current state and future directions. Yearb Med Inform 2017; 26 (01) 252-256
- 11 Singer JS, Cheng EM, Baldwin K, Pfeffer MA. ; UCLA Health Physician Informaticist Committee. The UCLA health resident informaticist program - a novel clinical informatics training program. J Am Med Inform Assoc 2017; 24 (04) 832-840
- 12 Hübner U, Shaw T, Thye J, Egbert N, Marin H, Ball M. Towards an international framework for recommendations of core competencies in nursing and inter-professional informatics: the TIGER competency synthesis project. Stud Health Technol Inform 2016; 228: 655-659
- 13 Dorsey AD, Clements K, Garrie RL, Houser SH, Berner ES. Bridging the gap: a collaborative approach to health information management and informatics education. Appl Clin Inform 2015; 6 (02) 211-223
- 14 Fridsma DB. Developing the health informatics workforce of the future: academic and industry partners. J Am Med Inform Assoc 2017; 24 (03) 677-678
- 15 Khairat S, Sandefer R, Marc D, Pyles L. A review of biomedical and health informatics education: a workforce training framework. J Hosp Adm 2016; 5 (05) 10-20
- 16 Palis AG, Quiros PA. Adult learning principles and presentation pearls. Middle East Afr J Ophthalmol 2014; 21 (02) 114-122
- 17 White SK. The 13 most in-demand tech jobs for 2019 - and how to hire for them. Updated 2018. Available at: https://www.cio.com/article/3235944/hiring-the-most-in-demand-tech-jobs-for-2018.html . Accessed March 27, 2019
- 18 Hersh W, Margolis A, Quirós F, Otero P. Building a health informatics workforce in developing countries. Health Aff (Millwood) 2010; 29 (02) 274-277
- 19 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385
- 20 Furukawa MF, Vibbert D, Swain M. HITECH and health IT jobs: Evidence from online job postings. Updated 2012. Available at: https://www.healthit.gov/sites/default/files/pdf/0512_ONCDataBrief2_JobPostings.pdf . Accessed March 13, 2019
- 21 Anderson JG. Evaluation in health informatics: social network analysis. Comput Biol Med 2002; 32 (03) 179-193
- 22 Park JY, Im I, Chang-Soo S. Is social networking a waste of time? The impact of social network and knowledge characteristics on job performance. Knowl Manag Res Pract 2017; 15 (04) 560-571
- 23 Bennett J, Owers M, Pitt M, Tucker M. Workplace impact of social networking. Property Manage 2009; 28 (03) 138
- 24 Gaggioli A, Mazzoni E, Milani L, Riva G. The creative link: investigating the relationship between social network indices, creative performance and flow in blended teams. Comput Human Behav 2015; 42: 157
- 25 Dinter B, Moslein KM, Kollwitz C. , et al. Combining open innovation and knowledge management for a community of practice - an analytics driven approach. Twenty-second Americas Conference on Information Systems; 2016
- 26 Wenger E, McDermott RA, Snyder W. Cultivating Community of Practice: A Guide to Managing Knowledge. Boston, MA: Harvard Business School Press; 2002
- 27 Nagy P, Kahn Jr CE, Boonn W. , et al. Building virtual communities of practice. J Am Coll Radiol 2006; 3 (09) 716-720
- 28 Wenger-Trayner E, Wenger-Trayner B. . Updated 2015. Available at: Https://Wenger-trayner.com/wp-content/uploads/2015/04/07-brief-introduction-to-communities-of-practice.pdf https://wenger-trayner.com/introduction-to-communities-of-practice/ . Accessed March 27, 2019
- 29 Rouse DN. Employing Kirkpatrick's evaluation framework to determine the effectiveness of health information management courses and programs. Perspect Health Inf Manag 2011; 8: 1c
- 30 Moldovan L. Training outcome evaluation model. Procedia Technol 2016; 22: 1184-1190
- 31 Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco: Berett-Koehler; 1994
- 32 Fast facts: Johns Hopkins Medicine. Updated 2018. Available at: https://www.hopkinsmedicine.org/about/downloads/JHM-Fast-Facts.pdf . Accessed March 27, 2019
- 33 The Data Trust. Updated 2018. Available at: https://intranet.insidehopkinsmedicine.org/data_trust/index.html . Accessed March 27, 2019
- 34 Phillips JJ. Handbook of Training Evaluation and Measurement Methods. 3rd ed. Houston, TX: Gulf Publishing; 1997
- 35 Kerby DS. The simple difference formula: an approach to teaching nonparametric correlation. Compr Psychol 2014; 3 (01) 1-10