Evidence-based Health Informatics: How Do We Know What We Know?
13 November 2014
accepted: 04 March 2015
22 January 2018 (online)
Background: Health IT is expected to have a positive impact on the quality and efficiency of health care. But reports on negative impact and patient harm continue to emerge. The obligation of health informatics is to make sure that health IT solutions provide as much benefit with as few negative side effects as possible. To achieve this, health informatics as a discipline must be able to learn, both from its successes as well as from its failures.
Objectives: To present motivation, vision, and history of evidence-based health informatics, and to discuss achievements, challenges, and needs for action.
Methods: Reflections on scientific literature and on own experiences.
Results: Eight challenges on the way towards evidence-based health informatics are identified and discussed: quality of studies; publication bias; reporting quality; availability of publications; systematic reviews and meta-analysis; training of health IT evaluation experts; translation of evidence into health practice; and post-market surveil-lance. Identified needs for action comprise: establish health IT study registers; increase the quality of publications; develop a taxonomy for health IT systems; improve indexing of published health IT evaluation papers; move from meta-analysis to meta-summaries; include health IT evaluation competencies in curricula; develop evidence-based implementation frameworks; and establish post-marketing surveillance for health IT.
Conclusions: There has been some progress, but evidence-based health informatics is still in its infancy. Building evidence in health informatics is our obligation if we consider medical informatics a scientific discipline.
- 1 Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med 2014; 160 (01) 48-54.
- 2 Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, D.C: The National Academic Press; 2011
- 3 Dowling Jr AF. Do hospital staff interfere with computer system implementation?. Health Care Manage Rev 1980; 5 (04) 23-32.
- 4 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
- 5 Magrabi F, Ong MS, Runciman W, Coiera E. Patient safety problems associated with heathcare information technology: an analysis of adverse events reported to the US Food and Drug Administration. AMIA Annu Symp Proc 2011. 2011: 853-857.
- 6 ECRI Institute. Deep Dive: Health Information Technology. 2012. [accessed Jan 14, 2015]; Available from http://www.healthit.gov/facas/sites/faca/files/STF_Deep_Dive_Health_Information_Technology_2014-06-13.pdf.
- 7 Goodman KW, Adams S, Berner ES, Embi PJ, Hsiung R, Hurdle J. et al. AMIA’s code of professional and ethical conduct. J Am Med Inform Assoc 2013; 20 (01) 141-143.
- 8 Haux R. Medical informatics: past, present, future. Int J Med Inform 2010; 79 (09) 599-610.
- 9 Koppel R. Is healthcare information technology based on evidence?. Yearb Med Inform 2013; 8 (01) 7-12.
- 10 Sackett D, Rosenberg W, Gray J, Haynes R, Richardson S. Evidence based medicine: what it is and what it isn’t. BMJ 1996; 312 7023 71-72.
- 11 Ammenwerth E, de Keizer N. A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics. J Am Med Inform Assoc 2007; 14 (03) 368-371.
- 12 Ware GO, Park MK. Evaluation of search time for two computerized information retrieval systems at the University of Georgia. J Chem Doc 1972; 12 (04) 224-227.
- 13 McFarlane AH, Norman GR. A medical care information system: evaluation of changing patterns of primary care. Med Care 1972; 10 (06) 481-487.
- 14 van der Loo R. Overview of Published Assessment and Evaluation Studies. In van Gennip EMSJ, Talmon JS. editors Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995: 261-282.
- 15 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: Trends in evaluation research 1982–2002. Methods Inf Med 2005; 44: 44-56.
- 16 Krobock JR. A taxonomy: hospital information systems evaluation methodologies. J Med Syst 1984; 8 (05) 419-429.
- 17 Gremy F. Hardware, software, peopleware, subjectivity. A philosophical promenade. Methods Inf Med 2005; 44 (03) 352-358.
- 18 Flagle C, Grémy F, Perry S. editors Assessment of Medical Informatics Technology Joint Working Conference Montpellier du 22 au 26 octobre 1990. Rennes: Éditions ENSP; 1991
- 19 Anderson JG, Aydin CE, Jay SJ. editors Evaluating Health Care Information Systems - Methods and Applications. London, New Delhi: Sage Publications; 1994
- 20 van Gennip E, Talmon J. editors Assessment and evaluation of information technologies in medicine. Amsterdam: IOS Press; 1995
- 21 Lorenzi NM, Riley RT. Managing Change: An Overview. J Am Med Inform Assoc 2000; 7 (02) 116-124.
- 22 Anderson J, Aydin C. editors Evaluating the Organizational Impact of Healthcare Information Systems. New York: Springer; 2005
- 23 Friedman C, Wyatt JC. Evaluation Methods in Medical Informatics. 2nd ed New York: Springer; 2006
- 24 Brender J. Handbook of evaluation methods for health informatics. Burlington, MA: Elsevier Academic Press; 2006
- 25 Rigby M, Ammenwerth E, Beuscart-Zephir M, Brender J, Hyppönen H, Melia S. et al. Evidence Based Health Informatics: 10 years of efforts to promote the principle. Yearb Med Inform 2013; 8 (01) 34-46.
- 26 Haynes RB, Hayward RS, Jadad AR, Sebaldt RJ. Evidence based health informatics: an overview of the Health Information Research Unit at McMaster University. Leadersh Health Serv 1996; 5 (03) 41-44.
- 27 Rigby M. Evaluation: 16 Powerful Reasons Why Not to Do It - And 6 Over-Riding Imperatives. In Patel V, Rogers R, Haux R. editors Proceedings of the 10th World Congress on Medical Informatics (Medinfo 2001). Amsterdam: IOS Press; 2001: 1198-1202.
- 28 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (02) 105-120.
- 29 Seroussi B, Jaulent MC, Lehmann CU. Looking for the evidence: value of health informatics. Editorial. Yearb Med Inform 2013; 8 (01) 4-6.
- 30 UMIT, EFMI WG Eval. Health IT Evaluation Database. 2014. [accessed Jan 14, 2015]. Available from http://evaldb.umit.at.
- 31 Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of Health Information Sys-tems - Problems and Challenges. Int J Med Inform 2003; 71 2–3 125-135.
- 32 Heathfield H, Buchan I. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313 7063 1008.
- 33 Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316: 1959-1961.
- 34 Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: a case study. MIS Quarterly 1988; 12 (04) 571-586.
- 35 Heathfield H, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating Large Scale Health Information Systems: From Practice Towards Theory. In Masys D. editor AMIA Annual Fall Symposium. Philadelphia: Hanley & Belfus; 1997: 116-120.
- 36 Sim J, Sharp K. A critical appraisal of the role of triangulation in nursing research. Int J Nurs Stud 1998; 35 1–2 23-31.
- 37 Barbour RS. Mixing qualitative methods: quality assurance or qualitative quagmire?. Qual Health Res 1998; 8 (03) 352-361.
- 38 Grémy F, Degoulet P. Assessment of health information technology: which questions for which systems? Proposal for a taxonomy. Med Inform 1993; 18 (03) 185-193.
- 39 Kaplan B. An Evaluation Model for Clinical Information Systems: Clinical Imaging Systems. In Greenes R, Peterson H, Protti D. editors Medinfo 95- Proceedings of the 8th World Congress on Medical Informatics. Amsterdam: North Holland; 1995: 1087.
- 40 AHRQ. Ageny for Healthcare Research and Quality: AHRQ Evaluation Toolkit. 2009. [accessed Jan 14, 2015]. Available from http://healthit.ahrq.gov/health-it-tools-and-resources/health-it-evaluation-toolkit-and-evaluation-measures-quick-reference.
- 41 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for Good Evaluation Practice in Health Informatics (GEP-HI). Int J Med Inform 2011; 80: 815-827.
- 42 Malicki M, Marusic A, Consortium O. Is there a solution to publication bias? Researchers call for changes in dissemination of clinical research results. J Clin Epidemiol 2014; 67 (10) 1103-1110.
- 43 Friedman C, Wyatt J. Publication bias in Medical Informatics. J Am Med Inform Assoc 2001; 8 (02) 189-191.
- 44 Wager E, Williams P. Consortium of Project Overcome failure to Publish Negative Findings. “Hardly worth the effort?” Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ 2013; 347: f5248.
- 45 Eysenbach G. Tackling publication bias and selective reporting in health informatics research: register your eHealth trials in the International eHealth Studies Registry. J Med Internet Res 2004; 6 (03) e35.
- 46 Ammenwerth E, Schnell-Inderst P, Siebert U. Vision and challenges of Evidence-Based Health Informatics: a case study of a CPOE meta-analysis. Int J Med Inform 2010; 79 (04) e83-8.
- 47 Peute LW, Driest KF, Marcilly R, Bras Da Costa S, Beuscart-Zephir MC, Jaspers MW. A framework for reporting on human factor/usability studies of health information technologies. Stud Health Technol Inform 2013; 194: 54-60.
- 48 de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005?. Int J Med Inform 2008; 77 (01) 41-49.
- 49 Schulz KF, Altman DG, Moher D, Group C. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c332.
- 50 Bossuyt PM, Reitsma J, Bruns D, Gatsonis C, Glasziou P, Irwig L. et al. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Int Med 2003; 138 (01) 40-44.
- 51 von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007; 370 9596 1453-1457.
- 52 Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010; 8 (05) 336-341.
- 53 EQUATOR. The Equator Network: Enhancing the Quality and Transparency of Health Research. 2014. [accessed Jan 14, 2015]. Available from www.equator-network.org/.
- 54 Talmon J, Ammenwerth A, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9.
- 55 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 4 (03) 331-358.
- 56 Dixon BE, Zafar A, McGowan JJ. Development of a taxonomy for health information technology. Stud Health Technol Inform 2007; 129 Pt 1 616-620.
- 57 AHRQ. Agency for Healthcare Research and Quality: Health IT Costs and Benefit Database. 2009. [accessed Jan 14, 2015]. Available from http://healthit.ahrq.gov/health-it-tools-and-resources/health-it-costs-and-benefits-database.
- 58 ISO. ISO/TR 14639:2012: Health informatics - Capacity-based eHealth architecture roadmap; 2012.
- 59 Cravens GD, Dixon BE, Zafar A, McGowan JJ. A health information technology glossary for novices. AMIA Annu Symp Proc. 2008: 917.
- 60 Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011. [accessed Jan 14, 2015]. Available from http://www.cochrane-handbook.org.
- 61 Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review. J Am Med Inform Assoc 2008; 15 (05) 585-600.
- 62 Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43.
- 63 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E. et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752.
- 64 Vervloet M, Linn AJ, van Weert JC, de Bakker DH, Bouvy ML, van Dijk L. The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: a systematic review of the literature. J Am Med Inform Assoc 2012; 19 (05) 696-704.
- 65 Cappuccio FP, Kerry SM, Forbes L, Donald A. Blood pressure control by home monitoring: meta-analysis of randomised trials. Bmj 2004; 329 7458 145.
- 66 Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996; 3 (06) 399-409.
- 67 Walton R, Dovey S, Harvey E, Frreemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ 1999; 318: 984-990.
- 68 Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health 2014; 17 (01) 30.
- 69 Jones ML. Application of systematic review methods to qualitative research: practical issues. J Adv Nurs 2004; 48 (03) 271-278.
- 70 Sandelowski M, Barroso J, Voils I C. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res Nurs Health 2007; 30 (01) 99-111.
- 71 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (02) 105-120.
- 72 Ammenwerth E, Craven C, Georgiou A, Mantas J. Health IT Evaluation in Health Informatics Curricula: International Overview and Recommendations. Workshop at Medical Informatics Europa (MIE2014), 1.9.2014, Istanbul. 2014. [accessed Jan 14, 2015]. Available from http://person.hst.aau.dk/ska/MIE2014/WorkshopsAndPanels/W03_ID_472.pdf.
- 73 Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B. et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005; 352 (10) 969-977.
- 74 Cresswell KM, Bates DW, Williams R, Morrison Z, Slee A, Coleman J. et al. Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two ‘early adopter’ hospitals. J Am Med Inform Assoc 2014; 21 e (02) e194-202.
- 75 Eslami S, Abu-Hanna A, de Keizer NF, de Jonge E. Errors associated with applying decision support by suggesting default doses for aminoglycosides. Drug Saf 2006; 29 (09) 803-809.
- 76 Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 Suppl (Suppl. 01) S69-76.
- 77 Durieux P. Electronic medical alerts--so simple, so complex. N Engl J Med 2005; 352 (10) 1034-1036.
- 78 Gross PA, Greenfield S, Cretin S, Ferguson J, Grimshaw J, Grol R. et al. Optimal methods for guideline implementation: conclusions from Leeds Castle meeting. Med Care 2001; 39 (Suppl. 08) Suppl 2 II85-92.
- 79 Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003; 362 9391 1225-1230.
- 80 Grol R, Grimshaw J. Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv 1999; 25 (10) 503-513.
- 81 FDA. MAUDE - Manufacturer and User Facility Device Experience Database. 2014. [accessed Jan 14, 2015]. Available from http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/search.cfm.
- 82 Magrabi F, Ong MS, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (01) 45-53.
- 83 Meeks DW, Smith MW, Taylor L, Sittig DF, Scott JM, Singh H. An analysis of electronic health record-related patient safety concerns. J Am Med Inform Assoc 2014; 21 (06) 1053-1059.
- 84 Lovis C. Evidence-based Biomedical Informatics. The Long Way from Pioneer to Science. Yearb Med Inform 2013; 8 (01) 47-50.
- 85 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112.
- 86 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of Unintended Consequences Related to Computerized Provider Order Entry. J Am Med Inform Assoc 2006; 13 (05) 547-556.
- 87 van der Sijs H, Kowlesar R, Aarts J, Berg M, Vulto A, van Gelder T. Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009; 65 (09) 919-925.
- 88 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 7–8 519-526.
- 89 Low DK, Reed MA, Geiduschek JM, Martin LD. Striving for a zero-error patient surgical journey through adoption of aviation-style challenge and response flow checklists: a quality improvement project. Paediatr Anaesth 2013; 23 (07) 571-578.
- 90 Randell R. Medicine and aviation: a review of the comparison. Methods Inf Med 2003; 42 (04) 433-436.
- 91 Sullivan F. What is health informatics?. J Health Serv Res Policy 2001; 6 (04) 251-254.
- 92 Uckert F, Ammenwerth E, Dujat C, Grant A, Haux R, Hein A. et al. Past and next 10 years of medical informatics. J Med Syst 2014; 38 (07) 74.
- 93 Hoyt R. Evidence-based health informatics: Replacing hype with science. 2013. [accessed Jan 14, 2015]. Available from https://prezi.com/zsbc329-25db/evidence-based-health-informatics.
- 94 Godlee F. How guidelines can fail us. BMJ 2014; 349: g5448.
- 95 Cochrane A. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust; 1972