Special Section: Assessing Information Technologies for Health: Evaluation and Implementation: A Call for Action

Summary

Objectives

To raise awareness for actions that are urgently needed to accompany the large scale implementations of ICT in Health Care that are currently taking place in many countries around the world.

Methods

An analysis of a few studies that have recently been described in the literature guided by recent suggestions for research and development of evaluation of health ICT.

Results and Conclusion

Six specific recommendations for action are specified:

Development of good implementation practice,

Development of an experience base of implementation of ICT in health care,

Setting up a surveillance system for unintended effects,

Build an evidence base of best evaluation practice,

Developing guidelines for proper reporting of evaluation studies,

Education of clinicians and decision makers.

Keywords

Medical informatics - medical informatics evaluation - assessement

• References

• 1 Adams ID, Chan M, Clifford PC, Cooke WM, Dallos V, de Dombal FT. et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. Br Med J (Clin Res Ed) 1986; 293 6550 800-4.
• 2 Willems JL. Common standards for quantitative electrocardiography. J Med Eng Technol 1985; 09 (05) 209-17.
• 3 Berner ES, Webster GD, Shugerman AA, Jackson JR, Algina J, Baker AL. et al. Performance of four computer-based diagnostic systems. N Engl J Med 1994; 330 (25) 1792-6.
• 4 Talmon J, Fox J. editors. Knowledge Based Systems in Medicine: Methods, Applications and Evaluation. Berlin: Springer; 1991
• 5 van Gennip E, Talmon J. editors. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995
• 6 Talmon J, Enning J, Castaneda G, Eurlings F, Hoyer D, Nykanen P. et al. The VATAM guidelines. Int J Med Inform 1999; 56 1-3 107-15.
• 7 Lorenzi NM, Riley RT. Organizational Aspects of Health Informatics: Managing Technological Change. New York: Springer-Verlag; 1995
• 8 Lorenzi NM, Riley RT, Ball MJ, Douglas JV. Transforming Health Care through Information: Case Studies. New York: Springer-Verlag; 1995
• 9 Jørgensen T. Measuring effect. In: van Gennip E, Talmon J. editors. Assessment and Evaluation of Information Technologies in Meicine. Amsterdam: IOS Press; 1995: 99-109.
• 10 Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: guidance for the organizational decision maker. Methods Inf Med 2003; 42 (04) 458-62.
• 11 Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P. et al. A methodology for evaluation of knowledge-based systems in medicine. Artif Intell Med 1994; 06 (02) 107-21.
• 12 Brender J, Talmon J, Nykanen P, McNair P, Demeester M, Beuscart R. On the Evaluation of System Integration. In: van Gennip E, Talmon J. editors. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995: 189-208.
• 13 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-31.
• 14 Berg M, Aarts J, van der Lei J. ICT in health care: sociotechnical approaches. Methods Inf Med 2003; 42 (04) 297-301.
• 15 Ammenwerth E, Brender J, Nykanen P, Prokosch HU, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 2004; 73 (06) 479-91.
• 16 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
• 17 Bend J. Public Value and e-Health. In: Insitute for Public Policy Research 2004
• 18 Kohn L, Corrigan J, Donaldson M. editors. To err is human: Building a Safer Health System. Washington DC: National Academy Press; 2000
• 19 Medicine Io. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: National Academic Press; 2001
• 20 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE. et al. Role of computerized physician order entry systems in facilitating medication errors. Jama 2005; 293 (10) 1197-203.
• 21 Shortliffe EH. CPOE and the facilitation of medication errors. J Biomed Inform 2005; 38 (04) 257-8.
• 22 Bates DW. Computerized physician order entry and medication errors: finding a balance. J Biomed Inform 2005; 38 (04) 259-61.
• 23 Nemeth C, Cook R. Hiding in plain sight: what Koppel et al. tell us about healthcare IT. J Biomed Inform 2005; 38 (04) 262-3.
• 24 Horsky J, Zhang J, Patel VL. To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005; 38 (04) 264-6.
• 25 Briggs R, deVreede G-J, Nunamaker J, Sprague R. Special issue: Information systems success. editorial. J Management Information Systems 2003; 19 (04) 5-8.
• 26 Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser Permanente’s experience of implementing an electronic medical record: a qualitative study. Bmj 2005; 331 7528 1313-6.
• 27 Brender J, Ammenwerth E, Nykanen P, Talmon J. Factors influencing success and failure of health informatics systems—a pilot Delphi study. Methods Inf Med 2006; 45 (01) 125-36.
• 28 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-12.
• 29 Tenner E. Why things bite back: technology and the revenge of unintended consequences. NewYork, NY: Vintage Books; 1997
• 30 de Keizer N, Ammenwerth E. Trends in Evaluation Research 1982 - 2002: A Study on how the Quality of IT Evaluation Studies Develop. Stud Health Technol Inform 2005; 116: 581-6.
• 31 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I. et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. Jama 1996; 276 (08) 637-9.