Rapid Assessment of Clinical Information Systems in the Healthcare SettingAn Efficient Method for Time-pressed Evaluation
28 May 2010
accepted: 07 October 2010
18 January 2018 (online)
Objective: Recent legislation in the United States provides strong incentives for implementation of electronic health records (EHRs). The ensuing transformation in U. S. health care will increase demand for new methods to evaluate clinical informatics interventions. Timeline constraints and a rapidly changing environment will make traditional evaluation techniques burdensome. This paper describes an anthropological approach that provides a fast and flexible way to evaluate clinical information systems.
Methods: Adapting mixed-method evaluation approaches from anthropology, we describe a rapid assessment process (RAP) for assessing clinical informatics interventions in health care that we developed and used during seven site visits to diverse community hospitals and primary care settings in the U. S.
Setting: Our multidisciplinary team used RAP to evaluate factors that either encouraged people to use clinical decision support (CDS) systems or interfered with use of these systems in settings ranging from large urban hospitals to single-practitioner, private family practices in small towns.
Results: Critical elements of the method include: 1) developing a fieldwork guide; 2) carefully selecting observation sites and participants; 3) thoroughly preparing for site visits; 4) partnering with local collaborators; 5) collecting robust data by using multiple researchers and methods; and 6) analyzing and reporting data in a structured manner helpful to the oraganizations being evaluated.
Conclusions: RAP, iteratively developed over the course of visits to seven clinical sites across the U.S., has succeeded in allowing a multidisciplinary team of informatics researchers to plan, gather and analyze data, and report results in a maximally efficient manner.
KeywordsEthnography - qualitative methods - clinical informatics - healthcare evaluation mechanisms - computerized medical record systems
- 1 Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med 2009; 360 (15) 1477-1479.
- 2 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-231.
- 3 Ash J, Stavri P, Kuperman G. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc 2003; 10 (03) 229-234.
- 4 Kaplan B, Harris-Salamone KD. Health IT success and failure: recommendations from literature and an AMIA workshop. J Am Med Inform Assoc 2009; 16 (03) 291-299.
- 5 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346.
- 6 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 7494 765.
- 7 Garg AX, Adhikari NK, McDonald H, RosasArellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238.
- 8 Kaplan B. Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64 (01) 39-56.
- 9 Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316 7149 1959-1961.
- 10 McNall M, Foster-Fishman PG. Methods of Rapid Evaluation, Assessment and Appraisal. American Journal of Evaluation 2007; 28 (02) 151-168.
- 11 Patton MQ. Utilization-Focused Evaluation. 3rd ed. Thousand Oaks, CA: Sage; 1997
- 12 Guba EG LY. Fourth generation evaluation. Newbury Park, CA: Sage; 1989
- 13 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385.
- 14 Butler MO. Translating Evaluation Anthropology. NAPA Bulletin 2005; 24: 17-30.
- 15 Dewalt KM, Dewalt BR, Wyland CB. Participant observation – 1998. In: Bernard HR. editor. Handbook of Methods in Cultural Anthropology. Walnut Creek, CA: Altamira Press; 1998. pp 259-300.
- 16 Creswell JW. Qualitative Inquiry & Research Design: Choosing among Five Approaches. Thousand Oaks, CA: Sage Publications; 2007
- 17 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
- 18 Sittig DF, Ash JS. Clinical information systems: Overcoming adverse consequences. Sudbury, MA; 2010
- 19 Ash J, Sittig D, McMullen C, Guappone K, Dykstra R, Carpenter J. A rapid assessment process for clinical informatics interventions. AMIA Annu Symp Proc. 2008 pp 26-30.
- 20 Beebe J. Rapid Assessment Process: An Introduction. Walnut Creek, CA: AltaMira; 2001
- 21 Handwerker WP. Quick Ethnography. Walnut Creek, CA: AltaMira Press; 2001
- 22 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI – Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9.
- 23 Scrimshaw SCM HE. Rapid Assessment Procedures for Nutrition and Primary Health Care: Anthropological Approaches to Improving Programme Effectiveness. Los Angeles, CA: 1987
- 24 Needle RH, Tsukamoto T, Goosby E, Bates C, von Zinkmangel D. Crisis Response Teams and Communities Combet HIV/AIDS in Racial and Ethnic Minority Populations: A guide for conducting community-based Rapid Assessment, Rapid Response and Evaluating. Washington DC: 2000
- 25 Manderson L, Aaby P. Rapid Anthropological Procedures (RAP) and their applicability to tropical diseases. Health Policy Planning 1992; 7: 46-55.
- 26 Trotter RT, Goosby E, Needle RH, Bates C, Singer M. A methodological model for rapid assessment, response, and evaluation: The RARE Program in public health. Field Methods 2001; 13 (02) 137-159.
- 27 Trotter RT, Needle RH. RARE Field Team Principal Investigator Guide. Washington DC: 2000
- 28 Butler MO, Linstone HA. Decision making for technology executives: Using multiple perspectives to improve performance. Boston: Artech House; 1999
- 29 Manderson L, Aaby P. An epidemic in the field? Rapid assessment procedures and health assessment. Social Science and Medicine 1992; 35 (07) 839-950.
- 30 O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ 2010; 341: c4587.
- 31 Greene J, McClintock CM. Triangulation in evaluation: design and analysis issues. Evaluation Review 2010; 9 (05) 523-545.
- 32 Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530.
- 33 Utarini A, Winkirst A, Pelto GH. Appraising Studies in Health Using Rapid Assessment Procedures (RAP): Eleven Critical Criteria. Human Organization 2001; 60 (04) 390-400.
- 34 Thomas MD, Blacksmith J, Reno J. Utilizing insider-outsider research teams in qualitative research. Qual Health Res 2000; 10 (06) 819-828.
- 35 Fernald DH, Duclos CW. Enhance your team-based qualitative research. Ann Fam Med 2005; 3 (04) 360-364.
- 36 Patton MQ. Qualitative Evaluation and Research Methods. 3rd ed. Thousand Oaks, CA: Sage; 2002
- 37 Bernard HR, Ryan GW. Text analysis: Qualitative and Quantitative methods. Handbook of Methods in Cultural Anthropology. Walnut Creek, CA: AltaMira Press; 1998
- 38 Miles MB, Huberman AM. Qualitative Data Analysis. 2nd ed. Thousand Oaks, CA: Sage; 1994
- 39 Van den Hoonaard WC. Analytical Field Research: Working with Sensitizing Concepts. Thousand Oaks, CA: Sage; 1997
- 40 Bernard HR, Ryan GW. Analyzing Qualitative Data: Systematic Approaches. Thousand Oaks, CA: Sage Publications; 2010
- 41 Spradley JP. The Ethnographic Interview. Fort Worth, TX: Harcourt Brace Jovanovich; 1979
- 42 Khajouei R, Jaspers MW. The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010; 49 (01) 3-19.
- 43 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
- 44 Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. Int J Med Inform 2001; 64 (01) 15-37.
- 45 Patton MC. The View from Evaluation. NAPA Bulletin 2005; 24: 31-40.
- 46 Ash JS, Sittig DF, Dykstra R, Wright A, McMullen C, Richardson J. et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform 2010; 160: 806-810.
- 47 Schensul JJ, Schensul SL. Ethnographic evaluation of AIDS prevention programs: better date for better programs. New Directions for Program Evaluation 1990; 46: 51-62.
- 48 Checkland P, Holwell S. Action Research: Its Nature and Validity In Systemic Practice and Action Research. Action Research 1998; 11 (01) 9-21.
- 49 Reason P, Bradbury H. Handbook of Action Research. London UK: Sage Publications; 2001
- 50 Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health 1998; 19: 173-202.