Measurement Error in Performance Studies of Health Information Technology: Lessons from the Management Literature
16 February 2012
accepted: 20 May 2012
16 December 2017 (online)
Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature’s more conceptual examination of health IT’s limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mis-measured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.
- 1 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385.
- 2 Jha AK, DesRoches CM, Kralovec PD, Joshi MS. A progress report on electronic health records in US hospitals. Health Aff 2010; 29 (010) 1951-1957.
- 3 Chaudhry B. et al. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (010) 742-752.
- 4 Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. Effect of electronic health records in ambulatory care: Retrospective, serial, cross sectional study. Br Med J 2005; 330: (581).
- 5 DesRoches CM. et al. Electronic health records’ limited successes suggest more targeted uses. Health Aff 2010; 29 (04) 639-646.
- 6 Parente ST, McCullough JS. Health information technology and patient safety: Evidence from panel data. Health Aff 2009; 28 (02) 357-360.
- 7 Himmelstein DU, Wright A, Woolhandler S. Hospital computing and the costs and quality of care: A national study. Am J Med 2010; 123 (01) 40-46.
- 8 Koppel R. et al. Role of computerized physician order entry systems in facilitating medication errors. J Am Med Assoc 2005; 293 (010) 1197-1203.
- 9 Avgar AC, Litwin AS, Pronovost PJ. Using management research to conceptualize the drivers and barriers to health IT adoption. Working paper. 2012
- 10 Ancker JS, Kern LM, Abramson E, Kaushal R. The triangle model for evaluating the effect of health information technology on healthcare quality and safety. J Am Med Inform Assn 2012; 19 (01) 61-65.
- 11 Ammenwerth E. et al. Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in innsbruck. Int J Med Inf 2004; 73 (06) 479-491.
- 12 Nykänen P. et al. Introducing guidelines for good evaluation practice in health informatics. In: Adlassnig K, Blobel B, Mantas J, Masic I. editors. Medical Informatics in a United and Healthy Europe. Amsterdam; Washington, DC: IOS; 2009: 958-962.
- 13 Westbrook JI. et al. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assn 2007; 14 (06) 746-755.
- 14 Kochan TA. On the human side of technology. ICL Tech J 1988; 6 (02) 391-400.
- 15 MacDuffie JP, Krafcik JF. Integrating technology and human resources for high-performance manufacturing: Evidence from the international auto industry. In: Kochan TA, Useem M. editors. Transforming Organizations. New York: Oxford; 1992: 209-226.
- 16 MacDuffie JP. Human-resource bundles and manufacturing performance: Organizational logic and flexible production systems in the world auto industry. Ind Labor Relat Rev 1995; 48 (02) 197-221.
- 17 Kelley MR. Participative bureaucracy and productivity in the machined products sector. Ind Relat 1996; 35 (03) 374-399.
- 18 Batt RL. Work organization, technology, and performance in customer service and sales. Ind Labor Relat Rev 1999; 52 (04) 539-564.
- 19 Litwin AS. Technological change at work: The impact of employee involvement on the effectiveness of health information technology. Ind Labor Relat Rev 2011; 64 (05) 863-888.
- 20 Brynjolfsson E, Hitt LM. Beyond computation: Information technology, organizational transformation and business performance. J Econ Perspect 2000; 14 (04) 23-48.
- 21 Solow R. We’d better watch out. New York Times Book Review 1987; July (012) (36).
- 22 Brynjolfsson E, Hitt LM. Computing productivity: Firm-level evidence. Rev Econ Stat 2003; 85 (04) 793-808.
- 23 Cappelli P, Neumark D. Do ‘High-performance’ work practices improve establishment-level outcomes?. Ind Labor Relat Rev 2001; 54 (04) 737-775.
- 24 Freeman RB, Kleiner MM. Who benefits most from employee involvement: Firms or workers?. Am Econ Rev 2000; 90 (02) 219-223.
- 25 Yoo KH. et al. The impact of electronic medical records on timeliness of diagnosis of asthma. J Asthma 2007; 44 (09) 753-758.
- 26 Ammenwerth E. et al. The effect of electronic prescribing on medication errors and adverse drug events: A systematic review. J Am Med Inform Assoc 2008; 15 (05) 585-600.
- 27 Litwin AS. Not featherbedding but feathering the nest: Human resource management and investments in information technology. Ind Relat. forthcoming.
- 28 Ichniowski C. et al. What works at work: Overview and assessment. Ind Relat 1996; 35 (03) 299-333.
- 29 Sittig DF. et al. Lessons from “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics 2006; 118 (02) 797-801.
- 30 Ash JS. et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-423.
- 31 Talmon J. et al. STARE-HI –statement on reporting of evaluation studies in health informatics. Int J Med Inf 2009; 78 (01) 1-9.
- 32 Brynjolfsson E, Hitt LM, Yang S. Intangible assets: Computers and organizational capital. Brookings Pap Eco Ac 2002; 1: 137-181.
- 33 Poon EG. et al. Relationship between use of electronic health record features and health care quality: Results of a statewide survey. Med Care 2010; 48 (03) 203-209.
- 34 Yu FB. et al. Full implementation of computerized physician order entry and medication-related quality outcomes: A study of 3364 hospitals. Am J Med Qual 2009; 24 (04) 278-286.
- 35 McCullough JS, Casey M, Moscovice I, Prasad S. The effect of health information technology on quality in US hospitals. Health Aff 2010; 29 (04) 647-654.
- 36 Devaraj S, Kohli R. Performance impacts of information technology: Is actual usage the missing link. Manage Sci 2003; 49 (03) 273-289.
- 37 Keyhani S. et al. Electronic health record components and the quality of care. Med Care 2008; 46 (012) 1267-1272.
- 38 Freeman RB. Longitudinal analyses of the effects of trade unions. J Labor Econ 1984; 2 (01) 1-26.
- 39 Robinson JC. et al. Financial incentives, quality improvement programs, and the adoption of clinical information technology. Med Care 2009; 47 (04) 411-417.
- 40 Hitt LM, Brynjolfsson E. Information technology and internal firm organization: An exploratory analysis. J Manage Inform Syst 1997; 14 (02) 81-101.
- 41 Han YY. et al. Unexpected increased mortality after implementation of a commercially sold physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
- 42 Eaton AE. The survival of employee participation programs in unionized settings. Ind Labor Relat Rev 1994; 47 (03) 371-389.
- 43 Amarasingham R. et al. Clinical information technologies and inpatient outcomes: A multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
- 44 Skrondal A, Rabe-Hesketh S. Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton: Chapman and Hall/CRC; 2004
- 45 Huber GP, Power DJ. Retrospective reports of strategic-level managers: Guidelines for increasing their accuracy. Strateg Manage J 1985; 6 (02) 171-180.
- 46 Gerhart B, Wright PM, MacMahan GC, Snell SA. Measurement error in research on human resources and firm performance: How much error is there and how does it influence effect size estimates?. Pers Psychol 2000; 53 (04) 803-834.
- 47 Kaushal R. et al. Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med 2010; 25 (06) 530-536.
- 48 Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Aff 2011; 30 (03) 464-471.