Evaluating a Modular Decision Support Application For Colorectal Cancer Screening

Summary

Background: There is a need for health information technology evaluation that goes beyond randomized controlled trials to include consideration of usability, cognition, feedback from representative users, and impact on efficiency, data quality, and clinical workflow. This article presents an evaluation illustrating one approach to this need using the Decision-Centered Design framework. Objective: To evaluate, through a Decision-Centered Design framework, the ability of the Screening and Surveillance App to support primary care clinicians in tracking and managing colorectal cancer testing.

Methods: We leveraged two evaluation formats, online and in-person, to obtain feedback from a range primary care clinicians and obtain comparative data. Both the online and in-person evaluations used mock patient data to simulate challenging patient scenarios. Primary care clinicians responded to a series of colorectal cancer-related questions about each patient and made recommendations for screening. We collected data on performance, perceived workload, and usability. Key elements of Decision-Centered Design include evaluation in the context of realistic, challenging scenarios and measures designed to explore impact on cognitive performance.

Results: Comparison of means revealed increases in accuracy, efficiency, and usability and decreases in perceived mental effort and workload when using the Screening and Surveillance App. Conclusion: The results speak to the benefits of using the Decision-Centered Design approach in the analysis, design, and evaluation of Health Information Technology. Furthermore, the Screening and Surveillance App shows promise for filling decision support gaps in current electronic health records.

Keywords

Health information technology - decision support techniques - evaluation methodology - colorectal cancer - cancer screening

 

Conflict of interest

The Screening and Surveillance App evaluated in this project was designed and developed under the same contract that funded this evaluation. Many of the authors of this manuscript contributed to both the design and the evaluation of the health education materials. Laura Militello is co-owner of Applied Decision Science where the work was undertaken. Donald Haverkamp is a representative of the sponsoring agency that funded this work.

• References

• 1 Kaplan B. Evaluating informatics applications-clinical decision support systems literature review. Int J Med Inform 2001; 64: 15-37.
• 2 Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64: 39-56.
• 3 Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt. BMJ 2001; 326: 860-863.
• 4 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI-Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78: 1-9.
• 5 Hanauer D, Zheng K. Measuring the impact of health information technology. Applied clinical informatics 2012; 3 (Suppl. 03) 334-336.
• 6 Ammenwerth E, Brender J, Nykänen P, Prokosch HU, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 2004; 73: 479-491.
• 7 Lowry SZ, Quinn MT, Ramaiah M, Schumacher RM, Patterson ES, North R, Zhang J, Gibbons MC, Abbot P. Technical evaluation, testing, and validation of the usability of electronic health records. National Institute of Standards and Technology Interagency/Internal Report (NISTIR) 7804. 2012.
• 8 Lowry SZ, Ramaiah M, Patterson ES, Brick D, Gurses AP, Ozok A, Simmons D, Gibbons MC. Integrating Electronic Health Records into Clinical Workflow: An Application of Human Factors Modeling Methods to Ambulatory Care. National Institute of Standards and Technology Interagency/Internal Report (NISTIR) 7988. 2014.
• 9 Levin B, Lieberman DA, McFarland B, Smith RA, Brooks D, Andrews KS, Dash C, Giardiello FM, Glick S, Levin TR, Pickhardt P. Screening and surveillance for the early detection of colorectal cancer and adenomatous polyps, 2008: A joint guideline from the American Cancer Society, the US Multi-Society Task Force on Colorectal Cancer, and the American College of Radiology. CA Cancer J Clin 2008; 58: 130-160.
• 10 Frazier AL, Colditz GA, Fuchs CS, Kuntz KM. Cost-effectiveness of screening for colorectal cancer in the general population. JAMA 2000; 284 (Suppl. 15) 1954-1961.
• 11 Zauber AG, Winawer SJ, O’Brien MJ, Landsdorp-Vogelaar I, van Ballegooijen M, Hankey BF, Shi W, Bond JH, Schapiro M, Panish JF, Stewart ET. Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. N Engl J Med 2012; 366 (Suppl. 08) 687-696.
• 12 Subramanian S, Klosterman M, Amonkar MM, Hunt TL. Adherence with colorectal cancer screening guidelines: a review. Prev Med 2004; 38: 536-550.
• 13 Brawarsky P, Brooks DR, Mucci LA, Wood PA. Effect of physician recommendation and patient adherence on rates of colorectal cancer testing. Cancer Detect Prev 2004; 28: 260-268.
• 14 Janz NK, Wren PA, Schottenfeld D, Guire KE. Colorectal cancer screening attitudes and behavior: a population-based study. Prev Med 2003; 627-634.
• 15 Hudson SV, Ohman-Strickland P, Cunningham R, Ferrante JM, Hahn K, Crabtree BF. The effects of team-work and system support on colorectal cancer screening in primary care practices. Cancer Detect Prev 2007; 31: 417-423.
• 16 Sarfaty M, Wender R. How to increase colorectal cancer screening rates in practice. CA Cancer J Clin 2007; 57: 354-366.
• 17 Saleem JJ, Militello LG, Arbuckle N, Flanagan M, Haggstrom DA, Linder JA, Doebbeling BN. Provider perceptions of colorectal cancer screening decision support at three benchmark institutions. In: AMIA Annual Symposium Proceedings. 2009: 558-562.
• 18 Lowry SZ, Ramaiah M, Patterson ES, Prettyman SS, Simmons D, Brick D, Latkany P, Gibbons MC. Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically based use cases for validating safety – enhanced usability and guidelines for standardization. National Institute of Standards and Technology Interagency/Internal Report (NISTIR) 7804-1. 2015.
• 19 Militello LG, Saleem JJ, Borders MR, Sushereba CE, Haverkamp D, Wolf SP, Doebbeling BN. Designing colorectal cancer screening decision support: a cognitive engineering enterprise. Journal of Cognitive Engineering and Decision Making 2016; 10 (Suppl. 01) 74-90.
• 20 Kawamoto K. Integration of knowledge resources into applications to enable clinical decision support: architectural considerations. In: Greenes R. ed. Clinical Decision Support: The road ahead. Burlington, MA: Elsevier; 2007: 503-539.
• 21 Sim I, Gorman P, Greenes R, Haynes RB, Kaplan B, Lehnmann H, Tang PC. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (Suppl. 06) 527-534.
• 22 U.S. Preventive Services Task Force.. Screening for colorectal cancer: U.S. Preventive Services Task Force recommendation statement. Annals of Internal Medicine 2008; 1406-1422.
• 23 Militello LG, Dominguez CO, Lintern G, Klein G. The Role of Cognitive Systems Engineering in the Systems Design Process. Systems Engineering 2009; 13 (Suppl. 03) 261-273.
• 24 Kirwan B, Ainsworth LK. A Guide to Task Analysis. London: Tailor & Francis; 1992
• 25 Faulkner Laura. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers 2003; 35 (Suppl. 03) 379-383.
• 26 Vidulich MA, Tsang PS. Absolute magnitude estimation and relative judgment approaches to subjective workload assessments. In: Proceedings of the Human Factors Society 31st Annual Meeting. 1987: 1057-1061.
• 27 Vidulich MA. The use of judgment matrices in subjective workload assessment: The Subjective Workload Dominance (SWORD) technique. In: Proceedings of the Human Factors Society 33rd Annual Meeting. 1989: 1406-1410.
• 28 Zijlstra F, Van Doorn L. The construction of a subjective effort scale. Technical Report, Delft University of Technology 1985: 40.
• 29 Yen PY, Wantland D, Bakken S. Development of a customizable health IT usability evaluation scale. In: AMIA Annual Symposium Proceedings. 2010: 917.
• 30 Yen PY, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014; 21 (Suppl. 02) 241-248.
• 31 Saaty TL. How to make a decision: The analytic hierarchy process. Eur J Oper Res 1990; 48 (Suppl. 01) 9-26.
• 32 Vidulich MA, Ward GF, Schueren J. Using the Subjective Workload Dominance (SWORD) Technique for Projective Workload Assessment. Hum Factors 1991; 33 (Suppl. 06) 677-691.