A Concise and Practical Framework for the Development and Usability Evaluation of Patient Information Websites

Summary

Background: The Website Developmental Model for the Healthcare Consumer (WDMHC) is an extensive and successfully evaluated framework that incorporates user-centered design principles. However, due to its extensiveness its application is limited. In the current study we apply a subset of the WDMHC framework in a case study concerning the development and evaluation of a website aimed at childhood cancer survivors (CCS).

Objective: To assess whether the implementation of a limited subset of the WDMHC-framework is sufficient to deliver a high-quality website with few usability problems, aimed at a specific patient population.

Methods: The website was developed using a six-step approach divided into three phases derived from the WDMHC: 1) information needs analysis, mock-up creation and focus group discussion; 2) website prototype development; and 3) heuristic evaluation (HE) and think aloud analysis (TA). The HE was performed by three double experts (knowledgeable both in usability engineering and childhood cancer survivorship), who assessed the site using the Nielsen heuristics. Eight end-users were invited to complete three scenarios covering all functionality of the website by TA.

Results: The HE and TA were performed concurrently on the website prototype. The HE resulted in 29 unique usability issues; the end-users performing the TA encountered eleven unique problems. Four issues specifically revealed by HE concerned cosmetic design flaws, whereas two problems revealed by TA were related to website content.

Conclusion: Based on the subset of the WDMHC framework we were able to deliver a website that closely matched the expectancy of the end-users and resulted in relatively few usability problems during end-user testing. With the successful application of this subset of the WDMHC, we provide developers with a clear and easily applicable framework for the development of healthcare web-sites with high usability aimed at specific medical populations.

Citation: Peute LW; Knijnenburg SL; Kremer LC; Jaspers MWM. A concise and practical framework for the development and usability evaluation of patient information websites. Appl Clin Inf 2015; 6: 383–399

http://dx.doi.org/10.4338/ACI-11-RA-0109

Keywords

Patient education - childhood cancer - survivorship - website - human factors

 

Conflict of interest statement

The authors have declared no conflicts of interest.

^* These authors contributed equally to the paper

• References

• 1 Heymans HS, Caron HN. Childhood cancers in the Netherlands (1989–1997). Ned Tijdschr Geneeskd 2001; 145: 1442-1444. PM:11503310
• 2 Geenen MM, Cardous-Ubbink MC, Kremer LC, van den Bos C, van der Pal HJ, Heinen RC, Jaspers MW, Koning CC, Oldenburger F, Langeveld NE, Hart AA, Bakker PJ, Caron HN, van Leeuwen FE. Medical assessment of adverse health outcomes in long-term survivors of childhood cancer. JAMA 2007; 297: 2705-2715. PM:17595271
• 3 Jaspers MW, Van den Bos C, Heinen RC, Bakker PJ, Geenen MM, Kremer LC, Van Leeuwen F, Caron HN. Development of a national protocol to screen Dutch cancer survivors on late cancer treatment effects. Int J Med Inform 2007; 76: 297-305. PM:16533618
• 4 Jaspers MW, Caron H, Behrendt H, van den Bos C, Bakker P, Van Leeuwen F. The development of a new information model for a pediatric cancer registry on late treatment sequelae in The Netherlands. Stud Health Technol Inform 2000; 77: 895-899. PM:11187683
• 5 Kremer LCM, Jaspers MWM, van Leeuwen FE. et al. Landelijke richtlijnen voor follow-up van overlevenden van kinderkanker. Tijdschrift voor Kindergeneeskunde 2006; 74: 214-218.
• 6 Byrne J, Lewis S, Halamek L, Connelly RR, Mulvihill JJ. Childhood cancer survivors’ knowledge of their diagnosis and treatment. Ann Intern Med 1989; 110: 400-403. PM:2916808
• 7 Bashore L. Childhood and adolescent cancer survivors’ knowledge of their disease and effects of treatment. J Pediatr Oncol Nurs 2004; 21: 98-102. PM:15125553
• 8 Kadan-Lottick NS, Robison LL, Gurney JG, Neglia JP, Yasui Y, Hayashi R, Hudson M, Greenberg M, Mertens AC. Childhood cancer survivors’ knowledge about their past diagnosis and treatment: Childhood Cancer Survivor Study. JAMA 2002; 287: 1832-1839. PM:11939869
• 9 Lewis D. Computer-based approaches to patient education: a review of the literature. J Am Med Inform Assoc 1999; 6: 272-282. PM:10428001
• 10 Tuil WS, ten Hoopen AJ, Braat DD, de Vries Robbé PF, Kremer JA. Patient-centred care: using online personal medical records in IVF practice. Hum Reprod 2006; 21: 2955-2959. PM:16982658
• 11 Grant RW, Wald JS, Poon EG, Schnipper JL, Gandhi TK, Volk LA, Middleton B. Design and implementation of a web-based patient portal linked to an ambulatory care electronic health record: patient gateway for diabetes collaborative care. Diabetes Technol Ther 2006; 8: 576-586. PM:17037972
• 12 Johnson CM, Turley JP. A new approach to building web-based interfaces for healthcare consumers. Elec J Health Inform 2007; 2: e2. http://www.ejhi.net
• 13 Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a web-based health information database and call center. J Biomed Inform 2011; 44: 897-908. PM:21396486
• 14 Dix A, Finlay J, Abowd GD, Beale R. Cognitive models. In. Human-Computer Interaction. Harlow, United Kingdom: Pearson Education Limited; 2004. p 436-441. ISBN: 0130-461091.
• 15 Ruland CM, Jeneson A, Andersen T, Andersen R, Slaughter L, Bente-Schjødt-Osmo Moore SM. Designing tailored Internet support to assist cancer patients in illness management. AMIA Annu Symp Proc 2007; 635-639. PM:18693913
• 16 Grama LM, Beckwith M, Bittinger W, Blais D, Lollar C, Middleswarth A, Noone M, Price D, Quint-Kasner S, Shields V, Wright LW. The role of user input in shaping online information from the National Cancer Institute. J Med Internet Res 2005; 7: e25. PM:15998616
• 17 Sox CM, Gribbons WM, Loring BA, Mandl KD, Batista R, Porter SC. Patient-centered design of an information management module for a personally controlled health record. J Med Internet Res 2010; 12: e36. PM:20805091
• 18 Kushniruk AW, Patel VL, Cimino JJ. Evaluation of Web-based patient information resources: application in the assessment of a patient clinical information system. Proc AMIA Symp 2000; 443-447. PM:11079922
• 19 Jaspers MW. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform. 2008 PM:19046928
• 20 Nielsen J. Designing Web Usability. Indianapolis: New Riders Publishing; 1999. ISBN: <[26] ISSN/ISBN>.
• 21 Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res 2009; 11: e47. PM:19897458
• 22 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In. ACM INTERCHI’93 Conference on Human Factors in Computing Systems. New York: ACM Press; 1990. p 249-256.
• 23 Nielsen J. Heuristic Evaluation. In: John Wiley & Sons, editor. Usability Inspection Methods. New York: 1994. p 25-62
• 24 Nielsen J. Estimating the Number of Subjects Needed for A Thinking Aloud Test. International Journal of Human-Computer Studies 1994; 41: 385-397. ISI:A1994PR10100006
• 25 Brooke J. SUS: a ,,quick and dirty“ usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland A.. editors. Usability Evaluation in Industry.. London: Taylor and Francis 1996
• 26 Law ELC, Hvannberg ET. Complementarity and Convergence of Heuristic Evaluation and Usability Test: A Case Study of UNIVERSAL Brokerage Platform. NordiCHI 2002: 71-80.
• 27 Fu LM, Salvendy G, Turley L. Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology 2002; 21: 137-143. ISI:000177523400005
• 28 Beuscart-Zephir MC, Leroy N, Alao O, Darmoni S. Usability assessment study of a web site displaying medical resources on line: the CISMeF. Stud Health Technol Inform 2002; 90: 133-137. PM:15460675
• 29 Ostergren MJ, Karras BT. ActiveOptions: leveraging existing knowledge and usability testing to develop a physical activity program website for older adults. AMIA Annu Symp Proc 2007; 578-582. PM:18693902
• 30 Nahm ES, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004; 22: 326-334. PM:15602301
• 31 Lai TY. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform 2007; 76 (Suppl. 02) S317-S324. PM:17616431
• 32 Knijnenburg SL, Kremer LC, Van den Bos C, Braam KI, Jaspers MW. Health information needs of childhood cancer survivors and their family. Pediatr Blood Cancer 2010; 54: 123-127. PM:19743299
• 33 Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002; 9: 320-330. PM:12087113
• 34 Hess SL, Johannsdottir IM, Hamre H, Kiserud CE, Loge JH, Fosså SD. Adult survivors of childhood malignant lymphoma are not aware of their risk of late effects. Acta Oncol 2011; 50: 653-659. PM:21261507
• 35 Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: A mixed-methods approach. Int J Med Inform 2012; 81: 374-387. PM:22261086
• 36 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In. CHI ’93. New York, NY, USA: ACM; 1993. p 206-213.
• 37 Nielsen J. Usability Evaluation and Inspection Methods. In: Ashlund S, Mullet K, Henderson A. , et al., editors. Reading, MA: Addison-Wesley; 1993
• 38 Lathan CE, Sebrechts MM, Newman DJ, Doarn CR. Heuristic evaluation of a web-based interface for internet telemedicine. Telemed J 1999; 5: 177-185. PM:10908430
• 39 Knijnenburg SL, Kremer LC, Versluys AB, Braam KI, Mud MS, van der Pal HJ, Caron HN, Jaspers MW. Evaluation of a patient information website for childhood cancer survivors. Support Care Cancer. 2012 PM:23007883