Download PDF
CC BY-NC-ND 4.0 · Yearb Med Inform 2022; 31(01): 033-039
DOI: 10.1055/s-0042-1742498
Special Section: Inclusive Digital Health
Working Group Contributions

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals

Kathrin Cresswell
1   The University of Edinburgh, Usher Institute, Edinburgh, United Kingdom
,
Michael Rigby
2   Keele University, School of Social, Political and Global Studies and School of Primary, Community and Social Care, Keele, United Kingdom
,
Andrew Georgiou
3   Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
,
Zoie Shui-Yee Wong
4   St. Luke's International University, Tokyo, Japan
,
Polina Kukhareva
5   Department of Biomedical Informatics, University of Utah, USA
,
Stephanie Medlock
6   Department of Medical Informatics, Amsterdam Public Health Research Institute, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
,
Nicolet F. De Keizer
6   Department of Medical Informatics, Amsterdam Public Health Research Institute, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
,
Farah Magrabi
3   Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
,
Philip Scott
7   University of Wales Trinity St David, Swansea, Wales, United Kingdom
,
Elske Ammenwerth
8   UMIT TIROL, Private University for Health Sciences, Medical Informatics and Technology, Institute of Medical Informatics, Hall in Tirol, Austria
› Author Affiliations
› Further Information
   Permissions and Reprints

Summary

Objectives: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them.

Methods: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings.

Results: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time.

Conclusions: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Keywords

Patient portal, health equity - digital divide - formative evaluation - digital health


Publication History

Article published online:
02 June 2022

© 2022. IMIA and Thieme. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Jha AK, Doolan D, Grandt D, Scott T, Bates DW. The use of health information technology in seven nations. Int J Med Inform 2008 Dec 1;77(12):848-54.
    PubMed
  • 2 Bhambere HS, Abhishek B, Sumit H. Rapid digitization of healthcare: a review of COVID-19 impact on our health systems. Int J All Res Educ Sci Methods 2021;9:1457-9.
    PubMed
  • 3 Glöggler M, Ammenwerth E. Improvement and Evaluation of the TOPCOP Taxonomy of Patient Portals: Taxonomy-Evaluation-Delphi (TED) Approach. J Med Internet Res 31/07/2021:30701. In press.
    PubMed
  • 4 Hahn EE, Baecker A, Shen E, Haupt EC, Wakach W, Ahuja A, et al. A Patient Portal-Based Commitment Device to Improve Adherence with Screening for Colorectal Cancer: a Retrospective Observational Study. J Gen Intern Med 2021 Apr;36(4):952-60.
    PubMed
  • 5 Miller HN, Gleason KT, Juraschek SP, Plante TB, Lewis-Land C, Woods B, et al. Electronic Medical Record-based Cohort Selection and Direct-to-Patient, Targeted Recruitment: Early Efficacy and Lessons Learned. J Am Med Inform Assoc 2019 Nov 1;26(11):1209-17.
    PubMed
  • 6 Varad, NH, d'Amonville S, Chen AF. Electronic Patient Portal Use in Orthopaedic Surgery Is Associated with Disparities, Improved Satisfaction, and Lower No-Show Rates. J Bone Joint Surg Am 2020 Aug 5;102(15):1336-13.
    PubMed
  • 7 Turner AM, Osterhage K, Hartzler A, Joe J, Lin L, Kanagat N, et al. Use of Patient Portals for Personal Health Information Management: the Older Adult Perspective. In: AMIA Annu Symp Proc 2015 Nov 5;2015:1234-41.
    PubMed
  • 8 Braveman P, Gruskin S. Defining Equity in Health. J Epidemiol Community Health 2003 Apr 1;57(4):254-8.
    PubMed
  • 9 Berkman ND, Davis TC, Mc Cormack L. Health Literacy: What is it? J Health Commun 2010 Aug 31;15(S2):9-19.
    PubMed
  • 10 Buckingham D. Defining Digital Literacy. Nordic Journal of Digital Literacy 2016:21-34.
    PubMed
  • 11 Blair IV, Steiner JF, Havranek EP. Unconscious (implicit) Bias and Health Disparities: Where do we go from here? Permanente J 2011;15(2):71.
    PubMed
  • 12 Kruk ME, Myers M, Varpilah ST, Dahn BT. What is a Resilient Health System? Lessons from Ebola. Lancet 2015 May 9;385(9980):1910-2.
    PubMed
  • 13 Saeed SA, Masters RM. Disparities in Health Care and the Digital Divide. Curr Psychiatry Rep 2021 Sep;23(9):1-6.
    PubMed
  • 14 Ammenwerth E, Neyer S, Hörbst A, Mueller G, Siebert U, Schnell-Inderst P. Adult Patient access to Electronic Health Records. Cochrane Database Syst Revi 2021 Feb 26;2(2):CD012707.
    PubMed
  • 15 Wyatt J, Sathanandam S, Rastall P. Personal Health Record (PHR) Landscape Review final report. RCP; 2016. Available from: https://www.rcplondon.ac.uk/projects/outputs/personal-health-record-phr-landscape-review
    PubMed
  • 16 Symons JD, Ashrafian H, Dunscombe R, Darzi A. From EHR to PHR: let's get the record straight. BMJ Open 2019 Sep 1;9(9):e029582.
    PubMed
  • 17 Electronic Health Records. [cited 2021 October 4] Available from: https://post.parliament.uk/research-briefings/post-pn-0519/
    PubMed
  • 18 Halamka JD, Tripathi M. The HITECH Era in Retrospect. N Engl J Med 2017 Sep 7;377(10):907-9.
    PubMed
  • 19 Irizarry T, DeVito AD, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148.
    PubMed
  • 20 Powell KR. Patient-Perceived Facilitators of and Barriers to Electronic Portal Use: a Systematic Review. Comput Inform Nurs 2017 Nov;35(11):565-73.
    PubMed
  • 21 Varsi C, Gammon D, Wibe T, Ruland CM. Patients' Reported Reasons for Non-Use of an Internet-Based Patient-Provider Communication Service: Qualitative Interview Study. J Med Internet Res 2013;15(11):e246.
    PubMed
  • 22 Tieu L, Schillinger D, Sarkar U, Hoskote M, Hahn KJ, Ratanawongsa N, et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere? J Am Med Inform Assoc 2017; 24(e1):e47”54.
    PubMed
  • 23 Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The digital divide and patient portals. Med Care. 2016 Aug 1;54(8):772-9.
    PubMed
  • 24 Anthony D, Campos-Castillo C. Why Most of Your Patients Aren't Using an Online Portal, and What You Can Do About It. R I Med J (2013) 2020 Feb 3;103(1):32-4.
    PubMed
  • 25 Chen X, Schofield E, Hay JL, Waters EA, Kiviniemi MT, Orom H. Race/Ethnicity, Nativity Status, and Patient Portal Access and Use. J Health Care Poor Underserved 2021;32(2):700-11.
    PubMed
  • 26 Connolly SL, Etingen B, Shimada SL, Hogan TP, Nazi K, Stroupe K, et al. Patient portal use among veterans with depression: Associations with symptom severity and demographic characteristics. J Affect Disord 2020 Oct 1;275:255-9.
    PubMed
  • 27 Ukoha EP, Feinglass J, Yee LM. Disparities in Electronic Patient Portal Use in Prenatal Care: Retrospective Cohort Study. J Med Internet Res 2019 Sep 23;21(9):e14445.
    PubMed
  • 28 Marmot M, Bell R. The sustainable development goals and health equity. Epidemiology 2018 Jan 1;29(1):5-7.
    PubMed
  • 29 Morton S, Pencheon D, Squires N. Sustainable Development Goals (SDGs), and their implementation. A national global framework for health, development and equity needs a systems approach at every level. Br Med Bull 2017 Dec 1;124(1):81-90.
    PubMed
  • 30 Showell C, Turner P. The PLU problem: are we designing personal ehealth for people like us? Stud Health Technol Inform 2013;183:276-80.
    PubMed
  • 31 Rigby M, Georgiou A, Hyppönen H, Ammenwerth E, de Keizer N, Magrabi F, et al. Patient Portals as a Means of Information and Communication Technology Support to Patient-centric Care Coordination”the Missing Evidence and the Challenges of Evaluation. Yearb Med Inform 2015:148-59.
    PubMed
  • 32 Ammenwerth E, Rigby M, editors. Evidence-based Health Informatics - Promoting Safety and Efficiency Through Scientific Methods and Ethical Policy. Stud Health Tech Inform (volume 222). IOS Press; 2016.
    PubMed
  • 33 Wildenbos GA, Peute L, Jaspers M. Facilitators and Barriers of Electronic Health Record Patient Portal Adoption by Older Adults: a Literature Study. Stud Health Technol Inform 2017 Jan 1;235:308-12.
    PubMed
  • 34 Goedhart NS, Zuiderent-Jerak T, Woudstra J, Broerse JE, Betten AW, Dedding C. Persistent Inequitable Design and Implementation of Patient Portals for Users at the Margins. J Am Med Inform Assoc 2021 Feb;28(2):276-83.
    PubMed
  • 35 Showell C, Cummings EA, Turner P. The Invisibility of Disadvantage: Why do we not notice? Stud Health Tech Inform 2017;235:388-92.
    PubMed
  • 36 Grossman LV, Masterson Creber RM, Benda NC, Wright D, Vawdrey DK, Ancker JS. Interventions to Increase Patient Portal Use in Vulnerable Populations: a Systematic Review. J Am Med Inform Assoc 2019 Aug;26(8-9):855-70.
    PubMed
  • 37 Yasukawa K, Sawada T, Hashimoto H, Jimba M. Health-care Disparities for Foreign Residents in Japan. Lancet 2019 Mar 2;393(10174):873-4.
    PubMed
  • 38 Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and Facilitators to Online Portal Use among Patients and Caregivers in a Safety Net Health Care System: a Qualitative Study. J Med Internet Res 2015 Dec 3;17(12):e4847.
    PubMed
  • 39 Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why ? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018 Dec 1;37(12):1948-54.
    PubMed
  • 40 Mackert M, Mabry-Flynn A, Champlin S, Donovan EE, Pounders K. Health Literacy and Health Information Technology Adoption: The Potential for a New Digital Divide. J Med Internet Res 2016;18(10):e264.
    PubMed
  • 41 Coughlin SS, Stewart JL, Young L, Heboyan V, De Leo G. Health Literacy and Patient Web Portals. Int J Med Inform 2018 May 1;113:43-8.
    PubMed
  • 42 Lam BD, Bourgeois F, Dong ZJ, Bell S.. Speaking up about patient-perceived serious visit note errors: Patient and family experiences and recommendations. J Am Med Inform Assoc 2021 Mar 18;28(4):685-94
    PubMed
  • 43 Lyles CR, Fruchterman J, Youdelman M, Schillinger D. Legal, Practical, and Ethical Considerations for Making Online Patient Portals Accessible for All. Am J Public Health 2017 Oct;107(10):1608-11.
    PubMed
  • 44 Fraccaro P, Vigo M, Balatsoukas P, Buchan IE, Peek N, van der Veer SN. The Influence of Patient Portals on Users' Decision Making is Insufficiently Investigated: a Systematic Methodological Review. Int J Med Inform 2018 Mar 1;111:100-11.
    PubMed
  • 45 van Mens HJ, Duijm RD, Nienhuis R, de Keizer NF, Cornet R. Determinants and outcomes of patient access to medical records: systematic review of systematic reviews. Int J Med Inform 2019 Sep 1;129:226-33.
    PubMed
  • 46 van Mens HJ, Duijm RD, Nienhuis R, de Keizer NF, Cornet R. Towards an adoption framework for patient access to electronic health records: systematic literature mapping study. JMIR Med Inform 2020;8(3):e15150.
    PubMed
  • 47 Bao C, Bardhan IR, Singh H, Meyer BA, Kirksey K. Patient”provider engagement and its impact on health outcomes: a longitudinal study of patient portal use. MIS Quarterly 2020 Jun 1;44(2).
    PubMed
  • 48 Pollock N, Williams R, D'Adderio L. Global software and its provenance: generification work in the production of organizational software packages. Social Studies of Science 2007 Apr;37(2):254-80.
    PubMed
  • 49 Schloemer T. Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis. Implement Sci 2018;13(1):88.
    PubMed
  • 50 Coiera E, Ammenwerth E, Georgiou A, Magrabi F. Does health informatics have a replication crisis? J Am Med Inform Assoc 2018 Aug;25(8):963-8.
    PubMed
  • 51 Were MC, Sinha C, Catalani C. A systematic approach to equity assessment for digital health interventions: case example of mobile personal health records. J Am Med Inform Assoc 2019 Aug;26(8-9):884-90.
    PubMed
  • 52 Elwy AR, Wasan AD, Gillman AG, Johnston KL, Dodds N, Mc Farland C, et al. Using formative evaluation methods to improve clinical implementation efforts: description and an example. Psychiatry research. 2020 Jan 1;283:112532.
    PubMed
  • 53 Omachonu VK, Einspruch NG. Innovation in healthcare delivery systems: a conceptual framework. The Innovation Journal: The Public Sector Innovation Journal. 2010 Mar 1;15(1):1-20.
    PubMed
  • 54 Yilmaz NG, Sungur H, van Weert JC, van den Muijsenbergh ME, Schouten BC. Enhancing patient participation of older migrant cancer patients: needs, barriers, and eHealth. Ethn Health 2020 Dec 8:1-24.
    PubMed
  • 55 Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, Hinder Set al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017;19(11):e367.
    PubMed
  • 56 Otte-Trojel T, de Bont A, van de Klundert J, Rundall TG. Characteristics of patient portals developed in the context of health information exchanges: early policy effects of incentives in the meaningful use program in the United States. J Med Internet Res 2014;16(11):e258.
    PubMed
  • 57 Antonio MG, Petrovskaya O, Lau F. The State of Evidence in Patient Portals: Umbrella Review. J Med Internet Res 2020;22(11):e23851.
    PubMed
  • 58 Veinot TC, Mitchell H, Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018 Aug;25(8):1080-8.
    PubMed
  • 59 Casillas A, Abhat A, Mahajan A, Moreno G, Brown AF, Simmons S, et al. Portals of change: how patient portals will ultimately work for safety net populations. J Med Internet Res 2020;22(10):e16835.
    PubMed
  • 60 Personas. [cited 2021 October 4] Available from: https://www.usability.gov/how-to-and-tools/methods/personas.html
    PubMed
  • 61 Wachter RM. Expected and unanticipated consequences of the quality and information technology revolutions. JAMA 2006 Jun 21;295(23):2780-3.
    PubMed
  • 62 Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, Dabbs AD. Patient portals as a tool for health care engagement: a mixed-method study of older adults with varying levels of health literacy and prior patient portal use. J Med Internet Res 2017 Mar 30;19(3):e7099.
    PubMed
  • 63 Miller Jr DP, Latulipe C, Melius KA, Quandt SA, Arcury TA. Primary care providers' views of patient portals: interview study of perceived benefits and consequences. J Med Internet Res 2016 Jan 15;18(1):e4953.
    PubMed
  • 64 Kim Y, Ayvaci M, Raghunathan S, Tanriover B. Repairing the Digital Divide Can Increase the Service Divide: The Effects of Patient Portals on Kidney Allocation.
    PubMed
  • 65 Cresswell K, Sheikh A, Franklin BD, Krasuska M, Nguyen H, Hinder S, et al. Formative independent evaluation of a digital change programme in the English National Health Service: study protocol for a longitudinal qualitative study. BMJ Open 2020 Oct 1;10(10):e041275.
    PubMed
  • 66 Cresswell K, Williams R, Sheikh A. Developing and applying a formative evaluation framework for health information technology implementations: qualitative investigation. J Med Internet Res 2020 Jun 10;22(6):e15068.
    PubMed
  • 67 Braithwaite J, Runciman WB, Merry AF. Towards safer, better healthcare: harnessing the natural properties of complex sociotechnical systems. BMJ Qual Saf 2009 Feb 1;18(1):37-41.
    PubMed
  • 68 Lau F, Price M, Keshavjee K. Making sense of health information system success in Canada. Healthc Q. 2011;14(1):39-46.
    PubMed
  • 69 Ancker JS, Nosal S, Hauser D, Way C, Calman N. Access policy and the digital divide in patient access to medical records. Health Policy and Technology 2017 Mar 1;6(1):3-11.
    PubMed
  • 70 Antonio MG, Petrovskaya O, Lau F. Is research on patient portals attuned to health equity? A scoping review. J Am Med Inform Assoc 2019 Aug;26(8-9):871-83.
    PubMed
  • 71 Aanestad M, Grisot M, Hanseth O, Vassilakopoulou P. Information infrastructures within European health care: Working with the installed base. [cited 2021 October 14]. Available from: https://library.oapen.org/bitstream/handle/20.500.12657/27913/1002086.pdf?sequence=1
    PubMed
  • 72 Marler W. Mobile phones and inequality: Findings, trends, and future directions. New Media & Society 2018 Sep;20(9):3498-520.
    PubMed
  • 73 Digital technology and health inequalities: a scoping review. [cited 2021 October 14]. Available from: https://phw.nhs.wales/publications/publications1/digital-technology-and-health-inequalities-a-scoping-review/
    PubMed
  • 74 Scott P, de Keizer N Georgiou A., editors. Applied Interdisciplinary Theory in Health Informatics: A Knowledge Base for Practitioners. IOS Press; 2019.
    PubMed
  • 75 Wong ZS, Nøhr, C, Kuziemsky CE, Leung E, Chen F. Context Sensitive Health Informatics: Delivering 21st Century Healthcare - Building a Quality-and-Efficiency Driven System. Stud Health Technol Inform 2017;241:1-5.
    PubMed
  • 76 NHS data: Can web creator Sir Tim Berners-Lee fix it? [cited 2021 October 14]. Available from: https://www.bbc.co.uk/news/technology-54871705
    PubMed
  • 77 Oliveira P, Cunha MP. Centralized decentralization, or distributed leadership as paradox: The case of the Patient Innovation's COVID-19 Portal. Journal of Change Management 2021 Apr 28:1-9.
    PubMed