The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals

Summary

Objectives: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them.

Methods: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings.

Results: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time.

Conclusions: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Keywords

Patient portal, health equity - digital divide - formative evaluation - digital health

1 Introduction

International strategies increasingly focus on deploying health information technology (HIT) aiming to improve the quality, safety and efficiency of care [[1]]. This trend has been reinforced by the recent COVID-19 pandemic, which has accelerated uptake of digital tools across care settings and promoted acceptability among patients [[2]]. Remote access and monitoring tools designed to facilitate patient involvement in and access to care in particular are increasingly popular amongst policy makers and service providers. These include, amongst others, telehealth applications and patient portals. Patient portals are applications that allow patients to access information in their electronic health records (such as test results) and/or to request services (such as an appointment) (Box 1) [[3]]. They may be mandatory for all users or optional, the latter giving clinicians and patients the opportunity to assess a patient's suitability for portal access before use. Anticipated benefits include improved quality and safety, improved patient experience, increased empowerment and engagement of patients, and reduced costs [[4] [5] [6]].

However, despite these significant potential benefits, existing empirical work fails to show consistent evidence of the effectiveness of patient portals, in particular in high quality randomised controlled trials (RCTs). For example, a recent review found little or no effect on patient empowerment and health outcomes[[14]]. In addition, there is a lack of patient and clinician pull to drive the adoption of patient portals [[15] [16] [17] [18]], with reasons for non-use including a lack of perceived benefits [[19] [20]], issues with access (e.g. Wi-Fi, log-ins) [[21] [22] [23]], concerns relating to privacy and security (including cyber risks) [[24]], and issues with usability[[19]].

Box 1 Definition of key terms.

There are also challenges surrounding health equity, health literacy and bias (Box 1) associated with the deployment of patient portals. Frequently, citizens who could benefit most from using them fail to adopt patient portals [[6], [25] [26] [27]]. There are further significant variations across countries in relation to health equity, with some promoting system-level interventions to address the issue [[28], [29]]. Indeed, existing systems may inadvertently reinforce health inequity [[30]]. It has been suggested that better approaches to evaluation may help to promote equitable use and facilitate the realisation of potential benefits across user groups [[31]].

We here aim to discuss why existing evaluation approaches surrounding patient portals fail to address issues surrounding equity, literacy, and bias; the challenges existing evaluations of patient portals face in incorporating these issues; and how approaches may be reconceptualised going forward to address identified gaps. In doing so, we provide an explorative literature review building on recent developments in evidence-based health informatics [[32]].

2 How Evaluations of Patient Portals Address Issues Surrounding Equity, Literacy, and Bias

The following paragraphs will discuss why and how existing evaluations of patient portals fail to pay sufficient attention to issues surrounding equity, literacy, and bias. Firstly, existing work tends to study populations that are already using systems and are generally considered the “worried well” [[33]], thereby excluding those that are not using or cannot use patient portals [[34]]. This results in a selection bias excluding different or disadvantaged participants, amongst which portal usage is generally low [[35]]. For example, a recent review studying interventions to reach vulnerable populations found that most existing studies exclude non-English speakers (in countries where English is the principal language) [[36]]. Barriers to access to healthcare based on language barriers have also been found in other settings [[37]]. Vulnerable populations include older adults, those with low health and computer literacy, people from low-income, high-deprivation and rural areas, people with chronic illness and disabilities, and ethnic minorities [[36], [38], [39]].

Secondly, control over health information and improved communication with healthcare providers are key perceived benefits of patient portals [[20]], but realisation of these benefits requires two key modern skill sets: digital literacy and health literacy (Box 1). Low levels of these skills are generally associated with non-adoption of HIT, including patient portals [[40], [41]]. It has also been found that ethnic minority patients are less likely to correct errors in their records, so even if patients adopt a portal, they are still less likely to benefit [[42]]. There is some evidence that technical training and help, as well as improved portal usability reducing written text, can facilitate use in vulnerable populations but empirical work in this area is limited [[36], [43]].

However, as existing patient portals are designed for literate and motivated users, there is generally a lack of meaningful involvement of disadvantaged users in system design [[35]]. This may partly be due to system developers not prioritising equity considerations early in the development process. Their primary concern tends to be making the system work in a context known to them before tailoring it to other contexts of use and various user needs [[34], [35]].

Existing evaluations further pay insufficient attention to contextual factors and socio-organisational processes surrounding the implementation and adoption of patient portals. They are often superficial, focusing on quantifiable outputs such as rates of use (e.g., counting the number of log-ins). Such approaches are unlikely to identify issues with health disparities and ways to address them [[44]]. Where work explores social contexts, these tend to focus on individual behavioural dimensions of use, usability of the technology, and organisational components surrounding adoption (e.g., perceived facilitators and barriers). Other factors, such as existing information infrastructures, regulations, political strategies and wider societal contexts are often neglected [[45], [46]]. In addition, too often those with particular circumstances are considered “hard to reach” or “disadvantaged” - terms labelling a user group negatively as an outlier. An equity-led approach would specifically target these particular needs and seek to fulfil them. When viewed through a health equity and bias lens, new mind-sets are crucial in shaping the experiences of so-called disadvantaged groups, whose legitimate requirements may be viewed negatively by providers and vendors.

Lastly, existing work pays insufficient attention to the fact that benefits of HIT occur over substantial timeframes (in some instances decades). Longitudinal dimensions are often missing in current studies, which mainly consist of retrospective judgements of success of portals. This insufficient length in follow-up may partly explain the lack of evidence surrounding health outcomes. For instance, follow-up periods of studies included in a recent review of the effectiveness of patient portals ranged between three months to two years [[14]]. Where longitudinal studies over substantial timeframes (10 years or more) exist, these show positive impacts on health outcomes [[47]].

3 Difficulties Associated with Addressing Issues Surrounding Equity, Literacy, and Bias in Evaluations of Patient Portals

Having discussed shortcomings of existing evaluations, it is important to keep in mind that evaluation of complex HIT is a challenging field, especially when viewed in the context of equity, literacy and bias. Evaluators have to navigate complexities that are often characterised by a number of trade-offs. We will discuss some of these in the following paragraphs.

Most importantly, patient portals tend to be implemented in settings that are already relatively digitally advanced and have populations with high sociodemographic status. Disadvantaged users may not be visible in these settings and are by definition hard to reach. It is therefore challenging to involve them in research to assess their needs and in design-related activities [[30]].

On the other end of the spectrum, even the disadvantaged user population is distributed, including various viewpoints and agendas. This population does not only include one disadvantaged group but many. Such a diverse nature of the disadvantaged population poses a dilemma of deciding whose needs get researched and prioritised and why. Diversity of user groups also poses challenges for software development as good usability for one group may have implications for the perceived usability of another group. This usability dilemma is exemplified in the notion of “generification”, where a technology needs to be generic enough to satisfy a range of needs without being too specific, as this will compromise transferability to other contexts and threaten interoperability [[48]].

Similar tensions emerge when considering contexts of deployment and adoption. Contexts vary in relation to existing levels of health literacy and inclusion, and also existing technological, social, organisational, and environmental infrastructures. As a result, what works in one context may not necessarily work in another at any point in time. This also means that systems impacts are extremely difficult to anticipate with many emerging unanticipated benefits and disbenefits unfolding only when systems are used in real-world settings. Although a contextual assessment can help to inform transferability to other settings to some extent [[49]], evaluators face the challenge of identifying which factors are transferable across settings [[50]].

4 New Formative Evaluation Paradigms to Better Address Equity, Literacy, and Bias when Researching Patient Portals

Having outlined the shortcomings of existing approaches and the broader challenges facing the field, we will now discuss evaluation approaches that we believe can help to address considerations surrounding equity, literacy, and bias in researching patient portals. Key here is to embed health equity considerations at every stage of the technology lifecycle [[43]], ideally performing equity assessments as early as possible in the process of planning a digital intervention [[51]]. We will begin with needs assessment and co-design approaches, and move to discuss formative methods to study systems in use.

Formative evaluation methods focus on the question of how a technology can be made to work and how to maximize its benefits (as opposed to summative evaluations that make a judgement after the implementation on whether a technology was successful or not) [[52]]. In HIT research, formative work focuses on improving the development and implementation process.

4.1 Assessing Stakeholder Needs that a Technology Should Address Before Deployment

Many innovations in healthcare settings fail because they do not address an identified need [[53]]. This can be mitigated by strategically targeting those who could benefit most from a certain technology during development and implementation planning. In relation to patient portals, these may include, for instance, patients with high numbers of comorbidities, older demographics, and people with chronic conditions. In doing so, the current focus of portal designers needs to shift from generic uses towards mapping the needs of the most vulnerable users as a full spectrum of use cases [[34]]. Changes may be relatively simple, including for example, addressing language and accessibility needs by making tools available in different languages and by providing audio-visual formats as alternatives to written text [[54]].

However, it is also important to align value propositions across stakeholder groups [[55]]. Patient portals have to meet many (in some instances competing) value demands (i.e., the kind of value they are expected to bring). If there is a mismatch between value propositions, a system is less likely to be adopted and scaled. For example, in the USA, patient portal implementation was not driven by clinical or patient need but by meaningful use incentives (payments healthcare settings received if they could demonstrate meaningful use of HIT), leading to such a mismatch, and consequently issues with adoption [[56]].

4.2 Co-Design of Patient Portals Involving Vulnerable Groups

Despite the challenges of involving vulnerable groups in portal design, a user-centred design approach is crucial to improve health equity and reduce bias. Several frameworks and design approaches have been proposed to address the digital divide and improve health literacy. These include principles of inclusive and participatory design, and ways to improve system usability and accessibility (e.g., navigation, language, use of visuals, limiting text) [[57], [58]]. Effective processes to approach co-design include observing disadvantaged users and their interactions with systems, and partnering with system vendors to create effective user interfaces [[30], [59]]. There may also be scope to draw on the concept of “personas”, a notion commonly used in software engineering to explore key user needs and tailor information system features accordingly [[60]]. These personas need to include types of disadvantaged users who could benefit most from using portals.

4.3 Studying Unanticipated Consequences and Impacts of System in Use Over Time

Benefits of complex HIT systems such as patient portals not only take a long time to materialise but also emerge in unanticipated ways and in unanticipated places once the system is used within individual adoption contexts [[61]]. Amongst these are the observed adverse impacts of patient portals on health equity [[62], [63]]. Whilst portals may benefit some user groups, they might simultaneously increase health disparities in other groups [[64]]. Some studies have also found potential adverse impacts on patient-provider relationships [[63]].

Evaluation designs need to account for these complexities. This can be achieved by incorporating longitudinal dimensions, tracing changes and consequences for a variety of stakeholders over long periods of time and by including formative elements, thereby allowing emerging negative consequences to be mitigated and emerging positive impacts to be measured [[65], [66]]. There are various examples of formative evaluations shaping patient portal development and implementation in the literature [[59]], and these have been found to help reduce bias in systems associated with health inequity [[58]].

4.4 Studying Technological and Human Contexts in which the Portal Is Placed

As discussed above, contextual factors need to be considered in HIT adoption and evaluation. These include socio-organisational and technological dimensions, which are often interrelated [[67]]. Social dimensions involve individual people involved in implementing and adopting HIT and their needs. For example, existing portals often do not offer features in languages other than the native language and are not accessible for visually impaired users, which means that the needs of vulnerable users are often not met [[43]]. Other social factors include organisations implementing systems and their existing cultures, systems and processes. With regard to patient portals, studies have for example found that clinicians are concerned that patient portals create more work. They are therefore often hesitant to actively engage patients around using these technologies [[63]]. Macro environmental factors include standards, funding, legislation, incentives, political, and economic factors [[68]]. There are currently no standards enforcing usability or accessibility of patient portals, potentially hampering inclusive design [[43]]. This is despite some existing evidence that policies that mandate offering portal access to vulnerable groups can increase uptake and use [[69]]. It has also been argued that policy and governance contexts surrounding portals may reinforce health inequity [[70]].

In relation to technological dimensions, evaluators need to consider integration of the patient portal with existing information infrastructures and systems already used by patients (including integration with data already collected). For example, read-only portals are easier to integrate with existing systems as they do not require the creation of interfaces. As systems become more complex and as data is exchanged with other systems, there are increasing challenges surrounding integration with existing technological information infrastructures [[71]]. More sophisticated applications of the future are also likely to require increasing levels of health literacy and increased processing power of devices. This is however, especially problematic in disadvantaged groups who often struggle with access to Wi-Fi and smartphones [[72]]. Other types of digital exclusion include: people living in rural areas with less access to, and slower, internet infrastructure; older people who are less likely to own smartphones or have internet access; and differences between lower socio-demographic groups in levels of internet access [[73]].

In order to promote transferability of insights between settings, evaluators need to draw and build on existing theoretical models surrounding the implementation, adoption and optimisation of systems [[50], [74]]. We have discussed some of these that are relevant to health inequity in the paragraphs above.

The more complex sociotechnical systems become, the more important it is for research to consider the role of context and the risk of technologies reinforcing digital inequity and exclusion [[75], [76]]. For example, a recent development in relation to patient portals is Personal Online Data Stores (PODS), which attempt to overcome the interoperability challenge by making the patient the point of integration. This brings significant opportunities but may also carry significant risks as it may increase health and other inequity (see Box 2).

Box 2 SOLID (SOcial LInked Data).

5 Conclusion

Patient portals are increasingly implemented but, whilst there are many opportunities, current designs and implementations may reinforce health inequity and compromise health equity. Whilst portals provide increased access and opportunities for engagement for some patient populations, they also create new barriers for vulnerable groups who may benefit most from them. An ongoing challenge is finding a balance between creating comprehensive and complex systems that can be transferred between settings and satisfying the demands of many different stakeholder groups.

Existing approaches to evaluation do not sufficiently address issues surrounding equity, literacy, and bias as they do not effectively engage disadvantaged populations in research and design activities. They also do not pay sufficient attention to contextual factors and existing theoretical frameworks. There is therefore a need for developing frameworks for co-design to create systems that are easily accessible for those with low levels of digital and health literacy.

Formative evaluation approaches have significant potential to help tailor functionalities to the needs of those who could benefit most and track unintended positive and negative consequences (Box 3). Future functionalities may involve increasing interactions between healthcare professionals and patients, reducing barriers to access, and promoting specialist tailored resources for those who need it most. This could involve reconceptualising how portals are used, including for example new emerging roles such as patient facilitators for those with low levels of digital and health literacy, the use of real world use cases including anonymised records or composites at the design stage, or developing portals as innovation platforms for patient-led innovative uses [[77]].

Box 3 Formative HIT evaluation features addressing health equity, literacy and bias.

Such new reconceptualised evaluation approaches can help to facilitate the creation of a resilient health system serving the current and emerging needs of those within it.

No conflict of interest has been declared by the author(s).

• References

• 1 Jha AK, Doolan D, Grandt D, Scott T, Bates DW. The use of health information technology in seven nations. Int J Med Inform 2008 Dec 1;77(12):848-54.
• 2 Bhambere HS, Abhishek B, Sumit H. Rapid digitization of healthcare: a review of COVID-19 impact on our health systems. Int J All Res Educ Sci Methods 2021;9:1457-9.
• 3 Glöggler M, Ammenwerth E. Improvement and Evaluation of the TOPCOP Taxonomy of Patient Portals: Taxonomy-Evaluation-Delphi (TED) Approach. J Med Internet Res 31/07/2021:30701. In press.
• 4 Hahn EE, Baecker A, Shen E, Haupt EC, Wakach W, Ahuja A, et al. A Patient Portal-Based Commitment Device to Improve Adherence with Screening for Colorectal Cancer: a Retrospective Observational Study. J Gen Intern Med 2021 Apr;36(4):952-60.
• 5 Miller HN, Gleason KT, Juraschek SP, Plante TB, Lewis-Land C, Woods B, et al. Electronic Medical Record-based Cohort Selection and Direct-to-Patient, Targeted Recruitment: Early Efficacy and Lessons Learned. J Am Med Inform Assoc 2019 Nov 1;26(11):1209-17.
• 6 Varad, NH, d'Amonville S, Chen AF. Electronic Patient Portal Use in Orthopaedic Surgery Is Associated with Disparities, Improved Satisfaction, and Lower No-Show Rates. J Bone Joint Surg Am 2020 Aug 5;102(15):1336-13.
• 7 Turner AM, Osterhage K, Hartzler A, Joe J, Lin L, Kanagat N, et al. Use of Patient Portals for Personal Health Information Management: the Older Adult Perspective. In: AMIA Annu Symp Proc 2015 Nov 5;2015:1234-41.
• 8 Braveman P, Gruskin S. Defining Equity in Health. J Epidemiol Community Health 2003 Apr 1;57(4):254-8.
• 9 Berkman ND, Davis TC, Mc Cormack L. Health Literacy: What is it? J Health Commun 2010 Aug 31;15(S2):9-19.
• 10 Buckingham D. Defining Digital Literacy. Nordic Journal of Digital Literacy 2016:21-34.
• 11 Blair IV, Steiner JF, Havranek EP. Unconscious (implicit) Bias and Health Disparities: Where do we go from here? Permanente J 2011;15(2):71.
• 12 Kruk ME, Myers M, Varpilah ST, Dahn BT. What is a Resilient Health System? Lessons from Ebola. Lancet 2015 May 9;385(9980):1910-2.
• 13 Saeed SA, Masters RM. Disparities in Health Care and the Digital Divide. Curr Psychiatry Rep 2021 Sep;23(9):1-6.
• 14 Ammenwerth E, Neyer S, Hörbst A, Mueller G, Siebert U, Schnell-Inderst P. Adult Patient access to Electronic Health Records. Cochrane Database Syst Revi 2021 Feb 26;2(2):CD012707.
• 15 Wyatt J, Sathanandam S, Rastall P. Personal Health Record (PHR) Landscape Review final report. RCP; 2016. Available from: https://www.rcplondon.ac.uk/projects/outputs/personal-health-record-phr-landscape-review
• 16 Symons JD, Ashrafian H, Dunscombe R, Darzi A. From EHR to PHR: let's get the record straight. BMJ Open 2019 Sep 1;9(9):e029582.
• 17 Electronic Health Records. [cited 2021 October 4] Available from: https://post.parliament.uk/research-briefings/post-pn-0519/
• 18 Halamka JD, Tripathi M. The HITECH Era in Retrospect. N Engl J Med 2017 Sep 7;377(10):907-9.
• 19 Irizarry T, DeVito AD, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148.
• 20 Powell KR. Patient-Perceived Facilitators of and Barriers to Electronic Portal Use: a Systematic Review. Comput Inform Nurs 2017 Nov;35(11):565-73.
• 21 Varsi C, Gammon D, Wibe T, Ruland CM. Patients' Reported Reasons for Non-Use of an Internet-Based Patient-Provider Communication Service: Qualitative Interview Study. J Med Internet Res 2013;15(11):e246.
• 22 Tieu L, Schillinger D, Sarkar U, Hoskote M, Hahn KJ, Ratanawongsa N, et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere? J Am Med Inform Assoc 2017; 24(e1):e47”54.
• 23 Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The digital divide and patient portals. Med Care. 2016 Aug 1;54(8):772-9.
• 24 Anthony D, Campos-Castillo C. Why Most of Your Patients Aren't Using an Online Portal, and What You Can Do About It. R I Med J (2013) 2020 Feb 3;103(1):32-4.
• 25 Chen X, Schofield E, Hay JL, Waters EA, Kiviniemi MT, Orom H. Race/Ethnicity, Nativity Status, and Patient Portal Access and Use. J Health Care Poor Underserved 2021;32(2):700-11.
• 26 Connolly SL, Etingen B, Shimada SL, Hogan TP, Nazi K, Stroupe K, et al. Patient portal use among veterans with depression: Associations with symptom severity and demographic characteristics. J Affect Disord 2020 Oct 1;275:255-9.
• 27 Ukoha EP, Feinglass J, Yee LM. Disparities in Electronic Patient Portal Use in Prenatal Care: Retrospective Cohort Study. J Med Internet Res 2019 Sep 23;21(9):e14445.
• 28 Marmot M, Bell R. The sustainable development goals and health equity. Epidemiology 2018 Jan 1;29(1):5-7.
• 29 Morton S, Pencheon D, Squires N. Sustainable Development Goals (SDGs), and their implementation. A national global framework for health, development and equity needs a systems approach at every level. Br Med Bull 2017 Dec 1;124(1):81-90.
• 30 Showell C, Turner P. The PLU problem: are we designing personal ehealth for people like us? Stud Health Technol Inform 2013;183:276-80.
• 31 Rigby M, Georgiou A, Hyppönen H, Ammenwerth E, de Keizer N, Magrabi F, et al. Patient Portals as a Means of Information and Communication Technology Support to Patient-centric Care Coordination”the Missing Evidence and the Challenges of Evaluation. Yearb Med Inform 2015:148-59.
• 32 Ammenwerth E, Rigby M, editors. Evidence-based Health Informatics - Promoting Safety and Efficiency Through Scientific Methods and Ethical Policy. Stud Health Tech Inform (volume 222). IOS Press; 2016.
• 33 Wildenbos GA, Peute L, Jaspers M. Facilitators and Barriers of Electronic Health Record Patient Portal Adoption by Older Adults: a Literature Study. Stud Health Technol Inform 2017 Jan 1;235:308-12.
• 34 Goedhart NS, Zuiderent-Jerak T, Woudstra J, Broerse JE, Betten AW, Dedding C. Persistent Inequitable Design and Implementation of Patient Portals for Users at the Margins. J Am Med Inform Assoc 2021 Feb;28(2):276-83.
• 35 Showell C, Cummings EA, Turner P. The Invisibility of Disadvantage: Why do we not notice? Stud Health Tech Inform 2017;235:388-92.
• 36 Grossman LV, Masterson Creber RM, Benda NC, Wright D, Vawdrey DK, Ancker JS. Interventions to Increase Patient Portal Use in Vulnerable Populations: a Systematic Review. J Am Med Inform Assoc 2019 Aug;26(8-9):855-70.
• 37 Yasukawa K, Sawada T, Hashimoto H, Jimba M. Health-care Disparities for Foreign Residents in Japan. Lancet 2019 Mar 2;393(10174):873-4.
• 38 Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and Facilitators to Online Portal Use among Patients and Caregivers in a Safety Net Health Care System: a Qualitative Study. J Med Internet Res 2015 Dec 3;17(12):e4847.
• 39 Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why ? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018 Dec 1;37(12):1948-54.
• 40 Mackert M, Mabry-Flynn A, Champlin S, Donovan EE, Pounders K. Health Literacy and Health Information Technology Adoption: The Potential for a New Digital Divide. J Med Internet Res 2016;18(10):e264.
• 41 Coughlin SS, Stewart JL, Young L, Heboyan V, De Leo G. Health Literacy and Patient Web Portals. Int J Med Inform 2018 May 1;113:43-8.
• 42 Lam BD, Bourgeois F, Dong ZJ, Bell S.. Speaking up about patient-perceived serious visit note errors: Patient and family experiences and recommendations. J Am Med Inform Assoc 2021 Mar 18;28(4):685-94
• 43 Lyles CR, Fruchterman J, Youdelman M, Schillinger D. Legal, Practical, and Ethical Considerations for Making Online Patient Portals Accessible for All. Am J Public Health 2017 Oct;107(10):1608-11.
• 44 Fraccaro P, Vigo M, Balatsoukas P, Buchan IE, Peek N, van der Veer SN. The Influence of Patient Portals on Users' Decision Making is Insufficiently Investigated: a Systematic Methodological Review. Int J Med Inform 2018 Mar 1;111:100-11.
• 45 van Mens HJ, Duijm RD, Nienhuis R, de Keizer NF, Cornet R. Determinants and outcomes of patient access to medical records: systematic review of systematic reviews. Int J Med Inform 2019 Sep 1;129:226-33.
• 46 van Mens HJ, Duijm RD, Nienhuis R, de Keizer NF, Cornet R. Towards an adoption framework for patient access to electronic health records: systematic literature mapping study. JMIR Med Inform 2020;8(3):e15150.
• 47 Bao C, Bardhan IR, Singh H, Meyer BA, Kirksey K. Patient”provider engagement and its impact on health outcomes: a longitudinal study of patient portal use. MIS Quarterly 2020 Jun 1;44(2).
• 48 Pollock N, Williams R, D'Adderio L. Global software and its provenance: generification work in the production of organizational software packages. Social Studies of Science 2007 Apr;37(2):254-80.
• 49 Schloemer T. Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis. Implement Sci 2018;13(1):88.
• 50 Coiera E, Ammenwerth E, Georgiou A, Magrabi F. Does health informatics have a replication crisis? J Am Med Inform Assoc 2018 Aug;25(8):963-8.
• 51 Were MC, Sinha C, Catalani C. A systematic approach to equity assessment for digital health interventions: case example of mobile personal health records. J Am Med Inform Assoc 2019 Aug;26(8-9):884-90.
• 52 Elwy AR, Wasan AD, Gillman AG, Johnston KL, Dodds N, Mc Farland C, et al. Using formative evaluation methods to improve clinical implementation efforts: description and an example. Psychiatry research. 2020 Jan 1;283:112532.
• 53 Omachonu VK, Einspruch NG. Innovation in healthcare delivery systems: a conceptual framework. The Innovation Journal: The Public Sector Innovation Journal. 2010 Mar 1;15(1):1-20.
• 54 Yilmaz NG, Sungur H, van Weert JC, van den Muijsenbergh ME, Schouten BC. Enhancing patient participation of older migrant cancer patients: needs, barriers, and eHealth. Ethn Health 2020 Dec 8:1-24.
• 55 Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, Hinder Set al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017;19(11):e367.
• 56 Otte-Trojel T, de Bont A, van de Klundert J, Rundall TG. Characteristics of patient portals developed in the context of health information exchanges: early policy effects of incentives in the meaningful use program in the United States. J Med Internet Res 2014;16(11):e258.
• 57 Antonio MG, Petrovskaya O, Lau F. The State of Evidence in Patient Portals: Umbrella Review. J Med Internet Res 2020;22(11):e23851.
• 58 Veinot TC, Mitchell H, Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018 Aug;25(8):1080-8.
• 59 Casillas A, Abhat A, Mahajan A, Moreno G, Brown AF, Simmons S, et al. Portals of change: how patient portals will ultimately work for safety net populations. J Med Internet Res 2020;22(10):e16835.
• 60 Personas. [cited 2021 October 4] Available from: https://www.usability.gov/how-to-and-tools/methods/personas.html
• 61 Wachter RM. Expected and unanticipated consequences of the quality and information technology revolutions. JAMA 2006 Jun 21;295(23):2780-3.
• 62 Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, Dabbs AD. Patient portals as a tool for health care engagement: a mixed-method study of older adults with varying levels of health literacy and prior patient portal use. J Med Internet Res 2017 Mar 30;19(3):e7099.
• 63 Miller Jr DP, Latulipe C, Melius KA, Quandt SA, Arcury TA. Primary care providers' views of patient portals: interview study of perceived benefits and consequences. J Med Internet Res 2016 Jan 15;18(1):e4953.
• 64 Kim Y, Ayvaci M, Raghunathan S, Tanriover B. Repairing the Digital Divide Can Increase the Service Divide: The Effects of Patient Portals on Kidney Allocation.
• 65 Cresswell K, Sheikh A, Franklin BD, Krasuska M, Nguyen H, Hinder S, et al. Formative independent evaluation of a digital change programme in the English National Health Service: study protocol for a longitudinal qualitative study. BMJ Open 2020 Oct 1;10(10):e041275.
• 66 Cresswell K, Williams R, Sheikh A. Developing and applying a formative evaluation framework for health information technology implementations: qualitative investigation. J Med Internet Res 2020 Jun 10;22(6):e15068.
• 67 Braithwaite J, Runciman WB, Merry AF. Towards safer, better healthcare: harnessing the natural properties of complex sociotechnical systems. BMJ Qual Saf 2009 Feb 1;18(1):37-41.
• 68 Lau F, Price M, Keshavjee K. Making sense of health information system success in Canada. Healthc Q. 2011;14(1):39-46.
• 69 Ancker JS, Nosal S, Hauser D, Way C, Calman N. Access policy and the digital divide in patient access to medical records. Health Policy and Technology 2017 Mar 1;6(1):3-11.
• 70 Antonio MG, Petrovskaya O, Lau F. Is research on patient portals attuned to health equity? A scoping review. J Am Med Inform Assoc 2019 Aug;26(8-9):871-83.
• 71 Aanestad M, Grisot M, Hanseth O, Vassilakopoulou P. Information infrastructures within European health care: Working with the installed base. [cited 2021 October 14]. Available from: https://library.oapen.org/bitstream/handle/20.500.12657/27913/1002086.pdf?sequence=1
• 72 Marler W. Mobile phones and inequality: Findings, trends, and future directions. New Media & Society 2018 Sep;20(9):3498-520.
• 73 Digital technology and health inequalities: a scoping review. [cited 2021 October 14]. Available from: https://phw.nhs.wales/publications/publications1/digital-technology-and-health-inequalities-a-scoping-review/
• 74 Scott P, de Keizer N Georgiou A., editors. Applied Interdisciplinary Theory in Health Informatics: A Knowledge Base for Practitioners. IOS Press; 2019.
• 75 Wong ZS, Nøhr, C, Kuziemsky CE, Leung E, Chen F. Context Sensitive Health Informatics: Delivering 21st Century Healthcare - Building a Quality-and-Efficiency Driven System. Stud Health Technol Inform 2017;241:1-5.
• 76 NHS data: Can web creator Sir Tim Berners-Lee fix it? [cited 2021 October 14]. Available from: https://www.bbc.co.uk/news/technology-54871705
• 77 Oliveira P, Cunha MP. Centralized decentralization, or distributed leadership as paradox: The case of the Patient Innovation's COVID-19 Portal. Journal of Change Management 2021 Apr 28:1-9.

Publication History

Article published online:
02 June 2022