CC BY 4.0 · Methods Inf Med 2023; 62(05/06): 154-164
DOI: 10.1055/a-2155-2021
Original Article for a Focus Theme

A Proposal for a Robust Validated Weighted General Data Protection Regulation-Based Scale to Assess the Quality of Privacy Policies of Mobile Health Applications: An eDelphi Study

Jaime Benjumea
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
Jorge Ropero
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
Enrique Dorronzoro-Zubiete
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
Octavio Rivera-Romero
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
Alejandro Carrasco
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
› Author Affiliations
Funding This work was partially funded by the Cátedra de Telefónica “Inteligencia en la red” of the Universidad de Sevilla and by the Cátedra Indra “Sociedad Digital” of the Universidad de Sevilla. O.R.-R. has received funding from the Universidad de Sevilla and theMinisterio de Universidades of the Spanish Government under the Requalification of Spanish University System Program funded by European Union—NextGenerationEU.


Background Health care services are undergoing a digital transformation in which the Participatory Health Informatics field has a key role. Within this field, studies aimed to assess the quality of digital tools, including mHealth apps, are conducted. Privacy is one dimension of the quality of an mHealth app. Privacy consists of several components, including organizational, technical, and legal safeguards. Within legal safeguards, giving transparent information to the users on how their data are handled is crucial. This information is usually disclosed to users through the privacy policy document. Assessing the quality of a privacy policy is a complex task and several scales supporting this process have been proposed in the literature. However, these scales are heterogeneous and even not very objective. In our previous study, we proposed a checklist of items guiding the assessment of the quality of an mHealth app privacy policy, based on the General Data Protection Regulation.

Objective To refine the robustness of our General Data Protection Regulation-based privacy scale to assess the quality of an mHealth app privacy policy, to identify new items, and to assign weights for every item in the scale.

Methods A two-round modified eDelphi study was conducted involving a privacy expert panel.

Results After the Delphi process, all the items in the scale were considered “important” or “very important” (4 and 5 in a 5-point Likert scale, respectively) by most of the experts. One of the original items was suggested to be reworded, while eight tentative items were suggested. Only two of them were finally added after Round 2. Eleven of the 16 items in the scale were considered “very important” (weight of 1), while the other 5 were considered “important” (weight of 0.5).

Conclusion The Benjumea privacy scale is a new robust tool to assess the quality of an mHealth app privacy policy, providing a deeper and complementary analysis to other scales. Also, this robust scale provides a guideline for the development of high-quality privacy policies of mHealth apps.


A first non-peer-reviewed version of this article is available at Research Square.[57] The current version expands and improves the methodology section. It is also more focused on Participatory Health Informatics than the previous version.

Authors' Contribution

J.B. directed the study and took the lead in conducting the Delphi process. J.B. also supported the data analysis, the calculation of statistical values, and the interpretation of the data. J.R. took the lead in drafting the manuscript, supported by J.B., E.D.-Z., O.R.-R., and A.C. J.R. also participated in study direction, contributed to the calculation of statistical values, and supported the data analysis and interpretation of the data. E.D.-Z. participated in study direction, data analysis, and interpretation of the data. O.R.-R. supported the Delphi process, resolved discrepancies, and reviewed the final version of the manuscript. A.C. resolved discrepancies, acquired funding through a research project, participated in the interpretation of data, and reviewed the final version of the manuscript. J.B., J.R., E.D.-Z., O.R.-R., and A.C. selected and contacted experts for the Delphi process.

Supplementary Material

Publication History

Received: 06 July 2022

Accepted: 11 July 2023

Accepted Manuscript online:
17 August 2023

Article published online:
22 December 2023

© 2023. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Swan M. Health 2050: the realization of personalized medicine through crowdsourcing, the quantified self, and the participatory biocitizen. J Pers Med 2012; 2 (03) 93-118
  • 2 Denecke K, Gabarron E, Petersen C, Merolli M. Defining participatory health informatics - a scoping review. Inform Health Soc Care 2021; 46 (03) 234-243
  • 3 Caulfield BM, Donnelly SC. What is Connected Health and why will it change your practice?. QJM 2013; 106 (08) 703-707
  • 4 Whittaker R. Issues in mHealth: findings from key informant interviews. J Med Internet Res 2012; 14 (05) e129
  • 5 Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task?. Transl Behav Med 2011; 1 (01) 53-71
  • 6 Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One 2017; 12 (02) e0170581
  • 7 Ghani Z, Jarl J, Sanmartin Berglund J, Andersson M, Anderberg P. The cost-effectiveness of mobile health (mHealth) interventions for older adults: systematic review. Int J Environ Res Public Health 2020; 17 (15) 5290
  • 8 Rinaldi G, Hijazi A, Haghparast-Bidgoli H. Cost and cost-effectiveness of mHealth interventions for the prevention and control of type 2 diabetes mellitus: a systematic review. Diabetes Res Clin Pract 2020; 162: 108084
  • 9 de Batlle J, Massip M, Vargiu E. et al; CONNECARE-Lleida Group. Implementing mobile health-enabled integrated care for complex chronic patients: intervention effectiveness and cost-effectiveness study. JMIR Mhealth Uhealth 2021; 9 (01) e22135
  • 10 Giunti G, Guisado Fernández E, Dorronzoro Zubiete E, Rivera Romero O. Supply and demand in mhealth apps for persons with multiple sclerosis: systematic search in app stores and scoping literature review. JMIR Mhealth Uhealth 2018; 6 (05) e10512
  • 11 Martín Payo R, Harris J, Armes J. Prescribing fitness apps for people with cancer: a preliminary assessment of content and quality of commercially available apps. J Cancer Surviv 2019; 13 (03) 397-405
  • 12 Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digit Med 2018; 1: 12
  • 13 ISO/TS 82304–2:2021. Health Software. Part 2: Health and Wellness apps – Quality and reliability. Accessed August 22, 2023 at:
  • 14 Ribeiro N, Moreira L, Barros A, Almeida AM, Santos-Silva F. Guidelines for a cancer prevention smartphone application: a mixed-methods study. Int J Med Inform 2016; 94: 134-142
  • 15 Robertson MC, Tsai E, Lyons EJ. et al. Mobile health physical activity intervention preferences in cancer survivors: a qualitative study. JMIR Mhealth Uhealth 2017; 5 (01) e3
  • 16 Monteiro-Guerra F, Signorelli GR, Rivera-Romero O, Dorronzoro-Zubiete E, Caulfield B. Breast cancer survivors' perspectives on motivational and personalization strategies in mobile app-based physical activity coaching interventions: qualitative study. JMIR Mhealth Uhealth 2020; 8 (09) e18867
  • 17 Giunti G, Kool J, Rivera Romero O, Dorronzoro Zubiete E. Exploring the specific needs of persons with multiple sclerosis for mHealth solutions for physical activity: mixed-methods study. JMIR Mhealth Uhealth 2018; 6 (02) e37
  • 18 Benjumea J, Ropero J, Rivera-Romero O, Dorronzoro-Zubiete E, Carrasco A. Privacy assessment in mobile health apps: scoping review. JMIR Mhealth Uhealth 2020; 8 (07) e18868
  • 19 Minen MT, Stieglitz EJ, Sciortino R, Torous J. Privacy issues in smartphone applications: an analysis of headache/migraine applications. Headache 2018; 58 (07) 1014-1027
  • 20 Leigh S, Ouyang J, Mimnagh C. Effective? Engaging? Secure? Applying the ORCHA-24 framework to evaluate apps for chronic insomnia disorder. Evid Based Ment Health 2017; 20 (04) e20
  • 21 O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv 2018; 15: 110-115
  • 22 Knorr K, Aspinall D, Wolters M. On the privacy, security and safety of blood pressure and diabetes apps. In: Proceedings of the IFIP (International Federation for Information Processing) International Conference on ICT (Information and Communications Technology) Systems Security and Privacy Protection. Cham, Switzerland: Springer; 2015 May 9. Presented at: IFIP (International Federation for Information Processing) International Conference on ICT (Information and Communications Technology) Systems Security and Privacy Protection; May 26–28, 2015; Hamburg, Germany; 571–584
  • 23 Benjumea J, Ropero J, Rivera-Romero O, Dorronzoro-Zubiete E, Carrasco A. Assessment of the fairness of privacy policies of mobile health apps: scale development and evaluation in cancer apps. JMIR Mhealth Uhealth 2020; 8 (07) e17134
  • 24 Papageorgiou A, Strigkos M, Politou E, Alepis E, Solanas A, Patsakis C. Security and privacy analysis of mobile health applications: the alarming state of practice. IEEE Access 2018; 6: 9390-9403
  • 25 Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open 2019; 2 (04) e192542
  • 26 Sunyaev A, Dehling T, Taylor PL, Mandl KD. Availability and quality of mobile health app privacy policies. J Am Med Inform Assoc 2015; 22 (e1): e28-e33
  • 27 Scott K, Richards D, Adhikari R. A review and comparative analysis of security risks and safety measures of mobile health apps. AJIS Australas J Inf Syst 2015; 19: 1-18
  • 28 Zapata B, Hernández Niñirola A, Fernández-Alemán JL, Toval A. Assessing the privacy policies in mobile personal health records. In: Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014). New York, NY: IEEE; 2014; Presented at: 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014); August 26–30, 2014; Chicago, IL; 4956–4959
  • 29 Bachiri M, Idri A, Fernández-Alemán JL, Toval A. Evaluating the privacy policies of mobile personal health records for pregnancy monitoring. J Med Syst 2018; 42 (08) 144
  • 30 Hutton L, Price BA, Kelly R. et al. Assessing the privacy of mHealth apps for self-tracking: heuristic evaluation approach. JMIR Mhealth Uhealth 2018; 6 (10) e185
  • 31 Baumel A, Faber K, Mathur N, Kane JM, Muench F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J Med Internet Res 2017; 19 (03) e82
  • 32 Bondaronek P, Alkhaldi G, Slee A, Hamilton FL, Murray E. Quality of publicly available physical activity apps: review and content analysis. JMIR Mhealth Uhealth 2018; 6 (03) e53
  • 33 Ni Z, Wang Y, Qian Y. Privacy policy compliance of chronic disease management apps in China: scale development and content evaluation. JMIR Mhealth Uhealth 2021; 9 (01) e23409
  • 34 Robillard JM, Feng TL, Sporn AB. et al. Availability, readability, and content of privacy policies and terms of agreements of mental health apps. Internet Interv 2019; 17: 100243
  • 35 Powell AC, Singh P, Torous J. The complexity of mental health app privacy policies: a potential barrier to privacy. JMIR Mhealth Uhealth 2018; 6 (07) e158
  • 36 Adhikari AR, Richards D, Scott K. Security and privacy issues related to the use of mobile health apps. Paper presented at: Proceedings of the 25th Australasian Conference on Information Systems; 2014; Auckland, New Zealand; 8–10
  • 37 Vaughan-Graham J, Cott C. Defining a Bobath clinical framework - a modified e-Delphi study. Physiother Theory Pract 2016; 32 (08) 612-627
  • 38 Robustillo Cortés MdeL, Cantudo Cuenca MR, Morillo Verdugo R, Calvo Cidoncha E. High quantity but limited quality in healthcare applications intended for HIV-infected patients. Telemed J E Health 2014; 20 (08) 729-735
  • 39 Administrative Simplification HIPAA. 45 CFR Parts 160, 162, and 164 (unofficial version, as amended through March 26, 2013). Accessed October 2021 at:
  • 40 Savage M, Savage LC. Doctors routinely share health data electronically under HIPAA, and sharing with patients and patients' third-party health apps is consistent: interoperability and privacy analysis. J Med Internet Res 2020; 22 (09) e19818
  • 41 FTC Warns Health Apps and Connected Device Companies to Comply With Health Breach Notification Rule. FTC Press Releases. Accessed October 2021 at:
  • 42 European Parliament and Council. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), EUR-Lex. Accessed October 2021 at: . Last consulted: October 2021
  • 43 Mense MA, Urbauer P, Sauermann S, Wahl H. Simulation environment for testing security and privacy of mobile health apps. Paper presented at: Modeling and Simulation in Medicine Symposium; 2016; Pasadena, CA; 3–6
  • 44 Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manage Sci 1963; 9 (03) 458-467
  • 45 Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. London: Addison-Wesley; 1975
  • 46 Fischer RG. The Delphi method: a description, review and criticism. J Acad Librariansh 1978; 4: 64-70
  • 47 Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017; 31 (08) 684-706
  • 48 McKenna HP. The Delphi technique: a worthwhile research approach for nursing?. J Adv Nurs 1994; 19 (06) 1221-1225
  • 49 Von der Gracht HA. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 2012; 79 (08) 1525-1536
  • 50 Schmidt RC. Managing Delphi surveys using non-parametric statistical techniques. Decis Sci 1997; 28: 763-774
  • 51 Golenko X, Pager S, Holden L. A thematic analysis of the role of the organisation in building allied health research capacity: a senior managers' perspective. BMC Health Serv Res 2012; 12 (276) 276
  • 52 Robles N, Puigdomènech Puig E, Gómez-Calderón C. et al. Evaluation criteria for weight management apps: validation using a modified Delphi process. JMIR Mhealth Uhealth 2020; 8 (07) e16899
  • 53 Studnek JR, Lerner EB, Shah MI. et al. Consensus-based criterion standard for the identification of pediatric patients who need emergency medical services transport to a hospital with higher-level pediatric resources. Acad Emerg Med 2018; 25 (12) 1409-1414
  • 54 Universidad de Sevilla. Record of processing activities for “Evaluación de Políticas de Privacidad: Estudio no experimental con expertos y usuarios” (in Spanish). Accessed August 22, 2023 at:
  • 55 Birko S, Dove ES, Özdemir V. Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation. PLoS One 2015; 10 (08) e0135162
  • 56 Brüggemann BT, Hansen J, Dehling T, Sunyaev A. Privacy technologies and policy. In: Schiffner S, Serna J. eds. An Information Privacy Risk Index for mHealth Apps. Cham: Springer; 2016
  • 57 Benjumea J, Ropero J, Dorronzoro-Zubiete E, Rivera-Romero O, Carrasco A. A proposal for a robust validated weighted general data protection regulation-based scale to assess the fairness of privacy policies of mobile health applications: a Delphi study. 21 June 2022, PREPRINT (Version 1) available at Research Square DOI: 10.21203/