Evaluation of Mobile Apps Used among Medical Students for Learning and Education: A Mixed-Method Concurrent Triangulation Approach

Abstract

Background The use of medical mobile applications (apps) to enhance learning is widespread in medical education. Despite the large number of medical apps used among medical students for learning, many apps have not been evaluated using validated instruments. Finding relevant and valuable apps for student learning might be difficult.

Objective The objective is to evaluate four medical apps (AMBOSS, ISABEL, Medscape, and OSMOSIS) in improving just-in-time learning among medical students using the Mobile App Rubric for Learning (MARuL).

Methods We employed a mixed-method concurrent triangulation approach. The study included the entire population of third-year medical students at King Saud University. These students were selected due to their use of medical apps for learning for at least 1 year. The MARuL, which consists of four measures: Teaching and Learning, User-centered, Professional, and Usability, was electronically distributed to medical students for assessment. Descriptive statistics were calculated for all measures, and thematic content analysis was applied to analyze responses to open-ended questions.

Results The MARuL evaluation revealed that the OSMOSIS app received the highest Usefulness for Learning Score, with Usability Measures being the most positively evaluated measure. Following was the AMBOSS app, which scored highest in the Professional Measures, along with the Medscape and the ISABEL apps. We identified several key features and challenges from analyzing students' responses to open-ended questions regarding the use of the four medical apps. These included six themes: (1) study support and learning tools, (2) comprehensive information, (3) interface experience, (4) cost and accessibility issues, (5) learning methods and content limitations, and (6) user experience and technical issues.

Conclusion Equipping medical students with the ability to evaluate and choose medical apps that facilitate just-in-time learning is a crucial element that should be integrated into the medical curriculum.

Keywords

medical education apps - mHealth - just-in-time learning - evaluation - Saudi Arabia

Background and Significance

In recent years, mobile applications (apps) in medical education have become increasingly prevalent.[1] The proliferation of smartphones and tablets has provided medical students with convenient access to various educational resources, including medical apps. As a result, medical students can access apps for various purposes, including medical reference, drug information, clinical decision-making, anatomy, and procedural guidance. In addition, these apps offer the convenience of instant access to up-to-date medical information, which can significantly facilitate the learning process and enhance the educational value and just-in-time learning experience.[2] [3] [4] [5]

While these apps offer numerous benefits and include a diverse range of topics and functionalities, they also have drawbacks, including app quality, accuracy, and the lack of standardized evaluation methods.[6] [7] To ensure the quality and effectiveness of medical apps to be integrated into the teaching and learning process for medical students, some apps may need additional features and rigorous evaluation. Furthermore, the numerous apps available in the market can make it challenging for medical students to navigate and select the most suitable ones for their educational needs.

Few studies in the literature evaluate medical apps designed to facilitate the learning of medical students.[8] A study by Hudder et al focused on the Osmosis app and involved 1,135 medical students. Authors reported that the highest users experienced an increase in their average test scores and enhanced confidence when approaching course examinations, while moderate users noted that the app aided them in staying organized in their studies.[9] Another study by Golenhofen et al investigated the eMed app, a mobile learning instrument for anatomy, and found that frequent users achieved better results in an anatomy examination compared to low users. However, the study did not establish a causal relationship, as other factors may also influence students' outcomes.[10]

App evaluation methods have been developed, which often examine content accuracy, user interface design, functionality, interactivity, and adherence to evidence-based guidelines.[3] [11] Validated evaluation instruments and rating scales have also been suggested to provide a standardized approach to app assessment such as the Information Literacy Conceptual Framework[11] and the Quality Principles and Operationalization framework,[12] enabling medical students to make informed decisions about which apps to incorporate into their learning process.

Although several studies have developed and evaluated the quality and educational value of medical apps,[13] and their use by patients,[14] there remains a need for a greater understanding of how medical students assess the learning value of these apps. The lack of specific research in this area makes it challenging to guide medical students in selecting apps that effectively support their just-in-time learning needs and to incorporate medical apps within the medical curriculum.

Objectives

This study aims to evaluate the educational value of four medical apps in enhancing just-in-time learning among medical students. Educational value refers to the usefulness of an object in achieving short- or long-term objectives.[15] Just-in-time learning is a method of learning that is independent of time and place, facilitated by easy access to learning resources through technology.[16] [17] Additionally, we aim to assess these apps based on their teaching and learning aspects, user-centeredness, professionalism, and usability, utilizing the standardized Mobile App Rubric for Learning (MARuL) evaluation instrument.[18] Through a systematic evaluation and comparison of these apps' educational value, quality, and user experience, this study seeks to provide valuable insights to guide faculty and students in selecting the most suitable and practical apps for medical education.

Methods

Study Setting, Design, and Participants

Third-year medical students at King Saud University (KSU) participated in a mandatory course in medical informatics, which equips them with foundational knowledge in the field. As part of this course,[19] three workshops titled “Health Apps Evaluation” were held from December 5 to 7, 2022. These workshops, conducted over 3 days, aimed to empower students to assess medical apps using a standardized framework. Each workshop lasted 3 hours. The list of medical apps to be included in the workshop was selected based on a recently published study that identified the most used medical apps among medical students at KSU.[20] These apps included (1) the AMBOSS app, a medical learning platform that includes features such as articles, videos, flashcards, and an extensive question bank used for self-assessment[21]; (2) the Isabel app, a clinical reasoning platform that encourages real-life learning by using case-based learning and evidence-based instruments[22]; (3) the Medscape, an app offering up-to-date online medical resources and clinical tools such as medical calculators, drug interaction checker, and procedural videos[23]; and (4) the Osmosis app, a learning platform providing videos, questions, flashcards, notes, and other resources in visual formats.[24]

We employed a mixed-method concurrent triangulation approach in this study. The participants included the total population of third-year medical students at KSU, totaling 275 students. These students were enrolled in the medical informatics course and had been using medical apps for learning purposes for at least 1 year.

This study was approved by the Institutional Review Board at the College of Medicine, KSU (IRB no.: E-23-7811).

App Evaluation Form

We used the MARuL instrument, a validated and reliable tool used by educators to assess the value of apps for supporting just-in-time learning.[18] The MARuL instrument was rigorously tested by Gladman et al, the instrument's developers, using a sample of 46 medical apps. These apps were randomly selected from both the iOS and Google Play Store, ensuring a diverse range of apps in terms of features and use cases. The comprehensive testing process demonstrated the instrument's reliability and its applicability across various types of learning apps. The instrument consists of 25 items with four measures: (1) Teaching and Learning, (2) User-centered, (3) Professional, and (4) Usability. The Teaching and Learning measures evaluate the app's ability to teach and learn in several aspects, such as purpose, capacity to generate learning, relevance to study/course, and instructional features. The User-centered measures aim at assessing the user's engagement in terms of subjective quality, satisfaction, perceived usefulness, perceived importance, user experience, and intention to reuse. The Professional measures assess the app's lining with professional standards, the credibility of developers, and information quality. The Usability measures assess the app's aesthetics, functionality, differentiation, ease of use, lack of advertisement, technical specification, and whether the app is better than the web-based version.[18]

Before the workshop, the MARuL instrument underwent a content validity review by three experts in medical informatics and e-learning. Some wording was adjusted based on their feedback, and an online form was created using Google Forms.[21] The form was kept in English, as all courses at KSU are taught in English, and students must complete their preparatory year in English before entering medical college. Two open-ended questions were added at the end of the form to gather students' experiences and challenges faced when using the apps.

Survey Distribution and App Selection

At the beginning of the workshop, instructors led a discussion on utilizing the MARuL instrument for medical app evaluation and explained the use of the instrument and its items in detail. The survey distribution was conducted in two phases: (1) individual work and (2) group work. In Phase 1 of the individual work, instructors presented a predetermined list of medical apps, selected based on a previous study outlining the top-used medical apps by medical students at KSU.[20] Students were then instructed to choose an app from the list based on their prior experience and current usage of the apps, after which they initiated the evaluation process by accessing the form link. Phase 2, group work was then introduced, where instructors organized students into groups based on their chosen apps. These groups collaborated to collectively respond to the open-ended questions. Overall, there were 31 student groups, each comprising 6 to 12 students.

Statistical Analysis

The collected data were exported in Excel File format from Google Forms. Data were analyzed using Microsoft Excel and descriptive statistics were used to present frequencies and percentages. Descriptive statistics were used to examine the Overall Usefulness of Learning Score for each app. To analyze responses to the open-ended questions, a Thematic Content Analysis was deployed as described by Braun and Clarke.[25] At the start, two researchers (R.A.A. and J.N.A.) thoroughly reviewed all responses to create the coding schema. The researcher (R.A.A.) subsequently applied the coding schema to code the responses. Afterward, the researcher (J.A.A.) conducted a detailed review of the codes and themes ensuring 100% agreement and consensus.

Results

Medical Apps Evaluation

Out of the 275 student participants, 115 (41.8%) used the AMBOSS app, 72 (26.1%) used the Medscape app, 47 (15.2%) used the Osmosis app, and 41 (14.9%) used the Isabel app. Regarding the average Usefulness for Learning Score, the Osmosis app received the highest score (88.2), followed by AMBOSS (83.2), as shown in [Fig. 1].

Fig. 1 Average usefulness for learning score out of 104.

As illustrated in [Fig. 2], the Osmosis app received the highest scores in each of the four measures when compared to other apps, followed by the AMBOSS app, the Medscape app, and then the Isabel app. Among the four measures for each app, the Osmosis received the highest evaluation in the Usability measures (86) compared to the other measures. The AMBOSS and the Isabel apps received the highest evaluation in the Professional measures (84 and 72, respectively), with Medscape also scoring highest in the Professional measures (77). However, Osmosis scored the lowest in the Teaching and Learning measures (84) compared to the other measures. For the AMBOSS and the Isabel apps, the User-centered measures were the lowest score among the four measures (75 and 52, respectively). Lastly, the Medscape received the lowest score in the Usability measures (68).

Fig. 2 Medical app evaluation percentages based on the MARuL four measures. MARuL, Mobile App Rubric for Learning.

Students' Experience with Using Medical Apps

There was a total of 231 statements written by student groups in response to the open-ended questions. Out of these, 42 statements pertained to the Osmosis app, 114 to the AMBOSS app, 43 to the Medscape app, and 32 to the Isabel app. Analysis of the statements yielded numerous distinct categories of comments related to the advantages, features, and limitations of the four apps. From these categories, six themes emerged that together encompass all comments collected during the workshop. While some data elements could be assigned to unique categories, many represented more than one concept and were therefore cross-categorized under the same theme. For example, if a comment addressed the availability of visual aids as learning instruments in preparation for exams, it was coded as relating to both support through visual aids and readiness for examinations. The six broad themes included (1) study support and learning instruments, (2) comprehensive information, (3) interface experience, (4) cost and accessibility issues, (5) learning methods and content limitations, and (6) user experience and technical issues. The six themes along with their constituent categories, app group, and example quotations are summarized in [Table 1].

Table 1 Summary table of emergent themes, their underlying categories, app group, and representative quotations

Themes	Underlying categories	App groups	Representative quotations
Theme 1: Study Support and Learning Instruments	• Support through visual aids	Osmosis app AMBOSS app Medscape app	• “Osmosis flash cards are amazing and the videos that they produce are one of the best. They're short but super detailed, and helps me to really understand some tough concepts that lectures alone can't convey.” • “The app [AMBOSS] recommends videos for a better understanding of the topic, and a lot of graphs and pictures to clarify the information.” • “Medscape helped us with visual aids to learn about surgical procedures and radiology”
	• Readiness for examinations • Integral component of studying and learning • Organize study schedules and time management	Osmosis app	• “Deeper understanding of concepts and feeling more prepared for the exam” • “As a medical student, we're inundated with so much information, it's difficult to know what's important. Osmosis provides you with the right study techniques to help us retain, understand, and study more efficiently throughout med school.” • “The ability of setting an exam date & the subjects I'll be tested on and getting a schedule of revision was of immense help with my time management.”
	• Extensive question bank and performance analysis	AMBOSS app	• “The app [AMBOSS] provides a Q-bank for every topic, the answers are explained in detail to help provide better understanding”
	• Cases and quizzes for concept recall	Medscape app	• “It [Medscape app] provide cases and quizzes which helps in recalling the concepts better”
	• Comprehensive clinical diagnosis instruments	Isabel app	• “Give all the differential diagnosis and mark the most common. It has more than 6000 differential diagnosis”
Theme 2: Comprehensive information	• Knowledge and skills enhancement	Osmosis app AMBOSS app Medscape app	• “Osmosis is very useful in making the topic simple with basic knowledge and clinical practice” • “Videos for theoretical topics and clinical sessions are available for most topics” • “The app offers both theoretical and clinical content”
	• Information sources available • Essential information highlighted	AMBOSS app	• “Presenting only up-to-date clinical information with reference to the source material.” • “It [AMBOSS app] has a characteristic feature which we have not found in any other website or app. Which is basically providing the high-yield option, which eliminates unnecessary information while keeping only the most important ones.”
	• Up-to-date medical content • Drug information database • Medical calculators and medical news	Medscape app	• “The [Medscape app] app is well made, maintained and updated regularly.” • “It [Medscape app] provides a comprehensive pill identifier which quickly and easily identifies drug name by imprint, shape, color and form.” • “Provides multiple medical calculators” “Provides daily medical news with a functional podcast availability”
	• Enhanced diagnostic capabilities • Reliable evidence-based clinical resources	Isabel app	• “Give all the differential diagnosis and mark the most common It [Isabel app] has more than 6000 differential diagnosis” • “…the resources used in aiding clinical judgment are credible evidence based resources.”
Theme 3: Interface Experience	• User-friendly interface	Osmosis app AMBOSS app Isabel app	• “…easy to use and has no ads” • “AMBOSS has a marvelous interface that is easy enough to be familiar with from the first time you use it” • “User friendly interface with good online navigation”
	• Customizability	AMBOSS app	• “I can challenge myself by initially hiding answer options and I can explore examples of clinical findings like X-rays and EKG which are very useful.” “…plus the app design is modern and familiar.”
	• Smooth performance	Isabel app	• “The app was smooth and clear of bugs.”
	• Complicated user interface	Medscape app	• “User interface has many icons and it is complicated for first-time users, The news feature is integrated with educational content which makes the interface complicated.”
Theme 4: Functional, Accessibility, and Cost Issues	• Functional and accessibility limitations	Osmosis app Medscape app Isabel app	• “Connect to the internet in term to use the application unless you download the material.” • “Patient education materials can't be used offline” • “Not available on iOS, only on Android”
	• Onboarding and navigation challenges	Medscape app	• “Difficult onboarding experience due to lack of tutorials compared to other applications”
	• Expensive app subscription	Osmosis app AMBOSS app ISABEL app	• “Expensive (73 SAR per month) which is a big challenge for a university student.” • “The app [AMBOSS app] is expensive for some students to afford For example, the basic subscription costs 29.99$ a month and 159$ a year ± 299$ for the Q bank” • “The app [Isabel app] is not free and it costs 149.99 USD annually so it cant be widely available”
Theme 5: Learning Methods and Content Limitations	• Restricted learning methods	Osmosis app AMBOSS app	• “Inadequate ability to capture and revise notes” • “pictures, videos and graphs can't be seen without internet connection”
	• Content shortcomings and limited resources	Osmosis app AMBOSS app	• “Not all topics are found” • “Doesn't have exclusive media and only provides media from other platforms
	• Advanced medical knowledge (may not be suitable for students)	Medscape app	• “One major issue that Medscape isn't for exam purposes and students, it's more directed towards physicians”
	• Limited patient information input • Lack of management plans	Isabel app	• “The app [Isabel app] does not provide an input to assign comorbidities or risk factors the patient known to have Beforehand e.g. (diabetes, hypertension, medications, smoking). You can only input the symptoms (not even signs or examinations! Does not have specific symptoms (Like The area of headache)” • “Doesn't provide a management plan”
Theme 6: User Experience and Technical Issues	• Technical glitches	Osmosis app AMBOSS app Isabel app	• “Some bugs and crashes” • “The application has several bugs and crashed multiple times” • Some technical issues with some of us (logging out spontaneously)”
	• Navigation complexity • Connectivity challenges	AMBOSS app	• “The inter-face of the application is confusion compared to the website” • “The app is server bound; if there are problems from the developer‘s end the user struggles with connection.”
	• Limited interactivity • Information displayed with ads	Medscape app	• “Cases and quizzes are very high level and no option to customize difficulty” “Lack user interactivity.” • “very useful application, but the disadvantage is the large number of ads”
	• Learning curve	Isabel app	• “It [Isabel]takes time to familiarize yourself ”

Theme 1: Study Support and Learning Instruments

Students identified several categories, which highlighted the significance of the four apps in supporting their studies. Three out of the four apps (Osmosis, AMBOSS, and Medscape) were highlighted for their ability to support students' learning through the use of different visual aids. One of the key categories identified was the Osmosis app's ability to enhance readiness for examinations through the use of visual aids, providing students with a valuable resource to prepare effectively “uses [the Osmosis app] interesting graphics and drawing for explaining, which makes it easier to remember in exams or discussions”. The AMBOSS app was highlighted for its effective learning instruments, including extensive question banks and performance analysis features, which allowed for comprehensive self-assessment and improvement: “We used the app [the AMBOSS app] since the beginning of the 3rd year. It proved to be a good source for learning and a great time saver for studying. It also provided us with a Q bank that had helped us improve significantly”. The inclusion of cases and quizzes for concept recall was also noted as a beneficial feature identified in the Medscape app: “Comprehensive with well rounded topics… cases and quizzes”. The availability of comprehensive clinical diagnosis instruments within the Isabel app was also highlighted: “Answers [the Isabel app] the related questions about the clinical features and gives us a ranked differential diagnosis”.

Theme 2: Comprehensive Information

Our analysis revealed a major theme among students, emphasizing the importance and value of having comprehensive and reliable information in medical apps. Students highly valued the Osmosis, AMBOSS, and Medscape apps for presenting content that enhanced their knowledge about specific topics and improved their clinical skills: “I feel like it's [the Osmosis app] an essential part of my study routine… Linking clinical practice with basic knowledge”, “Very inclusive app from A to Z including both clinical and basic science information”. The AMBOSS app was appreciated for providing the sources of information: “The app provide effective learning by taking the information from its source directly” and its feature in highlighting essential information for students “Delivery of options like ‘highlight’ and ‘high yield’ which help the student to focus on most relevant information for exams”. Students also appreciated the Medscape app for its extensive up-to-date medical content: “Latest updated medical and surgical guidelines”, and the drug information database: “Has a huge database for drugs and provides the ability for medical professionals to identify pills and check for drug-drug interactions”, and the ability to access to medical calculators and news: “It [the Medscape app] provides podcasts which we can listen to during free time to enhance our knowledge”. As for the Isabel app, students appreciated its feature in offering enhanced diagnostic capabilities and its reliable, evidence-based clinical resources “… the resources used in aiding clinical judgment are credible evidence based resources. An overall satisfactory experience”.

Theme 3: Interface Experience

The interface experience was a significant theme identified by students. Three out of the four apps (Osmosis, AMBOSS, and Isabel) were highlighted for their user-friendly interface. The AMBOSS app was found to be customizable, intuitive, and easy to navigate: “The user interface is unique in AMBOSS. An interface that is tailored to medical students and physicians needs with an option to view detailed articles or the high yield information only”. Students particularly appreciated the user-friendly navigation and smooth performance of the Isabel app: “…interface is intuitive and practical to utilize”. However, some students noted that the Medscape app had a complicated user interface, which made it challenging to use: “Lack of search history—The search function was only applicable to titles of articles, not the full text”.

Theme 4: Functional, Accessibility, and Cost Issues

The theme of cost and accessibility issues reveals significant challenges faced by students when using medical apps for learning. Students pointed out functional and accessibility limitations that hindered their usage experience with three apps: the Osmosis app “Osmosis needs Wi-Fi to work which can be challenging in some areas”, the Medscape app “Free version contains ads”, and the Isabel app “The iOS app got removed”. Additionally, they faced onboarding and navigation challenges when trying to use the Medscape app “…complicated for first-time users”. The expensive subscription fees for the Osmosis, AMBOSS, and Isabel apps were also major concerns among students, making them less accessible for those with limited financial resources.

Theme 5: Learning Methods and Content Limitations

The theme of learning methods and content limitations highlights several challenges faced by students. These challenges include restricted learning methods identified with the use of the Osmosis app: “There isn't a space for students to discuss and ask questions amongst themselves” and with the AMBOSS app: “lacking of mind maps and it needs more videos content”. Shortcomings and limited resources were also highlighted by some students using the Osmosis app “You cannot search for a specific keyword in an article” and the AMBOSS app “our study objectives didn't quite meet the website's article names”. Some students mentioned that the Medscape app presented advanced medical knowledge, which may not be suitable for medical students, stating: “The app has very detailed information that may confuse medical students; it suits physicians more than students”. Limitations in patient information input and the lack of management plans within the Isabel app were also identified, with some students: “No detailed questions about the patient (nationality, race, and past medical history)” and “Does not provide management recommendations”.

Theme 6: User Experience and Technical Issues

The theme of user experience and technical issues was derived from the various challenges highlighted by the students when they used these four apps. Technical glitches were reported with the Osmosis, the AMBOSS, and the Isabel apps, where some students experienced sudden crashes and nonresponsive features. Navigation complexity and connectivity issues were reported with the AMBOSS app “One of the issues was slow loading speed from our experience”. Limited interactivity and the presence of ads particularly in the Medscape, was seen as distracting and disruptive to the learning experience “…Couldn't be used [Medscape app] as a primary recourse”. Lastly, some students mentioned a learning curve associated the ISABEL app “…Lack of comprehension”.

Discussion

Medical students frequently use numerous medical apps for learning; however, many of these apps lack a thorough evaluation using validated instruments. With limited studies exploring the assessment of the educational value of specific medical apps, the responsibility often falls on faculty members, who are commonly regarded as the primary source of information.[26] This study aimed to evaluate the value of four medical apps in improving just-in-time learning among medical students using the MARuL instrument.[18]

The structured evaluation results indicated that among third-year medical students, the AMBOSS app was the most frequently used. This preference was attributed to features such as the High-yield Focus feature, which highlights essential information. These findings align with previous studies that have also reported high usage of the AMBOSS app among medical students.[27] [28] Despite being perceived as too advanced for student use by our participants, the Medscape app was found to be the second most used app, which is consistent with findings from other studies.[7] [29] [30] The Medscape app's extensive medical knowledge may make it more suitable for health care professionals, as supported by Rusatira et al, where it was predominantly utilized by health care professionals.[31]

The Osmosis app received positive reviews from students, scoring the highest for Overall Usefulness. It was particularly valued for helping to organize study schedules and prepare for examinations. The app's widespread use across more than 200 institutions worldwide, with over 5 million learners, highlights its popularity.[24] A study evaluating the app's usage indicated that increased adoption was linked to faculty champions actively involving students, highlighting the importance of faculty guidance in app selection.[9] In contrast, the Isabel app had the lowest number of users and evaluation scores among the included apps due to its lack of fundamental features. However, students recognized its potential utility in enhancing diagnostic capabilities, as it provides “The Isabel Clinical Educator” which supports clinical reasoning and diagnostic skills.[22]

The thematic analysis revealed both positive features and drawbacks of the evaluated medical apps, as perceived by medical students. Students emphasized several features that they believed enhanced their learning outcomes, including flashcards, question banks, note-taking capabilities, access to current articles, clinical videos, daily medical news, medical calculators, updated guidelines, medical cases, and a user-friendly interface. However, students also identified drawbacks that could reduce the educational benefits of these apps. These drawbacks encompassed subscription costs, technical issues, insufficient coverage of clinical skills topics, lack of standardized interface across different devices, absence of mind maps, and the perception that the app is not specifically tailored for students. Among the significant themes identified in the study were technical issues and limited internet access, which were consistent with findings from another study.[32] Addressing these concerns by providing offline content accessible anytime, regardless of internet access, could potentially enhance mobile learning opportunities for medical students. Another prevalent drawback highlighted in the study was the cost of app subscriptions, identified as a barrier in three out of four apps evaluated. This barrier was also noted in another study, which suggested that medical universities could contribute to covering the cost of these apps.[26] Therefore, advocating for and implementing college-based subscriptions could be a recommended solution to overcome these barriers.

Our study highlights the need for collaborative initiatives between faculty and students in selecting the best apps for medical education. While we provided examples, other studies have identified various apps such as the widely trusted UpToDate app,[33] emphasizing the importance of considering a range of options.[30] Overall, the findings indicate that content within these apps adheres to current standards, ensuring up-to-date, credible, accurate, and evidence-based information—a critical aspect of medical education.[34] Our study findings may also offer valuable insights for app developers, particularly in the health care education sector, by highlighting the specific needs and preferences of medical students. As one of the primary users of medical apps, understanding the perspectives and requirements of medical students is crucial for enhancing the design, functionality, and overall user experience of these apps.

Our study presents several limitations. Firstly, the research was conducted exclusively among third-year medical students as part of a medical informatics course, potentially restricting the generalizability of our findings. Exploring the experiences and perspectives of medical students in other years could yield additional insights into future research. Secondly, our study focused on four specific medical apps, with each app evaluated by students based on their usage experience. Expanding the evaluation to include a broader range of medical apps in future research could enrich our findings. Notably, we did not collect data on the actual app usage patterns of the students. Although students were asked to evaluate the app they primarily used at the outset of the workshop, collecting detailed usage data could have potentially provided a more comprehensive understanding.

Conclusion

In the 21st century, utilizing medical apps has become an integral aspect of medical education. Equipping medical students with the ability to assess and choose medical apps that facilitate just-in-time learning is crucial and should be integrated into the medical curriculum. This study serves as an initial step to aid both medical students and faculty in selecting medical apps that enhance educational value using validated instruments. To enhance the usability and effectiveness of future and existing medical apps, it is crucial to address the identified challenges faced by medical students, including technical and financial obstacles that may restrict their usage. By addressing these issues, app developers can tailor solutions to better meet the needs of students, ultimately improving the overall educational experience and outcomes for medical students.

Clinical Relevance Statement

This study provides valuable guidance that can help faculty and students in selecting the most suitable apps for medical education.

Multiple-Choice Questions

1. When introducing medical education apps for education, it is important to consider which of the following?

   • The student's learning needs.

   • The faculty teaching needs.

   • The app's developers' perspectives.

   • The app's technical features.

   Correct Answer: The correct answer is option a. It is essential to consider the students' learning needs from medical education apps including required features and drawbacks that minimize the learning benefits.

2. Using medical education apps enhances the just-in-time learning experience. Which of the following defines just-in-time learning?

   • Problem-based learning methods

   • Time- and place-independent learning method

   • Team-based learning methods

   • Time- and place-dependent learning method

   Correct Answer: The correct answer is option b. Just-in-time learning is a method that allows medical students to access educational apps whenever and wherever they need them, facilitated by technology.

Conflict of Interest

None declared.

Acknowledgments

The authors extend their gratitude to the participating students and acknowledge the support of the Deanship of Scientific Research at KSU.

Protection of Human Subjects

The ethical approval was obtained from the Institutional Review Board at King Saud University, College of Medicine (IRB no.: E-23-7811). All protocols were carried out following relevant guidelines and regulations. All the participants voluntarily participated in this study and provided written informed consent.

^* These authors contributed equally to this work.

• References

• 1 Chandran VP, Balakrishnan A, Rashid M. et al. Mobile applications in medical education: a systematic review and meta-analysis. PLoS ONE 2022; 17 (03) e0265927
• 2 Murfin M. Know your apps: an evidence-based approach to evaluation of mobile clinical applications. J Physician Assist Educ 2013; 24 (03) 38-40
• 3 Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T 2014; 39 (05) 356-364
• 4 Markman TM, Sampognaro PJ, Mitchell SL, Weeks SR, Khalifian S, Dattilo JR. Medical student appraisal: applications for bedside patient education. Appl Clin Inform 2013; 4 (02) 201-211
• 5 Sampognaro PJ, Mitchell SL, Weeks SR. et al. Medical student appraisal: electronic resources for inpatient pre-rounding. Appl Clin Inform 2013; 4 (03) 403-418
• 6 Baig MM, GholamHosseini H, Connolly MJ. Mobile healthcare applications: system design review, critical issues and challenges. Australas Phys Eng Sci Med 2015; 38 (01) 23-38
• 7 Khalifian S, Markman T, Sampognaro P, Mitchell S, Weeks S, Dattilo J. Medical student appraisal: searching on smartphones. Appl Clin Inform 2013; 4 (01) 53-60
• 8 Gladman T, Tylee G, Gallagher S, Mair J, Grainger R. Measuring the quality of clinical skills mobile apps for student learning: systematic search, analysis, and comparison of two measurement scales. JMIR Mhealth Uhealth 2021; 9 (04) e25377
• 9 Hudder A, Tackett S, Moscatello K. First-year experience implementing an adaptive learning platform for first- and second-year medical students at the lake erie college of osteopathic medicine. J Am Osteopath Assoc 2019; 119 (01) 51-58
• 10 Golenhofen N, Heindl F, Grab-Kroll C, Messerer DAC, Böckers TM, Böckers A. The use of a mobile learning tool by medical students in undergraduate anatomy and its effects on assessment outcomes. Anat Sci Educ 2020; 13 (01) 8-18
• 11 Airth-Kindree N, Vandenbark RT. Mobile applications in nursing education and practice. Nurse Educ 2014; 39 (04) 166-169
• 12 Albrecht UV, Malinka C, Long S, Raupach T, Hasenfuß G, von Jan U. Quality principles of app description texts and their significance in deciding to use health apps as assessed by medical students: survey study. JMIR Mhealth Uhealth 2019; 7 (02) e13375
• 13 Jonas-Dwyer DRD, Clark C, Celenza A, Siddiqui ZS. Evaluating apps for learning and teaching. Int J Emerg Technol Learn 2012; 7 (01) 54-57
• 14 Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med 2014; 4 (04) 363-371
• 15 Eccles JS, Wigfield A. In the mind of the actor: the structure of adolescents' achievement task values and expectancy-related beliefs. Pers Soc Psychol Bull 1995; 21 (03) 215-225
• 16 Riel M. Education in the 21st Century: Just-in-time learning or learning communities‏. In: Education in the 21st Century: Just-in-Time Learning or Learning Communities in Emirates Center for Strategic Studies and Research (Ed) Education and the Arab World: Challenges of the next Millennium. Abu Dhabi: Emirates Center for Strategic Studies and Research; 1998: 137-160 . Accessed June 20, 2023 at: https://www.researchgate.net/profile/Margaret-Riel/publication/260992158_2000_Just_in_time_Learning_or_Learning_Communities/data/6035db5ba6fdcc37a8497e3d/2000-Just-in-time-UAE-Final.pdf
• 17 Naseem A, Ghias KKW. Just-in-time Teaching and Learning in Medical Education: Lessons from the Iterative Design Process. In: Proceedings of the Scholarship of Teaching and Learning Conference; 2017
• 18 Gladman T, Tylee G, Gallagher S, Mair J, Rennie SC, Grainger R. A tool for rating the value of health education mobile apps to enhance student learning (MARuL): development and usability study. JMIR Mhealth Uhealth 2020; 8 (07) e18015
• 19 Albarrak AI, Zakaria N, Almulhem J, Khan SA, Karim NA. Modified team-based and blended learning perception: a cohort study among medical students at King Saud University. BMC Med Educ 2021; 21 (01) 199
• 20 Aldekhyyel R, Almulhem J, Binkheder S. et al. Medical Apps and Perceptions of Use among Medical Students in Saudi Arabia. In: MEDINFO23; 2023
• 21 AMBOSS. AMBOSS: medical knowledge platform for doctors and students. Accessed January 24, 2024 at: https://www.amboss.com/us/students
• 22 Isabel. Isabel Clinical Educator | Teach Clinical Reasoning Skills. Accessed June 19, 2023 at: https://www.isabelhealthcare.com/products/isabel-clinical-educator
• 23 Medscape. Medscape. Accessed January 24, 2024 at: https://www.medscape.com/public/medscapeapp
• 24 Osmosis. Illustrated Video Explainers For Teaching Students | Osmosis. Accessed June 19, 2023 at: https://educators.osmosis.org/
• 25 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3 (02) 77-101
• 26 Kagkelaris K, Mastronikolis S, Amasiadi N. et al. University faculty on student motivation to use medical education apps and barriers to their usage. Eur Rev Med Pharmacol Sci 2023; 27 (03) 852-858
• 27 Zupanic M, Rebacz P, Ehlers JP. Media use among students from different health curricula: survey study. JMIR Med Educ 2019; 5 (02) e12809
• 28 Abdulkhaliq A, Stassos K, Haji-Hassan M, Bolboacă SD. Medical learning applications used by undergraduate medical students in Cluj-Napoca: a cross-sectional survey. Appl Med Informatics Orig Res. 2020; 42 (02) 83-94
• 29 Sayedalamin Z, Alshuaibi A, Almutairi O, Baghaffar M, Jameel T, Baig M. Utilization of smart phones related medical applications among medical students at King Abdulaziz University, Jeddah: a cross-sectional study. J Infect Public Health 2016; 9 (06) 691-697
• 30 Quant C, Altieri L, Torres J, Craft N. The self-perception and usage of medical apps amongst medical students in the United States: a cross-sectional survey. Int J Telemed Appl 2016; 2016: 3929741
• 31 Rusatira JC, Tomaszewski B, Dusabejambo V. et al. Enabling access to medical and health education in Rwanda using mobile technology: needs assessment for the development of mobile medical educator apps. JMIR Med Educ 2016; 2 (01) e7
• 32 Chase TJG, Julius A, Chandan JS. et al. Mobile learning in medicine: an evaluation of attitudes and behaviours of medical students. BMC Med Educ 2018; 18 (01) 152
• 33 UpToDate. Evidence-Based Clinical Decision Support System. Wolters Kluwer; . Accessed June 19, 2023 at: https://www.wolterskluwer.com/en/solutions/uptodate
• 34 Fineout-Overholt E, Stillwell SB, Williamson KM. Teaching evidence-based practice in academic settings. Evid Based Pract Nurs Healthc 2011; 306-307

Publication History

Received: 25 February 2024

Accepted: 18 June 2024

Article published online:
04 September 2024