Appl Clin Inform 2014; 05(03): 731-745
DOI: 10.4338/ACI-2014-03-RA-0021
Research Article
Schattauer GmbH

Evaluating a federated medical search engine

Tailoring the methodology and reporting the evaluation outcomes
D. Saparova
1  School of Information Science and Learning Technologies, University of Missouri, Columbia, MO, 65211
,
J. Belden
2  Department of Family and Community Medicine, University of Missouri , Columbia, MO 65212
,
J. Williams
3  MedSocket, Columbia, MO 65211
,
B. Richardson
1  School of Information Science and Learning Technologies, University of Missouri, Columbia, MO, 65211
,
K. Schuster
1  School of Information Science and Learning Technologies, University of Missouri, Columbia, MO, 65211
› Author Affiliations
Further Information

Correspondence to:

Dinara Saparova
111 London Hall, Columbia, MO 65211
Phone: (573) 884–2737   

Publication History

received: 26 March 2014

accepted: 02 July 2014

Publication Date:
19 December 2017 (online)

 

Summary

Background: Federated medical search engines are health information systems that provide a single access point to different types of information. Their efficiency as clinical decision support tools has been demonstrated through numerous evaluations. Despite their rigor, very few of these studies report holistic evaluations of medical search engines and even fewer base their evaluations on existing evaluation frameworks.

Objectives: To evaluate a federated medical search engine, MedSocket, for its potential net benefits in an established clinical setting.

Methods: This study applied the Human, Organization, and Technology (HOT-fit) evaluation framework in order to evaluate MedSocket. The hierarchical structure of the HOT-factors allowed for identification of a combination of efficiency metrics. Human fit was evaluated through user satisfaction and patterns of system use; technology fit was evaluated through the measurements of time-on-task and the accuracy of the found answers; and organization fit was evaluated from the perspective of system fit to the existing organizational structure.

Results: Evaluations produced mixed results and suggested several opportunities for system improvement. On average, participants were satisfied with MedSocket searches and confident in the accuracy of retrieved answers. However, MedSocket did not meet participants’ expectations in terms of download speed, access to information, and relevance of the search results. These mixed results made it necessary to conclude that in the case of MedSocket, technology fit had a significant influence on the human and organization fit. Hence, improving technological capabilities of the system is critical before its net benefits can become noticeable.

Conclusions: The HOT-fit evaluation framework was instrumental in tailoring the methodology for conducting a comprehensive evaluation of the search engine. Such multidimensional evaluation of the search engine resulted in recommendations for system improvement.

Citation: Saparova D, Belden J, Williams J, Richardson B, Schuster K. Evaluating a federated medical search engine: Tailoring the methodology and reporting the evaluation outcomes. Appl Clin Inf 2014; 5: 731–745

http://dx.doi.org/10.4338/ACI-2014-03-RA-0021


#

 


#

Conflicts of interest

One of the authors holds a position at MedSocket. This author’s contribution was limited to literature review, descriptions of MedSocket features and functionality, and discussion.


Correspondence to:

Dinara Saparova
111 London Hall, Columbia, MO 65211
Phone: (573) 884–2737