Download PDF
Methods Inf Med 2006; 45(03): 267-274
DOI: 10.1055/s-0038-1634078
Original Article
Schattauer GmbH

Quality Assurance of Medical Ontologies

J. E. Rogers
1   BioHealth Information Group, School of Computer Science, University of Manchester, Manchester, United Kingdom
› Author Affiliations
Further Information

Publication History

Publication Date:
06 February 2018 (online)

    Permissions and Reprints

Summary

Objective: To review the literature concerning the quality assurance of medical ontologies.

Methods: scholar.google.com was searched using the search strings (+ontology +”quality assurance”) and (+ontology +”evaluation/evaluating”). Relevant publications were selected by manual review. Other work already familiar to the author, or suggested by other researchers contacted by the author, were included. The papers were analysed for common themes.

Results: Four broad properties of an ontology were identified that may be quality-assured: philosophical validity, compliance with meta-ontological commitments, ‘content correctness’, and fitness for purpose. Each published methodology addressed only a subset of these properties. ‘Content’ may be divided into domain knowledge content, and metadata describing either the provenance of domain knowledge content, or relationships between it and lexical information (e.g. for display and retrieval). ‘Correctness’ (whether of domain knowledge content or metadata) may also be further subdivided into truth, completeness, parsimony and internal consistency.

Conclusions: Understanding of how to assure the quality of ontologies, or evaluate their fitness for specific purposes, is improving but remains poor. A combination of methodologies is required, but tools to support a comprehensive quality assurance programme remain lacking.

Perfect quality of an ontology is not provable and may not be desirable: an ontology compliant with all current philosophical theories, following necessary ontological commitments, and with entirely ‘correct’ content, may be too complex to be directly usable or useful.

The extent to which an ontology’s fitness for purpose is predicted or influenced by its other properties remains to be determined. Field studies of ontologies in use, including interrater effects, are required.

Keywords

Terminology - quality control - information systems
 

  • References

  • 1 Borgida A, Patel-Schneider PF. A Semantics and Complete Algorithm for Subsumption in the CLASSIC Description Logic. J Art Intell Research 1994; 1: 277-308.
    PubMedGoogle Scholar
  • 2 Borgida A. On the relative expressive power of Description Logics and Predicate Calculus. Artificial Intelligence 1996; 82: 353-67.
    CrossrefPubMedGoogle Scholar
  • 3 Rogers JE, Roberts A, Solomon WD, van der Haring E, Wroe CJ, Zanstra PE, Rector AL. GALEN Ten Years On: Tasks and Supporting tools. In Patel V. et al (eds). Proc MEDINFO 2001. IOS Press; 2001: 256-60.
    Google Scholar
  • 4 Bada M, McEntire R, Wroe C, Stevens R. GOAT: The Gene Ontology Annotation Tool. In: Proc 2003 UK e-Science All Hands Meeting. Nottingham, UK 514-9.
    PubMedGoogle Scholar
  • 5 Baader F, Calvanese D, McGuinness D, Nardi D, Patel-Schneider P. The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press 2003. ISBN: 0521781760
    PubMedGoogle Scholar
  • 6 Spackman KA. Normal forms for description logic expressions of clinical concepts in SNOMED RT. Proc AMIA Symp 2001; 627-31.
    PubMedGoogle Scholar
  • 7 Spackman KA, Dionne R, Mays E, Weis J. Role grouping as an extension to the description logic of Ontylog, motivated by concept modeling in SNOMED. Proc AMIA Symp 2002; 712-6.
    PubMedGoogle Scholar
  • 8 Rogers JE, Wroe CJ, Roberts A, Swallow A, Stables D, Cantrill JA, Rector AL. Automated quality checks on repeat prescribing. Br J Gen Pract 2003; 53 (496) 838-44.
    PubMedGoogle Scholar
  • 9 Wroe CJ, Stevens R, Goble CA, Ashburner M. A Methodology to Migrate the Gene Ontology to a Description Logic Environment Using DAML+OIL. Pacific Symposium on Biocomputing 2003; 8: 624-35.
    PubMedGoogle Scholar
  • 10 Rogers J, Price C, Rector A. et al Validating Clinical Terminology Structures: Integration and Cross-Validation of Read Thesaurus and GALEN. In: Proc AMIA Symp Orlando FL 1998. Philadelphia, PA: Hanley & Belfus Inc.; 1998: 845-9.
    Google Scholar
  • 11 Rogers JE. Development of a methodology and an ontological schema for medical terminology. MD Thesis,. University of Manchester 2005 http://www.cs.man.ac.uk/mig/people/jeremy/papers/2004-Rogers-MD-Thesis.pdf (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 12 Gruber TR. Toward Principles for the Design of Ontologies Used for Knowledge Sharing. Int Journal of Human-Computer Studies 1993; 43: 907-28.
    PubMedGoogle Scholar
  • 13 Chandrasekaran B, Josephson JR, Benjamins VR. What are ontologies, and why do we need them?. IEEE Intelligent Systems 1999; 14: 20-6.
    PubMedGoogle Scholar
  • 14 Cimino JJ. Desiderata for Controlled Medical Vocabularies in the Twenty-First Century. Methods Inform Med 1998; 37: 394-403.
    PubMedGoogle Scholar
  • 15 Goble C, Wroe C. The Montagues and the Capulets. Comp Funct Genom 2004; 5: 623-32.
    CrossrefPubMedGoogle Scholar
  • 16 Baader F. Restricted Role-value-maps in a Description Logic with Existential Restrictions and Terminological Cycles. CEUR Workshops Proc Description Logics 2003 http://sunsite.informatik.rwth-aachen.de/Publications/CEURWS/Vol-81/ (visited Aug 9, 2005)
    PubMed
  • 17 Schulz EB, Barrett JW, Price C. Read Code Quality Assurance: From Simple Syntax to Semantic Stability. J Am Med Inform Assoc 1998; 5 (04) 337-46.
    CrossrefPubMedGoogle Scholar
  • 18 Gómez-Pérez A. From Knowledge Based Systems to Knowledge Sharing Technology: Evaluation and Assessment. Knowledge Systems Laboratory Abstract KSL-94–73 (1994) http://www.ksl.stanford.edu/KSL_Abstracts/KSL94-73.html (visited Aug 9, 2005)
    Google Scholar
  • 19 Gómez-Pérez A. Evaluation of Ontologies. Int J Intell Systems 2001; 16: 391-409.
    CrossrefPubMedGoogle Scholar
  • 20 Fernández-López M, Gómez-Pérez A. A survey on methodologies for developing, maintaining, evaluating and reengineering ontologies. Ontoweb consortium: Deliverable 1.4 2002. http://ontoweb.org/About/Deliverables/D1.4-v1.0.pdf (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 21 Schulz EB, Barrett JW, Price C. Semantic quality through semantic definition: Refining the Read Codes through internal consistency. J Am Med Inform Assoc 1997; 4 (Symp Supp) 615-9.
    PubMedGoogle Scholar
  • 22 Campbell KE, Tuttle MS, Spackman KA. A “lexically- suggested logical closure” metric for medical terminology maturity. Proc AMIA Symp Philadelphia: Hanley and Belfus; 1998: 785-9.
    Google Scholar
  • 23 Spackman KA, Campbell KE. SNOMED RT: A Reference Terminology for Health Care. Proc AMIA Symp 1997; 640-4.
    PubMedGoogle Scholar
  • 24 Chute CG, Cohn SP, Campbell KE, Oliver DE, Campbell JR. The content coverage of clinical classifications. J Am Med Inform Assoc 1996; 3: 224-33.
    CrossrefPubMedGoogle Scholar
  • 25 Brown PJB, Odusanya L. Does Size Matter? – Evaluation of Value Added Content of Two Decades of Successive Coding Schemes in Secondary Care. In Bakken S. (ed). Proc AMIA Symp Philadelphia: Hanley and Belfus 2001; 71-5.
    PubMedGoogle Scholar
  • 26 Brown PJB, Warmington V, Laurence M, Prevost AT. Randomised crossover trial comparing the performance of Clinical Terms Version 3 and Read Codes 5 byte set coding schemes in general practice. BMJ 2003; 326: 1127
    CrossrefPubMedGoogle Scholar
  • 27 Rogers JE, Wroe CJ, Roberts A, Rector AL, Swallow A, Stables D, Cantrill JA. Feasibility and Reliability of Computerised Review of Long Term Prescribing: Final Report (2002). http://www.cs.man.ac.uk/mig/projects/old/prescribingindicators/FinalReportRTF.zip Supplementary Report 88-96. (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 28 Ceusters W, Smith B, Kumar A, Dhaen C. Ontology- based error detection in SNOMED-CT. In: Fieschi, et al. (eds). Proc MEDINFO 2004 IOS Press 2004; 482-6.
    PubMedGoogle Scholar
  • 29 Bodenreider O, Smith B, Kumar A, Burgun A. Investigating subsumption in DL-based terminologies: A Case Study in SNOMED CT. In: Proc KR-MED 2004. Whistler 12-20. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS//Vol-102/ (visited Aug 15, 2005)
    PubMedGoogle Scholar
  • 30 Cornet R, Abu-Hanna A. Description logic based methods for auditing frame-based medical terminology systems. Artificial Intelligence in Medicine 2005; 34: 201-17.
    CrossrefPubMedGoogle Scholar
  • 31 Arts DGT, Cornet R, de Jonge E, de Keizer NF. Methods for evaluation of medical terminological systems; a literature review and a case study. Methods Inf Med 2005; 44: 616-25.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 32 ISO/TC215 WG 3 Standard Specification for Quality Indicators for Controlled Health Vocabularies 2000 July Report No.: TS17117.
    PubMedGoogle Scholar
  • 33 Gómez-Pérez A. Some Ideas and Examples to Evaluate Ontologies. Knowledge Systems Laboratory Abstract KSL-94-65 (1994) http://www.ksl.stanford.edu/KSL_Abstracts/KSL-94-65.html (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 34 Gómez-Pérez A. Criteria to Verify Knowledge Sharing Technology. Knowledge Systems Laboratory Abstract KSL-94-65 (1995) http://www.ksl.stanford.edu/KSL_Abstracts/KSL-95-10.html (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 35 Uschold M. Building Ontologies: towards a unified methodology. In Proc Expert Systems. 16th Annual Conference of BCS SG on Expert Systems 1996
    PubMedGoogle Scholar
  • 36 Grüninger M, Fox MS. Methodology for the Design and Evaluation of Ontologies. In: IJCAI’95, Workshop on Basic Ontological Issues in Knowledge Sharing, April 13 1995
    PubMedGoogle Scholar
  • 37 Masolo C, Borgo S, Gangemi A, Guarino N, Oltramari A. Wonderweb: Deliverable 18. Ontology Library. WonderWeb consortium (2003) http://wonderweb.semanticweb.org/deliverables/D18.shtml (visited Aug 9, 2005)
    PubMedGoogle Scholar
  • 38 Grenon P, Smith B, Goldberg L. Biodynamic Ontology: Applying BFO in the Biomedical Domain. In: Pisanelli (ed). Ontologies in Medicine. Amsterdam: IOS Press 2004; 20-38.
    PubMedGoogle Scholar
  • 39 Donnelly M, Bittner T, Rosse C. A Formal Theory for Spatial Representation and Reasoning in Biomedical Ontologies. AI Med 2005; 36 (01) 1-27.
    PubMedGoogle Scholar
  • 40 Donnelly M. Containment Relations in Anatomical Ontologies In Friedman C, Ash J, Tarczy Honoch. (eds). Proc AMIA 2005 Annual Symposium Washington DC. 206-10.
    PubMedGoogle Scholar
  • 41 Guarino N, Welty C. Evaluating ontological decisions with OntoClean. Communications of the ACM 2002; 45 (02) 61-5.
    CrossrefPubMedGoogle Scholar
  • 42 Gangemi A, Guarino N, Masolo C. Sweetening ontologies with DOLCE. In: Gomez-Perez, Benjamins (eds). EKAW 2002. Proc 13th Int Conf Knowl Eng & Knowl Mangmt. Ontologies and the Semantic Web Heidelberg: Springer Verlag; 2003: 166-81.
    Google Scholar
  • 43 Flett A, Casella dos Santos M, Ceusters W. Some Ontology Engineering Processes and Their Supporting Technologies. In Gomez-Perez Benjamins. (eds). EKAW 2002. Proc 13th Int Conf Knowl Eng & Knowl Mangmt. Ontologies and the Semantic Web. Heidelberg: Springer Verlag; 2003: 154-65.
    Google Scholar
  • 44 Casalla de Santos M, Dhaen C, Fielding M, Ceusters W. Philosophical scrutiny for run-time support of application ontology development. In: Proc FOIS 2004, Turin
    PubMedGoogle Scholar
  • 45 Fielding JM, Simon J, Ceusters W, Smith B. Ontological theory for ontological engineering: biomedical systems information integration. In: Principles of Knowledge Representation and Reasoning. Proc KR2004. Whistler: AAAI Press; 2004: 116647-52.
    Google Scholar
  • 46 Welty CA, Mahindru R, Chu-Carroll J. Evaluating Ontological Analysis. In: Proc ISWC-03 2003. CEUR-WS vol 82. http://CEUR-WS//Vol-82 (visited Aug 9 2005)
    Google Scholar
  • 47 Brewster C, Alani H, Dasmahapatra S, Wilks Y. Data Driven Ontology Evaluation. In: Proc Int Conf Language Resources and Evaluation. Lisbon, Portugal 2004
    PubMedGoogle Scholar
  • 48 Hartman J, Spyns P, Giboin A, Maynard D, Cuel R, Suárez-Figueroa C, Sure Y. KnowledgeWeb D1.2.3 Methods for Ontology Evaluation. (2005) http://www.starlab.vub.ac.be/research/projects/knowledgeweb/KWeb-Del-1.2.3-Revised-v1.3.1.pdf (visited Aug 9, 2005)
    PubMed
  • 49 ECOR European Centre for Ontological Research http://www.ecor.uni-saarland.de (visited Aug 9, 2005)
    PubMed
  • 50 Smith B, Ceusters W, Temmerman R. Wüsteria. In: Proc MIE2005, Geneva. Stud Health Technol Inform 2005; 116: 647-52.
    PubMedGoogle Scholar
  • 51 O’Neil M, Payne C, Read J. Read Codes Version 3: A user led terminology. Methods Inf Med 1995; 34: 187-92.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 52 Spackman KA, Reynoso G. Examining SNOMED from the Perspective of Formal Ontological Principles: Some Preliminary Analysis and Observations. in Proc KR-MED 2004 Whistler 12-20. http://sunsite.informatik.rwthaachen. de/Publications/CEUR-WS//Vol-102/ (visited Aug 15 2005)
    PubMedGoogle Scholar
  • 53 Kitchenham B, Pfleeger SL. Software Quality: The Elusive Target. IEEE Software 1996; 13 (01) 12-21.
    PubMedGoogle Scholar
  • 54 Kitchenham B, Jones L. Evaluating software engineering methods and tool (Twelve part series). ACM SIGSOFT Software Engineering Notes 1996-8; 21:1;11-14 to 23 5: 21-24.
    PubMedGoogle Scholar
  • 55 CAP (College of American Pathologists) SNOMED Clinical Terms User Guide. July 2004 Release
    PubMedGoogle Scholar
  • 56 Friedland NS, Allen PG, Witbrock M. et al Towards a quantitative, platform-independent analysis of knowledge systems. In: Principles of Knowledge Representation and Reasoning. Proc KR2004, Whistler. AAAI Press 2004; 116: 507-15.
    PubMedGoogle Scholar
  • 57 Fernández-López M, Gómez-Pérez A. The integration of OntoClean in WebODE. In: CEUR Workshop Proceedings. Amsterdam, The Netherlands 2002 62. 38-52. http://CEUR-WS.org/Vol-62 (visited Aug 9, 2005)
    PubMedGoogle Scholar