Use of Statistical Techniques to Synthesize Explicit Criteria Developed by an Expert Panel

Summary

Objectives: Methodology based on expert panels has been commonly used to evaluate the appropriateness of interventions. An important issue is the adequate synthesis of the generated information in an applicable way to clinical decision making. This paper shows how statistical procedures help synthesize the results of an expert panel.

Methods: Three statistical techniques were applied to an expert panel that developed explicit criteria to assess the appropriateness of total hip joint replacement: classification tree, regression tree and multiple correspondence analysis combined with automatic classification.

Results: Results provided by the three models were shown in graphical displays and were compared to the original panel results using crude and weighted probability of misclassification. Results were also applied to real interventions in order to know the implication of the misclassification on real patients.

Conclusions: The statistical techniques help summarize data from panels of experts and provide useful decision models for clinical practice, especially when the number of indications is big. However, degree of misclassification and its implication should be taken into account.

Keywords

Appropriateness - statistical models - classification error - clinical guidelines

• References

• 1 Wennberg J. The paradox of appropriate care. JAMA 1987; 258: 2568-9.
• 2 Hadorn DC, Brook RH. The health care resource allocation debate. Defining our terms. JAMA 1991; 266: 3328-31.
• 3 Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care 1986; 2: 53-63.
• 4 Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, Kosecoff J, Solomon DH. Physician ratings of appropriate indications for six medical and surgical procedures. Am J Public Health 1986; 76: 766-72.
• 5 Gale RP, Park RE, Dubois RW, Herzing GP, Hocking WG, Horowitz MM, Keating A, Kenpin S, Linker CA, Schiffer CA, Wiernik PH, Weisdorf DJ, Rai KR. Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in chronic myelogenous leukemia in chronic phase. Leuk Res 1999; 23: 817-26.
• 6 Merrick NJ, Fink A, Park RE, Brook RH, Kosecoff J, Chassin MR, Solomon DH. Derivation of clinical indications for carotid endarterectomy. Am J Public Health 1987; 77: 187-90.
• 7 Wietlisbach V, Vader JP, Porchet F, Constanza MC, Burnand B. Statistical approaches in the development of clinical practice guidelines from expert panels. Med Care 1999; 37: 785-97.
• 8 Intrator O, Kooperberg C. Trees and splines in survival analysis. Stat Methods Med Res 1995; 4: 237-61.
• 9 Landrum MT, Normand SLT. Applying Bayesian ideas for the development of medical guidelines. Stat Med 1999; 18: 117-37.
• 10 Uebersax JS. Statistical modeling of expert ratings in medical treatment appropriateness. J Am Stat Assoc 1993; 88: 421-27.
• 11 Kattan MW, Hess KR, Beck JR. Experiments to determine whether Recursive Partitioning (CART) or an artificial neural network overcomes theoretical limitations of Cox proportional hazards regression. Comput Biomed Res. 1998; 31: 363-73.
• 12 Speroff T, Connors AF, Dawson NV. Lens model analysis of hemodynamic status in the critically ill. Med Decis Making 1989; 9: 243-52.
• 13 Benzécri JP. Statistical analysis as a tool to make patterns emerge from data. In Watanabe S. (ed.) Methodologies of Pattern Recognition. New York: Academic Press; 1969. pp 35-74.
• 14 Quintana JM, Arostegui I, Azkarate J, Goenaga JI, Elexpe X, Letona J, Arcelai A. Development of explicit criteria for total hip joint replacement. J Clin Epidemiol. 2000; 53: 1200-8.
• 15 Zhang H, Singer B. Recursive Partitioning in the Health Sciences. New York: Springer-Verlag; 1999
• 16 Segal MR. Extending the elements of tree-structured regression. Stat Methods Med Res 1995; 4: 219-36.
• 17 Venables WN, Ripley BD. Modern Applied Statistics with S-Plus. New York: Springer-Verlag; 1994
• 18 Greenacre M. Correspondence analysis in medical research. Stat Methods Med Res 1992; 1: 97-117.
• 19 Jambu M. Classification Automatique pour l’Analyse des Données. Tome 1: Méthodes, et algoritmes. Paris: Dunod; 1978
• 20 Ward JH. Hierarchical grouping to optimize an objective function. J Am Stat Assoc 1963; 58
• 21 SAS Institute Inc. SAS Procedures Guide, Version 6. Cary, NC: SAS Institute 1994
• 22 S-PLUS 4. Guide to Statistics. Data Analysis Products division. Seattle, WA: MathSoft, Inc.; 1997
• 23 Lébart L, Morineau A, Lambert T. SPAD. N: Systême Portable pour l’Analyse des Données. Manuel de réference. Sèvres: CISIA 1987
• 24 Faulkner A, Kennedy LG, Baxter K, Donovan J, Wilkinson M, Bevan G. Effectiveness of hip prostheses in primary total hip replacement: a critical review of evidence and economic model. Health Technol Assessment 1998; 2: 1-133.
• 25 Shekelle PG, Shriger DL. Evaluating the use of the appropriateness method in the Agency for Health Care Policy and Research Clinical Practice Guideline development process. Health Serv Res 1996; 31: 453-68.
• 26 Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med 1998; 338: 1888-95.
• 27 Leape LL, Park RE, Kahan JP, Brook RH. Group judgments of appropriateness: the effect of panel composition. Qual Assur Health Care 1992; 4: 151-9.
• 28 Bernstein SJ, Kosecoff J, Gray D, Hampton JR, Brook RH. The appropriateness of the use of cardiovascular procedures. British versus U.S. perspectives. Int J Technol Assess Health Care 1993; 9: 3-10.
• 29 Kahan JP, Park RE, Leape LL, Bernstein SJ, Hilborne LH, Parker L, Kamberg CJ, Ballard DJ, Brook RH. Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures. Med Care 1996; 34: 512-23.
• 30 Phelps CE. The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med 1993; 329: 1241-5.
• 31 Hicks NR. Some observations on attempts to measure appropriateness of care. BMJ 1994; 309: 730-3.
• 32 Phelps CE. Appropriateness studies. N Engl J Med 1994; 330: 433-4.
• 33 Schilling J, Faisst K, Kapetanios E, Wyss P, Norrie MC, Gutzwiller F. Appropriateness and necessity research on the internet: Using a “second opinion system”. Methods Inf Med 2000; 39: 233-7.
• 34 Naylor CD, Williams J. Primary hip and knee replacement surgery: Ontario criteria for case selection and surgical priority. Qual Health Care 1996; 5: 20-30.
• 35 McConnochie KM, Roghmann KJ, Pasternack J. Developing prediction rules and evaluating observation patterns using categorical clinical markers: two complementary procedures. Med Decis Making 1993; 13: 30-42.
• 36 Marshall RJ. The use of classification and regression trees in clinical epidemiology. J Clin Epidemiol 2001; 54: 603-9.
• 37 Leclerc A, Luce D, Lert F, Chastang JF, Logeay P. Correspondence analysis and logistic modeling: complementary use in the analysis of a health survey among nurses. Stat Med 1988; 7: 983-95.
• 38 Escofier B, Pagès J. Analyses factorielles simples et multiples. Paris: Dunod 1990
• 39 Maurer W. Creative and innovative statistics in clinical research and development. Methods Inf Med 2005; 44: 551-60.