Journal of Clinical & Academic Ophthalmology 2016; 08(01): e1-e9
DOI: 10.1055/s-0036-1572409
Editorial
Thieme Medical Publishers 333 Seventh Avenue, New York, NY 10001, USA.

Criteria for Evaluating Ophthalmology Departments Based on the U.S. News & World Report Ranking System

Warren J. Scherer
1   Department of Ophthalmology, Eye and Ear Institute, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania
2   Newsom Eye & Laser Center, Tampa, Florida
,
Sandra Danoff
3   Strategy and Special Projects, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
,
Joel S. Schuman
1   Department of Ophthalmology, Eye and Ear Institute, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania
4   Department of Bioengineering, University of Pittsburgh, Pittsburgh, Pennsylvania
› Author Affiliations
Further Information

Address for correspondence

Warren J. Scherer, MD, PhD
Newsom Eye & Laser Center
13904 N. Dale Mabry Hwy, Suite 200
Tampa, FL 33618

Publication History

Publication Date:
26 February 2016 (online)

 

Abstract

Purpose To propose a methodology for ranking U.S. clinical ophthalmology programs which utilizes an existing framework of both objective structural (or resource) information and outcome data as well as subjective reputational scores currently used by U.S. News & World Report for most clinical specialties.

Design Evidence-based manuscript.

Topic The U.S. News & World Report annual “Best Hospital Rankings by Specialty” provides the public with information regarding the relative strengths of U.S. medical and surgical departments in providing care and treatments for patients with complex diseases. For most medical and surgical specialties, these rankings are determined by considering a combination of objective structural (or resource) information and outcome data along with subjective reputational scores.

Clinical Relevance In ophthalmology, U.S. News & World Report annual rankings are currently generated only from subjective reputational scores submitted by a small number of voting ophthalmologists. No objective clinical or outcome data are considered.

Literature Reviewed Methodology for determining “Best Hospital Rankings by Specialty” utilized by RTI International on behalf of U.S. News & World Report.

Results The ranking system of U.S. departments of ophthalmology presently employed by U.S. News & World Report presents an incomplete picture of the relative strengths and weaknesses of the departments listed.

Conclusion With the changes in health insurance that increasingly shift decision-making responsibility and cost to consumers, providing mechanisms to allow the public to make informed decisions is an emerging imperative for ophthalmology and other specialties. Revision of the current “reputation-only” ranking system used for ophthalmology to also include objective clinical and outcome data would benefit patients by providing more accurate and impartial information on which to base their eye care decisions.


#

Summary of the Current U.S. News & World Report Ranking System and the Index of Hospital Quality

In 1990, the news magazine U.S. News & World Report first published its “Best Hospitals” list. The institutions included in this initial report were listed in alphabetical order. Beginning in 1993, U.S. News & World Report established a ranking system for hospitals within each specialty. It stated that the rationale underlying the ranking system was to help patients determine which institutions provided the best care for the most serious or complicated medical conditions. This is a worthy objective. For most common conditions and procedures, there are many excellent programs that are convenient to consumers and demonstrate good outcomes; however, increasingly major insurers and employers are recognizing that, for certain highly specialized procedures, high-volume programs and/or narrowly defined networks with specific expertise offer better “value” as measured in outcomes and cost.[1] [2] [3] [4] [5] [6]

Factors contributing to good outcomes in relatively uncommon and specialized procedures include but are not limited to significant clinical experience, appropriate technology, and specialized postsurgical nursing. However, the availability of such information to both consumers and referring physicians is limited at best, particularly for procedures where a bad outcome is not death but some other major disability or limitation.

U.S. News & World Report has attempted to fill the void, and the national ranking of 16 adult medical specialty areas, as well as a “Hospital Honor Roll” of institutions that have at least 6 of 16 specialties ranked at least 3 standard deviations above the mean, is aimed at providing a high degree of transparency in its annual publication.

The U.S. News & World Report ranking of best hospitals has grown to carry considerable influence among health care providers, policy makers, and patients, both domestic and international.

Among professionals, ranking of a hospital can be used to retain or attract physicians. Politically, favorable ranking can provide a hospital with an edge in receiving public funds. With regard to patients, it has been shown that choice of hospital is based on final ordinal rank lists and other shortcuts rather than the evaluation of “complex” objective quality scores.[7] It was also found that patients remember relative hospital rankings for as long as 2 years after they were initially published.[8] From an economic standpoint, a representative study of California hospitals found that if accompanying objective quality scores are flexibly controlled, the average hospital experiences a 5% change in patient volume (and revenue stream) from year to year based solely on rank changes. This translates into the transfer of more than 15,000 Medicare patients and 750 million dollars from lower to higher ranked hospitals over the past 10 years.[7]

Beginning in 2005, the methodology, data compilation, and reporting for the U.S. News & World Report ranking system have been performed by the Social, Statistical, and Environmental Sciences Division of RTI International.[9] For 12 of the 16 adult specialties, rankings are based upon a composite score, the Index of Hospital Quality (IHQ). The IHQ is a reflection of three components of health care: structure, process, and outcomes. In the health care setting, “structure” refers to resources used to directly provide patient care, such as staffing, availability of key technologies, and other patient services. The “process” through which health care is delivered is defined by accurate diagnosis, treatment, disease prevention, and patient education. Because this facet of care is difficult to measure on a national level, RTI International uses a survey-based physician vote, that is, a “best of” list, as a proxy for the entire “process” component. This process is aimed at tapping the current knowledge of practicing physicians as to programs that have the expertise, volume, technology, and nursing excellence to warrant inclusion on the national listing of where they would send a patient if distance was not a barrier. The “outcomes” component is scored based upon risk-adjusted mortality rates within each specialty. Additionally, a new measure related to patient safety and medical errors was added in 2009. The weighting of each component in calculating the IHQ is as follows: structure, 30.0%; process, 32.5%; outcomes, 32.5%; and patient safety, 5.0%.


#

Reputation-Only Ranking and the Rationale for Its Abandonment in Ophthalmology

Rank scores for 4 of the 16 specialties—ophthalmology, psychiatry, rehabilitation, and rheumatology—are based solely on the reputational physician survey used to evaluate the process component of the IHQ. For each of these four specialties, 200 U.S. physicians in private or academic practice from each specialty are mailed blank ballots via U.S. mail (50 each in the Northeast, Southeast, Midwest, and Western census areas) on which to nominate, in order, the top five U.S. institutions that they consider to be the best in their respective fields. Two additional reminder ballots are sent to nonrespondents. An institution is eligible for ranking if it receives one or more physician nominations. Each year's rank list consists of a rolling average of the most recent 3 years of balloting.

Unfortunately, within the U.S. News & World Report methodology, it has been shown that the correlation of the current reputational process scores to objective measure scores is not significantly different from zero, indicating that no relationship exists between them.[10] Because the economic and reputational stakes of overall rank position have become so important, the methodology underlying the use of subjective, reputation-only ranking should be closely reviewed.

Among the issues that warrant review are the size of the survey sample and the response rate. The current methodology relies on a sample of 200 practicing board-certified physicians per specialty. This number is reduced further in ophthalmology when one considers the ∼50% overall response rate of ophthalmologists. There are currently ∼20,000 ophthalmologists in the United States. This translates to ∼0.5% of all practicing ophthalmologists within any given year, and if the results are expanded to include the 3-year rolling average, the electorate is representative of only 1.5% of ophthalmologists. With regard to sample number and reputation rankings, it was determined in a study by Teasley that an n of 100 has a sample error >8% for ranking purposes.[11] In that study, ranking specialties based only on reputation scores was impossible because the sample error was greater than the actual differences between the reputation scores. Aggregating the results of 3 years of voting increased the n and reduced the sample error to ∼5%; however, this error level was still not low enough to discriminate between ranks.[11] In a study examining the methodology of ranking of U.S. medical schools, McGaghie and Thompson indicate that survey response rates of 50 to 60% are barely acceptable scientifically and require additional information that contributes to the confidence of the data, whereas response rates of 50% or less are unreliable and not scientifically acceptable.[12] The implication is that the methodology, as currently designed, does not provide an adequate level of discrimination to fulfill the stated objective of identifying which institutions provided the best care for the most serious or complicated medical conditions.

The second reason is that small sample–based reputation-only ranking become self-fulfilling prophesies.[11] This result is related to the social psychological theories of “cue-taking,” “informational cascades,” and “political bandwagons.”[13] [14] [15] In “cue-taking,” an individual's beliefs, usually about a candidate, are influenced by the decisions of others. An informational cascade occurs when an individual, having observed the actions of those ahead of him, follows the behavior of the preceding individual without regard to his own information. In their model, Bikhchandani et al found that individuals rapidly converge on one action on the basis of very little information. They found that once started, an informational cascade will last forever and all subsequent decisions will be rendered uninformative.

Within ophthalmology, all of these factors are contributory to the observed results of reputation-only ranking. Most ophthalmologists have read previous rank lists and are very aware of the likely institutions listed in the top 10 (i.e., the choices of their predecessors). Most of these programs have remained in the top 10 of the U.S. News & World Report rankings since their inception in 1993. Given the paucity of readily available objective data, it is likely that voting ophthalmologists have very limited objective information, other than the choices of these early predecessors, to distinguish between ranked and unranked institutions. McGaghie and Thompson refer to the socially stratifying “principle of cumulative advantage,” as applied to institutions, that “allows the rich to get richer at a rate that makes the poor become relatively poorer” to help explain ranking surveys.[12]


#

The Index of Eye Center Quality

Index of Eye Center Quality

RTI International states in its Rankings Methodology Report that reputation-only ranking for ophthalmology was initiated because structural and outcomes measures were not applicable because ophthalmology was primarily an outpatient specialty with almost no mortality. The present report establishes a set of ophthalmology-specific structural, outcome, and patient safety components that can be used, in addition to the existing process component, to calculate a composite score, the Index of Eye Center Quality (IECQ). The IECQ can be utilized to objectively rank U.S. ophthalmology programs within the methodology established by RTI International.


#

Eligibility

All U.S. ophthalmology programs are considered eligible to participate in the ranking process and would not need to submit an application. Two stages of eligibility criteria are used and each ophthalmology program must satisfy the requirements of both stages to remain eligible for inclusion.


#

Stage 1: Eligibility Criteria

An ophthalmology program must meet all of the following criteria to be considered eligible:

  • Medical school affiliation (American Medical Association or American Osteopathic Association)

  • Have an Accreditation Council for Graduate Medical Education (ACGME)–accredited residency training program (probational status of the program would deny eligibility)

  • Have at least 6 of 14 key technologies and advanced ophthalmic services available on-site. Assignment of partial credit for off-site availability (see Key Ophthalmic Technologies and Advanced Ophthalmic Services under Structure)


#

Stage 2: Eligibility Criteria

Further evaluation of eligibility will be based on the performance of a specified number of representative surgical procedures in various subspecialty areas of ophthalmology. Possible procedures to consider including for eligibility are listed in [Table 1]. Surgical procedures will be identified by their Current Procedural Terminology (CPT) code. For each representative procedure, the ratio of the number of procedures performed to clinical patient volume within the respective subspecialty clinic from which that patient originated will be calculated for each U.S. ophthalmology programs. For example, if the glaucoma specialists at University X perform 75 trabeculectomies per year and the annual glaucoma clinic volume per year is 4,000 visits, the trabeculectomy ratio for University X would be 75/4,000 = 0.019. The median of this ratio for all U.S. programs will then be multiplied by the respective clinic patient volume from each institution to determine the minimum number of surgeries required for eligibility of that institution. If, at a given institution, more than one subspecialty division performs a certain procedure (e.g., the cornea and comprehensive services may both perform cataract extraction), the number of cases performed and patient visits from each division performing the procedure would be combined in calculating the ratio. To simplify calculations, combined procedures would be excluded for consideration of eligibility. Because the U.S. News & World Report methodology focuses on adult care, pediatric procedures would be excluded.

Table 1

Representative ophthalmic surgical procedures

Surgical procedure CPT code

• Extracapsular cataract removal with insertion of intraocular lens prosthesis: 66984

• Retinal detachment repair: 67108

• Trabeculectomy: 66170

• Aqueous shunt to external reservoir/with sclera graft: 66180/67255

• Penetrating keratoplasty/in pseudophakia/in aphakia: 65730/65755/65750

The RTI International methodology for the IHQ specifies a minimum number of hospital discharges to ensure that eligible hospitals had experience in treating “complex cases.” When the U.S. News & World Report ranking system was initiated in 1993, the concept of complex cases was not a recognized entity for ophthalmology. In recent years, the Center for Medicare & Medicaid Service (CMS) has recognized the concept of “complex cases” for common ophthalmic surgical procedures such as cataract (2003), retinal detachment repair (2008), and trabeculectomy (2003). It has established CPT codes that specifically define what constitutes a complex case in each instance ([Table 2]). Additionally, CMS has also provided estimates of the percentages of cases that would be expected to meet the criteria to be considered complex. The establishment of minimum numbers of surgical cases as eligibility criteria would therefore assure that ophthalmology departments had sufficient experience in treating “complex cases.”

Table 2

Examples of complex ophthalmic surgical procedures and their definitions

Extracapsular cataract removal with insertion of intraocular lens prosthesis (CPT 66982): Manual or mechanical technique (e.g., irrigation and aspiration or phacoemulsification), complex, requiring devices or techniques not generally used in routine cataract surgery (e.g., iris expansion device, suture support for intraocular lens, or primary posterior capsulorrhexis). Includes patients who present with diseased states, prior intraocular surgery, or with dense, hard, and/or white cataracts

Retinal detachment repair (CPT 67113): Repair of complex retinal detachment (e.g., proliferative vitreoretinopathy, stage C-1 or greater, diabetic traction retinal detachment, retinopathy of prematurity, retinal tear of greater than 90 degrees), with vitrectomy and membrane peeling, may include air, gas, or silicone oil tamponade, cryotherapy, endolaser photocoagulation, drainage of subretinal fluid, scleral buckling, and/or removal of lens

Trabeculectomy (CPT 66172): Trabeculectomy ab externo with scarring from previous ocular surgery or trauma (includes injection of antifibrotic agents)


#

Structure

The ranking methodology of U.S. News & World Report defines structure as “the tools, human and otherwise, available at hospitals for treating patients.” This definition is readily applicable to ophthalmology. The following proposal will present ophthalmology-based criteria for generating the structural component of the IECQ. The structural component will contribute 35% toward the IECQ.


#

Key Ophthalmic Technologies and Advanced Ophthalmic Services

Over the past 20 years, technological advancements have revolutionized the practice of ophthalmology. In addition to possessing devices required for providing outstanding care, an eye center must also distinguish itself by providing state-of-the-art advanced ophthalmic services to treat highly complex ocular conditions. Eye institutions would receive one point for each of the available technologies and advanced ophthalmic services performed by a full-time faculty member on-site. Institutions would be awarded partial credit, 0.5 points, for a technology or service performed by a full-time faculty member at an off-site location. No credit would be provided for a service provided by a physician who is not a full-time faculty member. A listing and brief description of each technology and service is provided in [Table 3].

Table 3

Key ophthalmic technologies and advanced ophthalmic services

Excimer laser: A type of ultraviolet laser which is used for reshaping the corneal surface during refractive surgery such as laser in situ keratomileusis (LASIK) or photorefractive keratectomy (PRK). It may also be used for phototherapeutic keratectomy (PTK) to remove corneal scar tissue

Femtosecond laser keratome: A device used to cut a corneal flap for LASIK. The device employs a laser that produces pulses of light of extremely short duration. Its advantage over a metal keratome is that there is no risk of irregular or abnormal flap creation

Spectral domain optical coherence tomography: An optical signal acquisition and processing method. It captures micrometer-resolution, three-dimensional images from within optical scattering media (e.g., biological tissue). In ophthalmology, it is used for three-dimensional visualization and analysis of the optic nerve and macula

Digital fluorescein angiography: A technique for examining the circulation of the retina or iris using a dye tracing method that involves injection of sodium fluorescein into the systemic circulation. An angiogram is obtained by digital imaging of the fluorescence emitted after illumination of the retina with blue light at a wavelength of 490 nm

Confocal microscopy: An optical imaging technique used to increase optical resolution and contrast of a micrograph by using point illumination and a spatial pinhole to eliminate out-of-focus light in specimens that are thicker than the focal plane. It enables the reconstruction of three-dimensional structures from the obtained images. In ophthalmology, it is useful for imaging corneal pathology and infection

Updated phacoemulsification technology: Phacoemulsification is a fundamental technique used for cataract extraction in which the eye's internal lens is emulsified by an ultrasonic handpiece and aspirated from the eye. Aspirated fluids are replaced with irrigation of balanced salt solution, thus maintaining the anterior chamber, as well as cooling the handpiece. In recent years, new advances in technology have resulted in safer and more efficient phacoemulsification devices. Institutions would receive points for acquiring new phacoemulsification technology within the past 4 years

Descemet Stripping Automated Endothelial Keratoplasty (DSAEK): A technique used to replace the diseased endothelial cells of the cornea. The diseased cells are peeled away and a donor sheet of endothelial cells on their basement membrane (Descemet membrane) are inserted into the eye through a small incision and positioned into place. DSAEK involves less risk and faster visual rehabilitation than a full-thickness corneal transplant

Keratoprosthesis: A surgical procedure where a severely damaged or diseased cornea is replaced with a synthetic cornea. While conventional cornea transplant uses human donor tissue for transplant, an artificial cornea is used in the keratoprosthesis procedure. The surgery is performed to restore vision in patients suffering from a severely damaged cornea due to congenital birth defects, infections, injuries, and burns

Ocular oncology: This refers to a division within the eye institution that specializes in the diagnosis and treatment (medically and surgically) of ocular cancers

Ocular surface reconstruction: Surgical techniques using amniotic membrane graft and/or cultivated limbal epithelial cells to restore normal integrity and corneal clarity to a patient whose ocular surface has been severely damaged by disease, infection, or trauma

Multifocal and toric intraocular lenses: These are specialized intraocular lenses (IOLs) that are inserted into the eye following cataract extraction. Multifocal IOLs allow patients to focus at multiple distances without the need for reading glasses. Toric IOLs are designed to reduce astigmatism and must be inserted into the eye in a highly accurate manner

Electroretinography: Measures the electrical responses of various cell types in the retina, including the photoreceptors (rods and cones), inner retinal cells (bipolar and amacrine cells), and ganglion cells. Electrodes are usually placed on the cornea and the skin near the eye. Electroretinography is typically used to diagnose and follow inherited retinal diseases

Electrooculography: A technique for measuring the electrical resting potential of the retina. The resulting signal is called the electrooculogram. The main applications are in ophthalmological diagnosis and in recording eye movements. In ophthalmologic diagnosis, electrooculography is used to assess the function of the pigmented epithelial cells underlying the retina

Visual evoked potentials: These are voltage changes that occur due to sensory stimulation of a subject's visual field and are observed using electroencephalography. Visual evoked potentials are a useful tool to detect whether visual stimuli are being detected by the brain


#

Volume

Providing quality care and developing experience in management of unusual or complex conditions is dependent upon evaluating and treating a large volume of patients. For ophthalmology, a volume index will be calculated for both clinical and surgical cases. Clinic volume will be determined based on the number of ICD-9 code that is submitted, whereas the surgical volume index will be based on submitted CPT codes. Concordant with RTI International's methodology, an inverse logit transformation will be performed on volume scores to reduce the effects of extreme values or outliers.[9]


#

Staffing Index

In calculating the IHQ, the nurse staffing index is a ratio that reflects the intensity of inpatient and outpatient nursing. The numerator is the total number of registered nurses (RNs) on staff expressed as full-time equivalents (FTEs). The denominator is the adjusted average daily census of patients. To assess staffing for calculation of the IECQ, both faculty and ancillary staff will be examined.

Inverse logit transformation:

Zoom Image

#

Clinical Faculty

Management of complex ocular conditions requires that eye institutions have full-time clinical faculty representing all subspecialty areas of ophthalmology. The clinical faculty score would award points based on whether an eye center has at least one full-time clinical faculty member representing each subspecialty area of ophthalmology. No points would be awarded for subspecialty areas covered by part-time, volunteer, or adjunct faculty members or faculty members who do not have formal fellowship training in that subspecialty area. These subspecialty areas include the following:

  • Comprehensive or general ophthalmology

  • Cornea

  • Retina

  • Glaucoma

  • Strabismus

  • Neuro-ophthalmology

  • Oculoplastics

  • Uveitis

  • Ocular oncology


#

Technical Staff

In ophthalmology, rather than RNs, preliminary patient work-ups and administration of medication and performance of diagnostic testing are typically performed by Certified Ophthalmic Assistants (COAs), Certified Ophthalmic Technicians (COTs), and Certified Ophthalmic Medical Technicians (COMTs). Certification is regulated and governed by the Joint Commission on Allied Health Personnel in Ophthalmology (JCAHPO). The COA certification level is JCAHPO's initial core level of certification and the entry-level pathway for a career as a certified allied health professional in ophthalmology. The COT certification level is JCAHPO's second core level designation. COTs have generally worked as a COA for at least a year. The COMT certification is JCAHPO's highest core level of certification. COMTs are among the top trained and educated personnel in the allied health profession. Additionally, JCAHPO offers the Certified Diagnostic Ophthalmic Sonographer (CDOS) certification. CDOSs specialize in B-Scan ultrasonography principles and instrumentation for eye examinations.

The technician staffing index will be calculated in a manner similar to the nurse staffing index. For the technician staffing index, the numerator will be the sum of the FTEs of all COAs, COTs, COMTs, and CDOSs on staff at the eye center. The patient measure in the denominator will be the adjusted average daily census of patients. This technician staffing index is an indicator of the number of allied health personnel available to provide patient care at each eye center.


#

Affiliation with a Level 1 Trauma Center

The presence of an emergency room and status as a Level 1 or Level 2 trauma care provider are considered indicators of hospital quality. Patients presenting with complex ocular trauma frequently have other nonophthalmic trauma requiring the services of a Level 1 or Level 2 trauma center. To receive scoring credit of one point, the eye center must be affiliated with a hospital that has a Level 1 or Level 2 trauma center. The American Hospital Association defines a Level 1 trauma center as “a regional resource trauma center capable of providing total care for every aspect of injury and plays a leadership role in trauma research and education.” Level 2 is defined as “a community trauma center, capable of providing trauma care to all but the most severely injured patients who require highly specialized care.”


#

Patient Services Index

The patient services index addresses services provided to patients by the eye center that enhance patient convenience, provide highly specialized patient care, or are otherwise the product of progressive thinking and awareness of community needs. Eye centers would receive a credit of one point for each of the following service that is provided on- or off-site:

  • Translation services: Provides a translator in-person or via telephone for all patient interactions with physicians or allied health care personnel.

  • Low-vision services: Staffed by an MD or OD, low-vision services evaluate patients' visual capabilities. Patients are then introduced to, and instructed in, the use of various low-vision devices.

  • Low-vision occupational therapy: Provides instruction to patients with low-vision aids, either in their homes or in a simulated environment, on how to best utilize their current vision. Instruction is typically performed by a licensed occupational therapist with training in low-vision therapy.

  • Genetic counseling: Provides counseling for families with children affected by inherited ophthalmic conditions and those with a family history of an inherited ophthalmic condition. Counselors would aid families in discussing prognosis, available treatments, and family planning.

  • Patient support groups: Provides information, peer support, and aid to patients afflicted with a common condition (e.g., macular degeneration or keratoconus). Groups would be led by a counselor and would provide practical advice and emotional support to patients.


#

Process

The process component will contribute 25.0% toward the IECQ score and will be methodologically the same as that currently used by U.S. News & World Report. A one-page ballot will be mailed to 200 U.S. ophthalmologists in the four census regions (50 each in the Northeast, Southeast, Midwest, and West). Ophthalmologists will write what they feel are the top five U.S. ophthalmology programs based on reputation. The process component will continue to represent a 3-year rolling average of balloting. It is recommended that the percentage that process contributes toward the IECQ be decreased relative to its contribution in other specialties for calculating the IHQ. Some of the reasons for adapting these changes were discussed previously (see Reputation-Only Ranking and the Rationale for Its Abandonment in Ophthalmology).

In addition to the lack of evidence that subjective reputation is a valid measure of process, it has been demonstrated that, due to the extremely high variations in reputation scores, they contribute a disproportionate amount to the total rankings.[10] Although Sehgal found very little relationship between reputation and objective quality measures of top hospitals, he found a high degree of correlation between reputation scores and final rankings. Green et al found that reliance on reputation scores results in a small group of hospitals rising to the top of the rankings regardless of their structure and outcome scores.[16] Pope's analysis reveals that rankings are being driven almost entirely by reputation score.[7] He found that reputation scores explain over 95% of the variation in final quality scores, while risk-adjusted mortality (the main component of the outcome score) explains less than 1%. He argues further that because reputation data are not normalized in any way, they contribute far more than the claimed 32.5%. Although RTI International attempted to remedy this problem by adding a log transformation of process data for the 2010/2011 U.S. News & World Report rankings, it acknowledges that this transformation only slightly diminishes the impact of reputation score on a hospital's final standing.[9]


#

Outcomes

Mortality is the accepted metric used to determine the outcome component of the IHQ. Because comparison of raw mortality data would unfairly discriminate against hospitals that treat patients with a higher mortality risk, the “expected” mortality rate for each specialty at a given institution is calculated. The actual mortality data for each hospital are then compared with the expected adjusted rate for that hospital and these relative rates are compared across all eligible hospitals to determine outcome scores.

Fortunately for ophthalmology, mortality is a rare outcome. However, judging outcome in terms of visual function or capacity is fraught with difficulties because of the diverse types of ophthalmic procedures performed and associated ocular comorbidities. As an example, the purpose of trabeculectomy is to lower intraocular pressure, whereas cataract extraction is typically performed to improve vision. Furthermore, a healthy patient undergoing cataract extraction would likely have a better visual outcome following surgery than a patient who has both cataract and macular degeneration. The question then arises, “is there any metric(s) in ophthalmology applicable to all intraocular procedures that is (are) independent of visual function and preexisting ocular comorbidities that could be used as a comparative gauge of outcome?” Rate of postoperative infection or endophthalmitis could serve as a measure of outcome.

The most dreaded occurrence following intraocular surgery is infection. Postoperative intraocular infection or endophthalmitis is a rare outcome that can occur following any intraocular procedure. Acute endophthalmitis typically develops within 2 to 5 days after surgery and follows a fulminant course.[17] Factors influencing the development of endophthalmitis following elective ocular surgery include intraoperative complications and operative technique.[17] [18] If representative surgical procedures are established to determine eligibility, the adjusted endophthalmitis rates associated with these same procedures can be used to calculate outcome scores in a manner similar to that used for the IHQ.

As with mortality rate calculations, comparisons between eye centers will be performed relative to an “expected” rate of endophthalmitis for each eye center and for each representative surgical procedure. The methods for calculating endophthalmitis rates have been described previously.[19] The “expected” rate will be calculated by the following formula:

Expected number of endophthalmitis cases at Eye Center A = number of procedures performed by Eye Center A × overall incidence of endophthalmitis across all eye centers,

where

Overall incidence of endophthalmitis across all eye centers = total number of endophthalmitis cases across all eye centers × total number of surgical procedures performed by all eye centers.

The ratio of expected to actual number of cases will then be compared across eye centers to determine the outcome component of the IECQ. In addition to comparing the actual/expected ratio for each representative surgical procedure, simple or weighted average of ratios from these representative procedures could be combined for calculation of a global outcome ratio.

To simplify the calculations, endophthalmitis rates will include only those cases that occur within 10 days of the surgical procedure, that is, acute endophthalmitis. Those cases occurring >10 days after surgery are considered chronic and are typically due to less virulent organisms. The ICD-9 codes to be included for identifying acute endophthalmitis are 360.0, 360.00, 360.01, 360.02, 360.1, and 360.11. Exclusion criteria are surgical cases associated with trauma, surgeries involving combined procedures (e.g., trabeculectomy and cataract extraction), patients with preexisting immunosuppression, or recent history of previous intraocular surgery at an outside practice or facility. The outcome component will contribute 35% toward the final IECQ.

In addition to ranking ophthalmology programs based on the ratio of actual/expected cases of endophthalmitis, actual cases could be compared with each eye center's upper 95% prediction interval and charted on a Poisson funnel plot.[19] If the number of an eye center's actual endophthalmitis cases fell above the upper 95% prediction interval, this institution would be considered an outlier and outcome points would be deducted. The upper 95% prediction interval is equal to twice the square root of an eye center's expected incidence. Using a Poisson funnel plot normalizes the data such that eye centers with lower caseloads would not be unjustly penalized. This analysis could be performed for each representative surgical procedure or for total surgical cases.


#

Patient Safety Index

The Institute of Medicine defines patient safety as “freedom from accidental injury.” In ophthalmology, as in all specialties, attention to detail and situational awareness are important factors in preventing patient injury. In surgical specialties, many of these preventable errors occur in the operating room.[20] For the purpose of evaluating patient safety in calculating the IECQ, specific adverse events relative to ophthalmology include the following:

  • Wrong site surgery: A surgical procedure is performed on the eye opposite to what was consented to by the patient.

  • Wrong surgical procedure: A procedure not listed on the informed consent is inadvertently performed instead of, or in addition to, the consented procedure.

  • Wrong intraocular lens (IOL): An IOL with a power differing from the one intended by the surgeon is inserted into the eye following cataract extraction.

  • Wrong patient: The identity of patient being operated on is incorrect.

  • Inadvertent globe puncture: The integrity of the globe is inadvertently violated by an instrument or suture needle during surgery or by a needle during injection into the periocular or adnexal tissues.

  • Intraocular injection error: Injection of the wrong medication or incorrect dosage of a medication into the eye.

Many states including Florida, Indiana, Minnesota, and Pennsylvania already have, or are adopting, laws that require mandatory reporting of such incidents. Standard operating room procedures for ophthalmology have been developed to help prevent these errors from occurring[21]; however, a recent position paper released by the Florida Board of Medicine[22] found that ophthalmology had the highest rate of wrong side/site/patient surgeries statewide (24.7 per million vs. a combined average of 12.7 per million for all surgical specialties). In determining the IECQ, points would be deducted for the number of adverse events that occur at an eye center in a given year relative to the total number of cases performed. Comparison of a ratio would help to normalize the data and not unjustly punish eye centers with lower caseloads. The patient safety index would constitute 5% of the IECQ.


#

Limitations and Sources of Data

A limitation of the present proposal is the lack of a central clearing house to collect and process and serve as a repository of objective data. There are two alternatives to address this limitation:

  1. Survey of hospitals: For pediatrics, which was historically a reputation-only specialty with data and outcome limitations similar to those that exist for ophthalmology, U.S. News & World Report has adopted a self-reported survey which includes detailed data on all three components of the methodology. This approach allows a more comprehensive array of data to be used in the evaluation.

  2. Development of a clearinghouse via an impartial national organization such as the American Academy of Ophthalmology (AAO) or the Association of University Professors of Ophthalmology (AUPO): The collection, tabulation, and storage of eligibility, structure, outcome, and patient safety data by either organization would result in a comprehensive, centralized database for objectively evaluating academic ophthalmology programs. A committee of representative ophthalmologists and IT specialists would be established to oversee this process. It is anticipated that funding for this venture would be provided by participating organizations. As an alternative, a mechanism to improve the validity of the process component would be to increase the electorate by allowing all U.S. ophthalmologists the opportunity to vote yearly in a manner similar to the current RTI format. Votes could be cast electronically via the AAO or AUPO Web site. Ophthalmologists interested in voting could register by name and receive a unique voting password electronically. To prevent a geographically skewed distribution of votes, a limitation on the number of ballots cast (i.e., first come, first serve voting) per region, calculated using a population-based formula established by the oversight committee, could be implemented. Additionally, a more informed electorate could be established by allowing ophthalmologists open access to a summarized version of each eligible institution's objective data via the AAO or AUPO Web site. On a yearly basis, all data would be downloaded to RTI for final calculation of the IECQ.


#

Calculation of the Index of Eye Center Quality

The metrics for ranking ophthalmology programs proposed in this report are designed to easily fit into the existing methodology used by U.S. News & World Report.[20] The IHQ equation is presented as a weighted sum of structural, process, and outcome measures:

Zoom Image

where:

IHQ = Index of Hospital Quality

Sn  = standardized value for structural indicator n

Fn  = factor loadings for structural indicator n

P = standardized nomination score (process)

M = standardized mortality score (outcome)

PS = standardized patient safety index score

As such, the equation used to calculate IHQ could be reformatted to calculate IECQ:

Zoom Image

where:

IECQ = Index of Eye Center Quality

Sn  = standardized value for structural indicator n

Fn  = factor loadings for structural indicator n

P = standardized nomination score (process)

E = standardized endophthalmitis score (outcome)

PS = standardized patient safety index score

The main difference between the IHQ and IECQ equations is the relative component weighting for each index (IECQ: structure = 35%; process = 25%; outcome = 35%; and patient safety = 5%). Further refinements in the equation may need to be made to accommodate the data being analyzed.


#
#

Conclusion

This report provides specialty-specific, objective metrics to be used to determine eligibility and evaluate U.S. ophthalmology programs utilizing the methodological framework already established by RTI International for U.S. News & World Report. A composite score, the IECQ, calculated using information from ophthalmology-related measures of “structure,” “process,” “outcome,” and “patient safety,” is used to determine the ranking of eligible programs. Although the present report introduces objective information to measure structure, outcome, and patient safety in ophthalmology, it retains the survey-based “process” component. It does, however, recommend reducing the weighting of the subjective “process” component when calculating the IECQ.


#
#

Conflict of Interest

Dr. Schuman receives royalties for intellectual property licensed by Massachusetts Institute of Technology and Massachusetts Eye and Ear Infirmary to Carl Zeiss Meditec, Inc.

Financial Support

This study is supported in part by the Eye and Ear Foundation (Pittsburgh, PA) and an unrestricted grant from Research to Prevent Blindness (New York, NY).

  • References

  • 1 Joynt KE, Orav EJ, Jha AK. The association between hospital volume and processes, outcomes, and costs of care for congestive heart failure. Ann Intern Med 2011; 154 (2) 94-102
  • 2 Magid DJ, Calonge BN, Rumsfeld JS , et al; National Registry of Myocardial Infarction 2 and 3 Investigators. Relation between hospital primary angioplasty volume and mortality for patients with acute MI treated with primary angioplasty vs thrombolytic therapy. JAMA 2000; 284 (24) 3131-3138
  • 3 McGrath PD, Wennberg DE, Dickens Jr JD , et al. Relation between operator and hospital volume and outcomes following percutaneous coronary interventions in the era of the coronary stent. JAMA 2000; 284 (24) 3139-3144
  • 4 Dudley RA, Johansen KL, Brand R, Rennie DJ, Milstein A. Selective referral to high-volume hospitals: estimating potentially avoidable deaths. JAMA 2000; 283 (9) 1159-1166
  • 5 Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med 2002; 137 (6) 511-520
  • 6 Nguyen NT, Paya M, Stevens CM, Mavandadi S, Zainabadi K, Wilson SE. The relationship between hospital volume and outcome in bariatric surgery at academic medical centers. Ann Surg 2004; 240 (4) 586-593 , discussion 593–594
  • 7 Pope DG. Reacting to rankings: evidence from “America's best hospitals and colleges.” 2006. Available at: opimweb.wharton.upenn.edu/documents/JobMarketPaper-Pope.pdf
  • 8 Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood) 2005; 24 (4) 1150-1160
  • 9 Murphy J, McFarlane E, Olmsted MG, Severance J, Drozd EM, Morley M, Hill C. U.S. News & World Report 2010/11 Best Hospitals Rankings Methodology. Washington, DC: U.S. News & World Report; 2010
  • 10 Sehgal AR. The role of reputation in U.S. News & World Report's rankings of the top 50 American hospitals. Ann Intern Med 2010; 152 (8) 521-525
  • 11 Teasley III CE. Where's the best medicine? The hospital rating game. Eval Rev 1996; 20 (5) 568-579
  • 12 McGaghie WC, Thompson JA. America's best medical schools: a critique of the U.S. News & World Report rankings. Acad Med 2001; 76 (10) 985-992
  • 13 Bartels LM. Presidential Primaries and the Dynamics of Public Choice. Princeton, NJ: Princeton University Press; 1988
  • 14 Bikhchandani S, Hirshleifer D, Welch I. A theory of fads, fashion, custom and cultural change as informational cascades. J Polit Econ 1992; 100 (5) 992-1026
  • 15 Callander S. Bandwagons and momentum in sequential voting. Rev Econ Stud 2007; 74: 653-684
  • 16 Green J, Wintfeld N, Krasner M, Wells C. In search of America's best hospitals. The promise and reality of quality assessment. JAMA 1997; 277 (14) 1152-1155
  • 17 Johns KJ, Feder RS, Hamill MB, Miller-Meeks MJ, Rosenfeld SI, Perry PE eds. Lens and Cataract: Section 11, Basic and Clinical Science Course. San Francisco, CA: American Academy of Ophthalmology; 2003. –2004: 171
  • 18 Ta CN. Minimizing the risk of endophthalmitis following intravitreous injections. Retina 2004; 24 (5) 699-705
  • 19 Ng JQ, Morlet N, Bremner AP, Bulsara MK, Morton AP, Semmens JB. Techniques to monitor for endophthalmitis and other cataract surgery complications. Ophthalmology 2008; 115 (1) 3-10
  • 20 Seiden SC, Barach P. Wrong-side/wrong-site, wrong-procedure, and wrong-patient adverse events: are they preventable?. Arch Surg 2006; 141 (9) 931-939
  • 21 Pennsylvania Patient Safety Reporting System (PAPSRS). Focusing on eye surgery. PAPSRS Patient Saf Advis 2005; 2 (1) 12-14
  • 22 Beebe J. Wrong side surgery position paper. 2009 . Available at: www.doh.state.fl.us/MQA/medical/info_wrong.doc

Address for correspondence

Warren J. Scherer, MD, PhD
Newsom Eye & Laser Center
13904 N. Dale Mabry Hwy, Suite 200
Tampa, FL 33618

  • References

  • 1 Joynt KE, Orav EJ, Jha AK. The association between hospital volume and processes, outcomes, and costs of care for congestive heart failure. Ann Intern Med 2011; 154 (2) 94-102
  • 2 Magid DJ, Calonge BN, Rumsfeld JS , et al; National Registry of Myocardial Infarction 2 and 3 Investigators. Relation between hospital primary angioplasty volume and mortality for patients with acute MI treated with primary angioplasty vs thrombolytic therapy. JAMA 2000; 284 (24) 3131-3138
  • 3 McGrath PD, Wennberg DE, Dickens Jr JD , et al. Relation between operator and hospital volume and outcomes following percutaneous coronary interventions in the era of the coronary stent. JAMA 2000; 284 (24) 3139-3144
  • 4 Dudley RA, Johansen KL, Brand R, Rennie DJ, Milstein A. Selective referral to high-volume hospitals: estimating potentially avoidable deaths. JAMA 2000; 283 (9) 1159-1166
  • 5 Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med 2002; 137 (6) 511-520
  • 6 Nguyen NT, Paya M, Stevens CM, Mavandadi S, Zainabadi K, Wilson SE. The relationship between hospital volume and outcome in bariatric surgery at academic medical centers. Ann Surg 2004; 240 (4) 586-593 , discussion 593–594
  • 7 Pope DG. Reacting to rankings: evidence from “America's best hospitals and colleges.” 2006. Available at: opimweb.wharton.upenn.edu/documents/JobMarketPaper-Pope.pdf
  • 8 Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood) 2005; 24 (4) 1150-1160
  • 9 Murphy J, McFarlane E, Olmsted MG, Severance J, Drozd EM, Morley M, Hill C. U.S. News & World Report 2010/11 Best Hospitals Rankings Methodology. Washington, DC: U.S. News & World Report; 2010
  • 10 Sehgal AR. The role of reputation in U.S. News & World Report's rankings of the top 50 American hospitals. Ann Intern Med 2010; 152 (8) 521-525
  • 11 Teasley III CE. Where's the best medicine? The hospital rating game. Eval Rev 1996; 20 (5) 568-579
  • 12 McGaghie WC, Thompson JA. America's best medical schools: a critique of the U.S. News & World Report rankings. Acad Med 2001; 76 (10) 985-992
  • 13 Bartels LM. Presidential Primaries and the Dynamics of Public Choice. Princeton, NJ: Princeton University Press; 1988
  • 14 Bikhchandani S, Hirshleifer D, Welch I. A theory of fads, fashion, custom and cultural change as informational cascades. J Polit Econ 1992; 100 (5) 992-1026
  • 15 Callander S. Bandwagons and momentum in sequential voting. Rev Econ Stud 2007; 74: 653-684
  • 16 Green J, Wintfeld N, Krasner M, Wells C. In search of America's best hospitals. The promise and reality of quality assessment. JAMA 1997; 277 (14) 1152-1155
  • 17 Johns KJ, Feder RS, Hamill MB, Miller-Meeks MJ, Rosenfeld SI, Perry PE eds. Lens and Cataract: Section 11, Basic and Clinical Science Course. San Francisco, CA: American Academy of Ophthalmology; 2003. –2004: 171
  • 18 Ta CN. Minimizing the risk of endophthalmitis following intravitreous injections. Retina 2004; 24 (5) 699-705
  • 19 Ng JQ, Morlet N, Bremner AP, Bulsara MK, Morton AP, Semmens JB. Techniques to monitor for endophthalmitis and other cataract surgery complications. Ophthalmology 2008; 115 (1) 3-10
  • 20 Seiden SC, Barach P. Wrong-side/wrong-site, wrong-procedure, and wrong-patient adverse events: are they preventable?. Arch Surg 2006; 141 (9) 931-939
  • 21 Pennsylvania Patient Safety Reporting System (PAPSRS). Focusing on eye surgery. PAPSRS Patient Saf Advis 2005; 2 (1) 12-14
  • 22 Beebe J. Wrong side surgery position paper. 2009 . Available at: www.doh.state.fl.us/MQA/medical/info_wrong.doc

Zoom Image
Zoom Image
Zoom Image