The importance of reliability adjustment

Ranking hospitals on surgical mortality:The importance of reliability adjustmentJustin B. Dimick, MD, MPHAssistant Professor of SurgeryUniversity of MichiganAcademyHealth Annual Research MeetingJuly 28 th , 2009

Surgical quality measurementOutcomes – “Gold standard”• Patients, payers, providers agree theyare important• Better than other alternatives• Hospital volume, process measures• Strong track record in cardiac surgery– NY, PA, CA report cards

Big problems with small samples

A technique for dealingwith statistical “noise”• Reliability adjustment: Adjust point estimate of anoutcome for uncertainty by “shrinking” it back towards thepopulation average• Empirical Bayes techniques• An application of hierarchical modeling (estimatingrandom effects)• Increasingly applied in measuring performance ineducation and health care

Adjusting mortality rates for reliability:Reliabilityadjustedmortality rate= HospitalReliability + Overallmortalitymortality1-ReliabilityReliability=SignalSignal + Noise=Variance (µi)Variance (µi) + Variance (εij)/n6/29/2009What actually happens:For hospitals with large number of cases, moreweight is afforded to the hospital’s mortality rate andless to overall rateFor hospitals with few cases, less weight is affordedto the hospital’s mortality rate and more to the overallrate

Research question:Are hospital rankings based on reliabilityadjustedmortality better at predicting futureperformance?6/29/2009

• Data Sources:Data Source, Study Population, andPrimary OutcomeNational Medicare data (2003-2006)• Procedures:3 high-risk operations; focus of quality assessmentactivities; common to uncommon.– Coronary artery bypass– Abdominal aortic aneurysm repair– Pancreatic resection• Primary outcome:– Death within 30-days or prior to discharge6/29/2009

Analytic approach:• Risk-adjustment:– Patient age, gender, race, admission acuity,coexisting diseases using Elixhauser index• Reliability-adjustment:– Hierarchical modeling to estimate randomeffects for each hospital (xtmelogit inSTATA)– Estimated using empirical Bayes techniques6/29/2009

Analytic approach:2003-04 2005-061Risk-adjustedmortalityNOT adjustedfor reliabilityRisk-adjustedmortalityAdjusted forreliabilityRank hospitals,create 5 groups(quintiles)23451234Risk-adjustedmortalityRisk-adjustedmortality56/29/2009

Number of patients and hospitalcaseloads, 2003-04OperationMedicare casesCoronary artery bypass 303,132 267 (256)Hospital caseloads,mean (SD)Abdominal aortic aneurysmrepair70,863 33 (49)Pancreatic resection 6,192 6 (13)

Impact of reliability adjustmentRisk-adjustedMortality (%)0 10 20 30 40 50 60 70 80 90 100Pancreatic ResectionBefore reliability adjustmentAfter reliability adjustmentRisk-adjusted mortality rates20 randomly sampled hospitals

Are reliability adjusted mortality rates better atpredicting future performance?Pancreatic cancer resection12.0%Riskadjustedmortality, %(2005-06)10.0%8.0%6.0%7.6%5.9%10.5%4.5%7.4%10.8%9.0%4.0%3.4%3.2%2.7%2.0%0.0%1 2 3 4 5 1 2 3 4 5Not adjusted for reliablityAdjusted for reliabilityHospital Rankings (Quintiles)(2003-04)

Are reliability adjusted mortality rates better atpredicting future performance?6.0%5.0%Abdominal aortic aneurysm repair4.9% 4.9%Risk-adjustedmortality(2005-06)4.0%3.0%4.0%3.1%3.4%3.7%3.2%3.3%3.9%4.0%2.0%1.0%0.0%1 2 3 4 5 1 2 3 4 5Not adjusted for reliablityAdjusted for reliabilityQuintiles of Hospital Rankings(2003-04)

Are reliability adjusted mortality rates better atpredicting future performance?Risk-adjusted-mortality(2005-06)6.0%5.0%4.0%3.0%Coronary artery bypass5.3% 5.4%4.5% 4.4%4.0% 4.0%3.4% 3.3%3.0% 3.0%2.0%1.0%0.0%1 2 3 4 5 1 2 3 4 5Not adjusted for reliablityAdjusted for reliabilityQuintiles of Hospital Rankings(2003-04)

Summary:• Reliability adjusted mortality betterforecasts future performance for high-risksurgery• Most important for uncommon operations– Pancreatectomy > AAA >> CABG6/29/2009

Implications:• Reliability adjustment should become thestandard for all efforts aimed at tracking surgicaloutcomes• Applications:– Massachusetts cardiac surgery report card– Blueprint for a new ACS-NSQIP– The Leapfrog Group’s composite measure6/29/2009

AcknowledgementsDouglas Staiger, PhDCollaborator, Dartmouth Medical School6/29/2009