Putting Hospitals to the Test
Ask about Knowledge at Wharton
Episode Description
By "matching" patients across a group of hospitals, a new paper co-authored by two Wharton professors suggests a fairer and more accurate way of assessing the quality of health care providers.
Read the full story: http://knlg.net/1rBavF1 📂 Innovation;Health Care Management 🏷️ Research
Read the full transcriptHide the full transcript
- our research really uh revolves around a problem that chief medical officers have
- uh they open up the morning paper and they find that their Hospital's been ranked very low in some quality of care
- report uh and their initial reaction is always our patients are really sicker and that's why we look so bad in the
- reports uh This research template matching uh tries to improve the reports so that there's a fair comparison across
- all hospitals and we really tried to look at a the problem in a different way using multivariate matching which has
- never been done before in looking at quality to uh be able to fairly compare hospitals uh and then the chief medical
- officer wouldn't be as upset because he'd probably believe either there was a problem or he'd look good in the reports
- template matching is a new way of looking at the way hospitals uh uh treat patients um the usual problem when we
- compare hospitals is that we say uh here are all the patients that you saw and then if your patients were going to be
- going to another hospital or the typical Hospital how would they do um that's uh
- a strange way to compare hospitals because each hospital has a different set of patients and so it's really not a
- fair way to compare hospitals cuz one Hospital might have an easier set of patients we're going to make some
- adjustments but they still have a very different set than another hospital that might have a much different uh more
- difficult set of patients what we decided to do was create a template of patients meaning a set of patients in
- our case it was 300 patients with general surgery and orthopedic surgery and we said let's take a a relevant
- template and then match at each Hospital their patients that would fit the template so we ended up with 300
- patients being matched at 217 Hospitals now we have very similar patients at each of the hospitals because they've
- all been matched to this cookie cutter template which is producing very similar patients across hospitals so now we can
- make a very fair comparison because we're looking at the same in a sense 300 patients at each and every hospital
- which is completely different than the way almost all report cards are made today where they look at the patients at
- an individual hospital then they try to estimate how would they have done at other hospitals where each hospital has
- different patients so we're completely changing things around the technique is and there's another word for it called
- direct standardization um and what we're doing is saying you have to have a fair exam
- the fair exam is how did the hospitals do on these 300 patients we find the hospitals 300 patients that look like
- the template and that's what we compare across hospitals so now it's very hard for a chief medical officer to say our
- patients were really easier sicker than another hospital it's very hard to do that because they've the patients we're
- going to be examining at their Hospital are the same 300 that are at each and every other Hospital same in the sense
- that we have multivariant matched them which means that we've matched them on literally hundreds of characteristics so
- that they very very similar and their risk of doing poorly or their chance of doing well is very very similar across
- all the hospitals what we say in the paper what we show in the paper is that the these 300 patients at each and every
- hospital are incredibly similar their AG is the same their rate of diabetes and heart failure and all the other
- characteristics we'd be interested in um are incredibly similar statistically UND
- differentiable to the other hospitals um but yet what we find is that the outcomes are very very
- different I think that there are two key takeaways the first is that we can do the matching uh and what we showed was
- that you can get very close matches across hospitals each Hospital Stamped Out to this template of patients the
- second is that there's just great variability in the way the hospitals handled the patients and the outcomes of
- the patients and uh I think seeing that the matches were so close and at the same time seeing that the outcomes were
- so different was uh something that I think people should realize that there is different there are differences in
- quality across hospitals and that there are I think better ways to measure them than what we've been doing up to
- now probably the biggest surprise was that after we did this very close matching such that all of the hospitals
- had an incredibly similar sample of 300 patients there was great variation in outcomes some hospitals had high
- complication rates some had low complication rates some had high mortality rates some had low mortality
- rates things that when I would see this in a standard report I might not believe
- because I knew the hospitals were seeing different patients but here after having
- matched so closely to see this great variation in outcomes I think that was an eye
- opener we've given that's some thought uh We've usually been applying this technique to the hospital sector but I
- think you could easily apply this to nursing home quality schools um we can imagine creating a template of students
- and stamping the students out across different schools seeing how the student outcomes were so yes you could you could
- U you could use this technique in other areas I think the key here is that we're
- taking uh a uh from one field which is from statistics and in particular multivariate m matching for which Paul
- rosenbom is the world's expert in in that area uh and we're applying it to Quality assessment and I just don't
- think that's been done and it certainly hasn't been done with a a template which
- kind of levels the playing field and lets you really see if your quality is different than than
- others I I think the biggest problem is that people um tend to uh believe report
- cards based on what's called direct standardization where we're looking at different patients at different
- hospitals different types of patients even non-overlapping populations and saying that we can extrapolate from the
- models to say that one Hospital's quality is better than another I think that's a misconception I don't think
- that the models currently allow us to do that and that if we had better techniques and I think this is one of
- them uh we'd be able to at least provide confidence to the reader both the user in terms of the patient and also the
- people at hospitals that are trying trying to improve safety and quality that uh there really are differences or
- not and I think uh uh that's probably the the main point there are many groups that are
- grading hospitals the federal government the it's called Hospital compare the website that lets you click on a
- hospital and compare it to another hospital uh has been doing this since 2007 um there are many different
- organizations uh and and web-based companies that uh great hospitals uh they often look to hospital compare
- which is the main model for example Consumer Reports looks to hospital compare um Hospital compare suffers from
- these same problems It suffers from using indirect standardization uh uh in the sense that
- it's kind of taking what the hospital the patients the hospital saw and making an extrapolation about how they would
- have done at another hospital um and so uh I I think that that uh if that could be changed that would be uh uh a great
- benefit to u to the application of looking at quality often hospitals have access to
- National data sets or comparative data sets so hospitals could in a sense form their own templates and then see how
- their patients would do at other hospitals using the same technique but they'd have to understand the details of
- multivariant matching uh we bring to the table a way to do this in in a very I think sophisticated way uh through the
- work uh of statisticians at at at at Warton um in particular Paul rosenbom um but uh they if they could do that
- matching then they could better compare their results to other hospitals and know whether they're really doing a good
- job or not uh I think a second area that uh we actually just talked about was this idea
- of making your own Boutique template so a chief medical officer could for example say uh let's construct uh a
- template on these difficult patients that we see and let's see how other hospitals would have handled these same
- difficult patients I think that Boutique template is a way that we're is the next
- step and that's where we're moving and then there's other applications of multivariant matching to looking at
- quality suppose you have a very small Hospital uh you don't match the template very often because you're so small you
- don't have the patience to be able to match to this overall National template well we can turn things around and we
- can say uh okay let's look at the small hospitals patients and then match them to the
- whole country so use the whole country as a kind of a frame of reference uh and then we have a very very good idea
- looking at the outcomes of the small hospitals patients and their matched sample how the how the small hospital
- did relative to the typical patient in the country so that's another application to multivariant uh with
- multivariant matching to help improve quality
Brief Summary
Hospital quality assessment improves with template matching for fair comparisons across patient demographics.
Key points
- Revolutionizing Hospital Comparisons. A new template matching technique allows for fair comparisons across hospitals.
- Surprising Variability in Outcomes. Despite similar patient profiles, hospitals show significant differences in outcomes.
Episode highlights
- Revolutionizing Hospital ComparisonsA new template matching technique allows for fair comparisons across hospitals.“We’re completely changing things around the technique.”2:46Jun 3, 2014
- Surprising Variability in OutcomesDespite similar patient profiles, hospitals show significant differences in outcomes.“The biggest surprise was that after we did this very close matching...”4:52Jun 3, 2014
Tension
40/ 100Low tension





