Putting Hospitals to the Test

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

By "matching" patients across a group of hospitals, a new paper co-authored by two Wharton professors suggests a fairer and more accurate way of assessing the quality of health care providers.

Read the full story: http://knlg.net/1rBavF1 📂 Innovation;Health Care Management 🏷️ Research

Read the full transcriptHide the full transcript
  1. our research really uh revolves around a problem that chief medical officers have
  2. uh they open up the morning paper and they find that their Hospital's been ranked very low in some quality of care
  3. report uh and their initial reaction is always our patients are really sicker and that's why we look so bad in the
  4. reports uh This research template matching uh tries to improve the reports so that there's a fair comparison across
  5. all hospitals and we really tried to look at a the problem in a different way using multivariate matching which has
  6. never been done before in looking at quality to uh be able to fairly compare hospitals uh and then the chief medical
  7. officer wouldn't be as upset because he'd probably believe either there was a problem or he'd look good in the reports
  8. template matching is a new way of looking at the way hospitals uh uh treat patients um the usual problem when we
  9. compare hospitals is that we say uh here are all the patients that you saw and then if your patients were going to be
  10. going to another hospital or the typical Hospital how would they do um that's uh
  11. a strange way to compare hospitals because each hospital has a different set of patients and so it's really not a
  12. fair way to compare hospitals cuz one Hospital might have an easier set of patients we're going to make some
  13. adjustments but they still have a very different set than another hospital that might have a much different uh more
  14. difficult set of patients what we decided to do was create a template of patients meaning a set of patients in
  15. our case it was 300 patients with general surgery and orthopedic surgery and we said let's take a a relevant
  16. template and then match at each Hospital their patients that would fit the template so we ended up with 300
  17. patients being matched at 217 Hospitals now we have very similar patients at each of the hospitals because they've
  18. all been matched to this cookie cutter template which is producing very similar patients across hospitals so now we can
  19. make a very fair comparison because we're looking at the same in a sense 300 patients at each and every hospital
  20. which is completely different than the way almost all report cards are made today where they look at the patients at
  21. an individual hospital then they try to estimate how would they have done at other hospitals where each hospital has
  22. different patients so we're completely changing things around the technique is and there's another word for it called
  23. direct standardization um and what we're doing is saying you have to have a fair exam
  24. the fair exam is how did the hospitals do on these 300 patients we find the hospitals 300 patients that look like
  25. the template and that's what we compare across hospitals so now it's very hard for a chief medical officer to say our
  26. patients were really easier sicker than another hospital it's very hard to do that because they've the patients we're
  27. going to be examining at their Hospital are the same 300 that are at each and every other Hospital same in the sense
  28. that we have multivariant matched them which means that we've matched them on literally hundreds of characteristics so
  29. that they very very similar and their risk of doing poorly or their chance of doing well is very very similar across
  30. all the hospitals what we say in the paper what we show in the paper is that the these 300 patients at each and every
  31. hospital are incredibly similar their AG is the same their rate of diabetes and heart failure and all the other
  32. characteristics we'd be interested in um are incredibly similar statistically UND
  33. differentiable to the other hospitals um but yet what we find is that the outcomes are very very
  34. different I think that there are two key takeaways the first is that we can do the matching uh and what we showed was
  35. that you can get very close matches across hospitals each Hospital Stamped Out to this template of patients the
  36. second is that there's just great variability in the way the hospitals handled the patients and the outcomes of
  37. the patients and uh I think seeing that the matches were so close and at the same time seeing that the outcomes were
  38. so different was uh something that I think people should realize that there is different there are differences in
  39. quality across hospitals and that there are I think better ways to measure them than what we've been doing up to
  40. now probably the biggest surprise was that after we did this very close matching such that all of the hospitals
  41. had an incredibly similar sample of 300 patients there was great variation in outcomes some hospitals had high
  42. complication rates some had low complication rates some had high mortality rates some had low mortality
  43. rates things that when I would see this in a standard report I might not believe
  44. because I knew the hospitals were seeing different patients but here after having
  45. matched so closely to see this great variation in outcomes I think that was an eye
  46. opener we've given that's some thought uh We've usually been applying this technique to the hospital sector but I
  47. think you could easily apply this to nursing home quality schools um we can imagine creating a template of students
  48. and stamping the students out across different schools seeing how the student outcomes were so yes you could you could
  49. U you could use this technique in other areas I think the key here is that we're
  50. taking uh a uh from one field which is from statistics and in particular multivariate m matching for which Paul
  51. rosenbom is the world's expert in in that area uh and we're applying it to Quality assessment and I just don't
  52. think that's been done and it certainly hasn't been done with a a template which
  53. kind of levels the playing field and lets you really see if your quality is different than than
  54. others I I think the biggest problem is that people um tend to uh believe report
  55. cards based on what's called direct standardization where we're looking at different patients at different
  56. hospitals different types of patients even non-overlapping populations and saying that we can extrapolate from the
  57. models to say that one Hospital's quality is better than another I think that's a misconception I don't think
  58. that the models currently allow us to do that and that if we had better techniques and I think this is one of
  59. them uh we'd be able to at least provide confidence to the reader both the user in terms of the patient and also the
  60. people at hospitals that are trying trying to improve safety and quality that uh there really are differences or
  61. not and I think uh uh that's probably the the main point there are many groups that are
  62. grading hospitals the federal government the it's called Hospital compare the website that lets you click on a
  63. hospital and compare it to another hospital uh has been doing this since 2007 um there are many different
  64. organizations uh and and web-based companies that uh great hospitals uh they often look to hospital compare
  65. which is the main model for example Consumer Reports looks to hospital compare um Hospital compare suffers from
  66. these same problems It suffers from using indirect standardization uh uh in the sense that
  67. it's kind of taking what the hospital the patients the hospital saw and making an extrapolation about how they would
  68. have done at another hospital um and so uh I I think that that uh if that could be changed that would be uh uh a great
  69. benefit to u to the application of looking at quality often hospitals have access to
  70. National data sets or comparative data sets so hospitals could in a sense form their own templates and then see how
  71. their patients would do at other hospitals using the same technique but they'd have to understand the details of
  72. multivariant matching uh we bring to the table a way to do this in in a very I think sophisticated way uh through the
  73. work uh of statisticians at at at at Warton um in particular Paul rosenbom um but uh they if they could do that
  74. matching then they could better compare their results to other hospitals and know whether they're really doing a good
  75. job or not uh I think a second area that uh we actually just talked about was this idea
  76. of making your own Boutique template so a chief medical officer could for example say uh let's construct uh a
  77. template on these difficult patients that we see and let's see how other hospitals would have handled these same
  78. difficult patients I think that Boutique template is a way that we're is the next
  79. step and that's where we're moving and then there's other applications of multivariant matching to looking at
  80. quality suppose you have a very small Hospital uh you don't match the template very often because you're so small you
  81. don't have the patience to be able to match to this overall National template well we can turn things around and we
  82. can say uh okay let's look at the small hospitals patients and then match them to the
  83. whole country so use the whole country as a kind of a frame of reference uh and then we have a very very good idea
  84. looking at the outcomes of the small hospitals patients and their matched sample how the how the small hospital
  85. did relative to the typical patient in the country so that's another application to multivariant uh with
  86. multivariant matching to help improve quality

Brief Summary

Hospital quality assessment improves with template matching for fair comparisons across patient demographics.

Key points

  • Revolutionizing Hospital Comparisons. A new template matching technique allows for fair comparisons across hospitals.
  • Surprising Variability in Outcomes. Despite similar patient profiles, hospitals show significant differences in outcomes.

Tension

40/ 100Low tension

Related episodes