What do you do when evidence changes?
Process measures may provide better proof of VTE care
There has been a lot of talk about the importance of shifting away from process measures to determine quality of care and making more use of outcomes measures. The argument goes like this: Ticking a box to indicate you did something is all well and good, but whether the patient does better because you have done whatever that checklist asked you to do could be a better determinant of quality care.
That premise may not always hold up. A recent study strongly indicates that the commonly used measures for determining if a hospital is providing good care for surgical patients to prevent venous thromboembolism (VTE) may not show that at all. Indeed, the measure in question may not only give a false sense of quality, but as value-based purchasing (VBP) becomes a reality, it could financially hurt the hospitals that are providing better care.
The study1 examined whether reporting of VTE rates is really a good way of showing which hospitals are doing the best to prevent it. Data from nearly 3,000 hospitals and close to a million surgical patients was used. The researchers looked at VTE event rates, VTE prophylaxis rates, whether a hospital was considered a quality facility (based on accreditations, size, and quality initiatives undertaken), and the relationships between these metrics.
Not surprisingly, providers who did more imaging studies on patients to find VTE found more events, and the hospitals that did more imaging and found more events scored higher on the quality metrics used by the study authors. Paradoxically, the increased number of events makes them look worse than other providers and hospitals that don't look as often for such events. More surprising was a correlation between increased use of VTE prophylaxis and increased VTE rates. One might expect that trying to prevent VTE would result in a decline in events. And it wasn't hospitals that might have quality problems that had this strange correlation evident: Researchers found a positive correlation between hospitals that had higher quality scores and higher prophylaxis rates with higher VTE rates.
The authors note that this isn't the first time a study has found that there isn't a relationship between more prophylaxis and surveillance and lower reported VTE rates. They note that this evident surveillance bias calls into question the use of VTE event rates as a way of measuring quality — something currently done by several organizations that create lists of top hospitals, such as the University Health Consortium (UHC) and the American College of Surgeons (ACS). VBP will make use of VTE rates to determine cuts in reimbursement from the Centers for Medicare & Medicaid Services (for low performers) beginning January 2015. Currently, CMS collects that data for the Hospital Compare program, but it is only reported in supplementary material.
Beyond the potential loss of reimbursement, hospitals that are doing exactly the right thing may see the public avoid those facilities because the VTE rates look bad compared to other hospitals. The authors also worry that providers may shift their behavior — do fewer studies, use chemo-prophylaxis on patients who aren't at high risk for VTE and thus increase the potential for dangerous bleeding, or using inferior vena cava (IVC) filters when they aren't needed.
Back to the drawing board?
"Clearly VTE is important," says lead author Karl Bilimoria, MD, MS, a surgeon at Northwestern Hospital in Chicago and assistant professor in surgery at the Center for Healthcare Studies in the Feinberg School of Medicine at Northwestern University. "But the outcome measure may not indicate quality of care. It might indicate the inverse — that those who look more because they are vigilant find more events and may then be penalized" because more VTE means lower quality of care, according to some organizations.
Bilimoria says that if you can't get an outcome measure to adequately and validly determine quality, then it's probably best to take another look at process measures. "The problem with that is that what we do currently in process measures around VTE is also inadequate," he says. "We only look at the 24 hours around a surgery, for instance. The process measures we have, though, could be expanded to be more comprehensive and thus give us a more accurate measure of quality of care related to VTEs."
The authors thought there might be some surveillance bias, but Bilimoria says the magnitude of it wasn't anticipated. The variations in use of imaging among hospitals were also a bit of a shock.
Bilimoria may be sure that this is the wrong way to measure VTE care, but he is also adamant that you can't stop caring about it or looking at metrics that can help get a handle on it. He suggests expanding the time you look at these issues from the 24 hours around surgery to the entire post-operative period. Whether the patient gets up and starts walking, and mechanical and chemo-prophylaxis used — even into the discharge period for the latter — may give you a better sense of if you are doing the right thing.
For now, the measure is something required by many parties, so collecting that data will continue, but consider the findings of this study when you look at them, Bilimoria says, and be aware of the potential for unintended consequences, particularly those that might cause harm, such as chemo-prophylaxis on patients that don't really need it.
The authors of the study are working to build a "more robust" process measure that will tell stakeholders more about the quality of care a provider or facility gives around VTE. "Some people will probably think we should use an outcome measure, but for now, we don't have one that works."
Next steps
Elements of VTE care have been on the radar of the National Quality Forum (NQF) since 2006. (See a report on measures included at http://www.qualityforum.org/Publications/2008/10/National_Voluntary_Consensus_Standards_for_Prevention_and_Care_of_Venous_Thromboembolism__Additional_Performance_Measures.aspx.)
NQF endorsed three of the related measures in 2012 as patient safety measures related to reducing complications — 0372: Intensive Care Unit Venous Thromboembolism Prophylaxis (Joint Commission), 0373: Venous Thromboembolism Patients with Anticoagulant Overlap Therapy (Joint Commission), 0450: Postoperative Pulmonary Embolism or Deep Vein Thrombosis Rate (PSI 12) (AHRQ).
Now that there is some evidence that they may not be appropriate, what happens? Erin Reese, the public outreach coordinator at NQF, says that there is an established ad hoc review process through which any party can request that a measure be reviewed, as long as there is "adequate evidence" to justify it. To meet those criteria, the evidence supporting the measure has to have changed, its implementation has to have led to unintended consequences, or there have to have been material changes made to the measure.
Reese says "very few measures undergo the ad hoc process. That said, the [VTE rate measure] seems to fit the criteria."
The Joint Commission's measures related to VTE are more process related — whether a patient has had prophylactic treatment for VTE — says Daniel Castillo, MD, MBA, medical director at the division of healthcare quality evaluation at the commission.
"Outcomes measures is a young science," he says, while process measures used by The Joint Commission for its reporting requirements are all based on science and results that have stood the test of time.
"This is a really strong, very good study," Castillo says of Bilimoria's findings. "It's interesting given the push for more outcomes measures, but we really should be careful to make sure that we limit the possibility of unintended consequences."
Castillo's advice to an organization is to make sure you aren't encouraging providers to "practice to a measure." Don't change what you do now — a requirement is still a requirement, even if new information makes it look of dubious use. Instead, look to other measures to determine if you are providing quality care, says Castillo. Ask your providers what they think makes a difference, and make sure you are looking at those metrics.
Reference
1. Bilimoria KY, Chung J, Ju MH, et al. Evaluation of Surveillance Bias and the Validity of the Venous Thromboembolism Quality Measure. JAMA. 2013 Oct 9;310(14):1482-1489.
For more information on this topic, contact:
- Karl Y. Bilimoria, MD, MS, Assistant Professor in Surgery-Surgical Oncology, Center for Healthcare StudiesInstitute for Public Health and Medicine and Medical Social Sciences, Feinberg School of Medicine, Northwestern University and Northwestern Hospital. Chicago, IL. Email: [email protected].
- Daniel Castillo, MD, MBA, Medical Director, Division of Healthcare Quality Evaluation, The Joint Commission. Oakbrook Terrace, IL. Email: [email protected].
- Erin Reese, Public Outreach Coordinator, National Quality Forum, Washington DC. Email: [email protected].
