
Real world evidence: Lessons learned from an FDA pilot show the limits of emulating RCTs
Only about half of a select group of clinical trials could be well-emulated with available real world evidence, according to the newly discussed results of an FDA pilot program.
The FDA-funded program, known as RCT-DUPLICATE, helped researchers from the Brigham and Women’s Hospital in Boston try to evaluate whether randomized controlled trials can be duplicated with RWE across a range of therapeutic areas.
The initiative is part of the FDA’s work, mandated by Congress in the 21st Century Cures Act, on how it plans to evaluate the use of RWE to assess the effectiveness of medical products.
John Concato, associate director for RWE analytics in the FDA’s Office of Medical Policy, made clear at a Duke-Margolis event on the topic last week that the standard for substantial evidence remains unchanged. He also noted that seeking to emulate a hypothetical RCT when designing a non-interventional study “is a fundamentally different task” than what the researchers did with RCT-DUPLICATE.
In the case of RCT-DUPLICATE, Shirley Wang, associate professor of medicine at Harvard Medical School, explained how the program sought to emulate 30 RCTs using clinical practice data.
The researchers found that 50% of the selected RCTs could be emulated closely regarding design and analysis, and they saw comparable treatment effects. But RCT and RWE findings were more likely to diverge when there were substantive emulation challenges, which could be because the database and RCTs are targeting different questions, or due to bias, or both.
Substantive challenges were seen in the researchers’ attempts to replicate studies in osteoporosis, chronic kidney disease, heart failure, asthma and COPD, Wang said.
However, other trials that could be closely emulated dealt with atrial fibrillation, VTE and hypertension, whereas trials related to diabetes (8 trials) and antiplatelet therapy (3 trials) saw varying degrees of challenges.
“Database studies can come to similar conclusions as RCTs when we are able to emulate them well. But it’s more challenging to emulate trials that are designed with many constraints to show effects under different conditions,” Wang said, noting that it’s easier to emulate trials with more pragmatic design features.
“With data that are fit-for-purpose and proper design and analysis, non-randomized RWE studies can come to similar conclusions about a drug’s treatment effect as randomized trials,” Wang added.
Sebastian Schneeweiss, professor of medicine at Harvard who also discussed results of the study last week, noted, “Most importantly, real-world evidence complements evidence from randomized trials and does not replace it. The conclusion of this demonstration project should not be that every RWE study needs to emulate an RCT. This would defeat the value of RWE in complement with RCTs.”
The FDA-funded Brigham Women’s research is still working on emulating 7 RCTs that are ongoing.
Overall, the FDA’s drug center has supported more than 20 RWE demonstration projections, including ones related to improving the use or quality of RWD, assessing RWE study designs (e.g. RCT-DUPLICATE), or evaluating tools for specific purposes.
Back in 2018, the FDA also unveiled its RWE framework, which has since been supplemented with 4 new draft guidance documents on EHRs and medical claims data, data standards for RWD submissions, assessing registries to support regulatory decisions making, and using RWD and RWE to support regulatory decision-making.
But in explaining how there are still multiple question marks around the use of RWE, FDA’s Concato also highlighted an NEJM article from 2020, in which researchers from the University of Oxford wrote, “The replacement of randomized trials with nonrandomized observational analyses is a false solution to the serious problem of ensuring that patients receive treatments that are both safe and effective.”
Even FDA Commissioner Rob Califf, who wasn’t commissioner at the time, was on board.
Critical summary statement on randomization. https://t.co/98g6onpqjL
— Robert M Califf (@califf001) February 12, 2020