The RCT-DUPLICATE project began in 2018 with the goal of informing the use of real-world evidence (RWE) for regulatory decision-making. Now, interim results published in Circulation show that RWE studies—designed according to principled database epidemiology—can meet regulatory standards for active comparator trials.
To refresh, RCT-DUPLICATE aims to emulate the results of 30 completed randomized controlled trials (RCTs), and predict the results of seven ongoing phase 4 trials, using RWE. The project, which uses Aetion Evidence Platform® for its analyses, is a collaborative effort between Brigham and Women’s Hospital, the FDA, and Aetion. Learn more about the methods and study design choices for the project here.
The interim results paper shares the results of 10 of the RWE studies: Nine of the 10 achieved either regulatory or estimate agreement with RCTs, and of those, eight achieved estimate agreement (meaning the RWE effect estimate was within the 95 percent confidence interval of the trial), and six achieved regulatory agreement (meaning they reached identical conclusions with respect to statistical significance as in the RCT). Three of the four that did not reach regulatory agreement were placebo-controlled trials, which highlights the challenges of emulating these types of trials in RWE.
Learn more about the interim results of the RCT-DUPLICATE demonstration project, and how biopharma should interpret them, from the project’s principal investigator, Jessica Franklin, Ph.D.
Responses have been edited for clarity and length.
Q: Can you please share the progress to date on RCT-DUPLICATE?
A: Of the 30 trials we’re emulating, 14 are now complete, and we’ve delivered results to the FDA. We have another 12 that are at various stages of progress—from developing a protocol to fine tuning analysis plans. That will take us up to 26 out of 30 trial emulations. The plan is to complete all 30 emulations next summer.
Q: How do you recommend that industry interpret the interim results?
A: So far, we’re only reporting on findings from the first 10 trial emulations, which take place within two therapeutic areas: cardiovascular and diabetes. While it’s too early to come away with strong conclusions, we have seen so far that we’ve had an easier time emulating active comparator trials than placebo comparator trials. This is in line with our expectations, as there is no real-life analogue for placebo control.
Q: What are some other recurring themes or challenges that you and your team have found thus far?
A: For one, we’ve seen that results can look quite different from one therapeutic area to the next. We’ve also seen that differences in design between RCTs and observational studies can create challenges in emulation. And some of the biggest challenges happen when we’re trying to emulate design choices from an RCT that don’t make sense in the context of a database study.
For example, some of the trials have complicated drug washout or run-in periods for testing adherence, which make sense in the context of an RCT, but not in an observational study. It’s been challenging trying to figure out what we can emulate and what we can’t.
We’ve also “dropped” more trials than we were expecting after we decided that we wouldn’t be able to emulate them in our databases. We started with a list of 40 studies to emulate, thinking we’d drop, at most, 10 of them. But once we started evaluating the trials in our data, we realized that we didn’t have enough statistical power to emulate many of them. For example, if a drug is approved based on a 10,000 person RCT, but it isn’t frequently used in routine care and therefore doesn’t have much real-world data (RWD) available, it can be difficult to emulate that trial in our analysis.
As a result, we’ve supplemented our original list of 40 trials with additional ones we think could be good candidates, which may change the balance of supplemental approval to primary approval trials that we end up with in the final results.
Q: Is there enough information included in the publication that other investigators could go deeper into study design choices?
A: Yes, we’ve tried to be as transparent as possible in providing our protocols on ClinicalTrials.gov, in hopes that other investigators will look at our protocol and think, perhaps, “I would have done things a bit differently.” They could then go into their own database and build a study based on our protocol with their proposed changes. We could all learn from the findings of such studies, especially if their study achieved results that were closer to the RCT.
Q: What milestones are still ahead for the project, and what is the predicted timeline?
A: There are lots of milestones ahead, though we will likely not publish additional results until the full set of 30 emulation studies are complete. We expect to finish next summer, so results will be published hopefully next fall.
Over the next year, we’ll focus more on the prediction studies. We’ll publish those results as we perform the emulations, hopefully sharing our results ahead of the RCT results. We expect to publish those over the next two years, finishing in mid-2022.
Q: What do you hope to learn in the work ahead?
A: We hope to learn more about which therapeutic areas are most amenable to database studies. We’ve studied two so far, but the remaining trials will look at several more therapeutic areas, which will give us a better sense of which are best suited to observational research.
We also hope to learn more about the methodological decisions that lead to strong agreement between RCTs and RWE emulation. In each of these studies, we run sensitivity analyses after our primary analyses to understand whether some reach closer results to the RCT than others, and hopefully gain a better sense of the most suitable design choices.
Q: What implications do this project and the interim findings have on regulatory decision-making in the era of COVID-19?
A: RCT-DUPLICATE definitely has relevance to the COVID era. We’ve seen an explosion in observational database studies on COVID-19—many of them, like the Surgisphere study, highly questionable in terms of data provenance and methods. I hope that this project is a reminder that when we use well-developed epidemiologic design principles, much of the heterogeneity in the literature subsides.
For regulatory decision-making, I hope that RCT-DUPLICATE can help inform the scenarios where an observational database study can be useful for providing evidence on effectiveness. We need more research before we can say to what extent COVID-19 therapies can be evaluated rigorously in a database, but this project can hopefully help to point us to which clinical areas can be evaluated in such a way.
Q: Are there any plans to expand the replication and prediction efforts into other RWD sources or therapeutic areas?
A: There are lots of plans to continue to expand this work. All of our work thus far has been in claims databases, and we have plans to expand into linked claims and electronic health record (EHR) data. Of course, EHR data has much greater clinical richness compared to claims, so our hope is that these data would allow us to emulate trials that we previously could not in a claims-only data source, or perhaps allow us to better emulate some of the trials we’re already working on in claims data.
As we expand our data sources, we hope to expand into other therapeutic areas. For example, we hadn’t previously studied oncology trials because we felt they couldn’t be emulated well in claims data alone. In rheumatoid arthritis, we couldn’t emulate trials at all in claims if they had a patient-reported outcome as the primary outcome. With EHRs or a rheumatoid arthritis registry, we can start to emulate these trials.