This week, the final results of the RCT-DUPLICATE (RCT-D) study were published in the Journal of the American Medical Association (JAMA). This omnibus paper, completed in collaboration with Brigham and Women’s Hospital/Harvard Medical School and the US Food and Drug Administration (FDA), takes the results from randomized control trials (RCTs) as a reference standard for valid causal inference and then demonstrates when similar clinical conclusions could have been drawn via the use of real-world data (RWD).
In the following Q&A, Aetion scientists Liz Garry, Head of Scientific Research, and Liza Gibbs, Principal Scientist II, summarize their experience working on the RCT-D project, including how the Aetion Evidence Platform® was employed, some surprises encountered along the way, and what this study might mean for the future adoption of real-world evidence (RWE) for regulatory decision-making.
- What was the primary impetus behind the RCT-D study?
RCT-DUPLICATE was selected as part of a series of demonstration projects to aid FDA in the development of the mandated guidance on use of real-world evidence to support regulatory decision-making. The FDA funded and worked closely with researchers at the Brigham and Women’s Hospital/Harvard Medical School on RCT-DUPLICATE with the aim to replicate the results of 30 completed trials and predict the results of 7 ongoing trials using healthcare databases in order to identify when RWE could provide robust estimates that would lead to the same regulatory decision as each corresponding trial.
- You mentioned that this project resulted from a collaboration between Brigham-Harvard and FDA. What role did Aetion play in supporting this work?
Aetion was brought on early as a collaborative partner, given the additional objective to pilot a process using a RWD analytics platform to enable scalability and transparency for reproducibility of RWE. The Aetion Evidence Platform® (AEP), or what is now referred to as Aetion® Substantiate, was selected. The Platform enabled many efficiency gains to make it possible to implement over 30 complex RWE studies, each in multiple data sources, over a relatively short period of time. The regulatory-recommended implementation transparency of the platform stretches into the study registration at clinicaltrials.com, which refers to specific analyses via URL links to the platform. It also enabled regulators to have direct access to results and the ability to further analyze the data themselves.
A team from Aetion was put together to support the scientists from Brigham and Women’s Hospital who were performing the implementation of the studies. We met routinely to discuss analysis implementation strategy, with much of this time spent aligning on the interpretation of each trial’s inclusion and exclusion criteria and how to use RWD to replicate or reasonably proxy them. Among other things, our team also participated in result readouts to identify areas in which post-hoc sensitivity analyses could be considered, leveraging the Platform to further explore and address additional questions.
- In what ways was the Aetion Evidence Platform® utilized throughout the RCT-D project?
The Platform was used in multiple steps throughout the emulation process, including evaluation of feasibility, such as confirmation of balance for measured covariates between treatment arms after application of propensity score matching, creation of cohorts based on the inclusion and exclusion criteria of the trial, and Cox proportional hazards modeling to evaluate comparative effectiveness or safety.
Other uses included aiding in the development and creation of all necessary measures, or operationalizations of RWD into study concepts. In addition to measures specific to the parameters of each trial (like the treatment and outcome variables), it enabled the creation of a large set of covariates pertaining to baseline characteristics (like comorbidities, medication use, and healthcare utilization) that could be considered in the propensity score models used for matching across all of the studies.
For full transparency, the FDA were also provided with unique web links for each set of results on the Platform, allowing them to view corresponding documentation and audit trails to confirm that the studies were implemented according to the protocols that were publicly posted.
- What were some of the biggest challenges or surprises you faced as you worked to emulate all 32 RCTs?
One of the biggest challenges was the sheer volume and complexity of the inclusion and exclusion criteria that some of the trials had. Most criteria, often more than 30, were objective and could be replicated or reasonably proxied (e.g., diagnosis of hypertension in the absence of blood pressure results). However, there were some surprisingly subjective criteria (e.g., anticipated life expectancy) that relied on the opinion of the enrolling physician. There were also some criteria that required conditional logic (e.g., slightly different criteria depending on age at enrollment).
Another challenge that added to the overall timeline was waiting for the accumulation of more data to achieve an adequate sample size, given the recency of trials selected and the lag in time between capture of medical events in the data and when the data are available for research. The lag and how frequently the data could be updated varied by source.
- How are RCT-DUPLICATE and REPEAT related?
RCT-DUPLICATE, designed to establish the scientific validity of well-conducted real-world evidence studies, is the logical consequence of REPEAT, which examined the reproducibility of RWE studies. REPEAT reproduced 150 published RWE studies in the same data sources using the Aetion Evidence Platform® and found a correlation of r=0.83 between published and reproduced studies. This is encouraging, yet too many studies, often using one-off line programming, did not report key design choices and measurement definitions, which is key to reproducible research and a prerequisite for regulatory-grade RWE.
- How might you expect the findings to impact the industry going forward?
These findings, along with recent guidance documents issued by the FDA, add to a growing body of evidence supporting the use of RWE to inform clinical decisions. Because trials are a known regulatory standard recognized in the broader public health community, using the trial results as a reference and identifying cases when and how RWD could be used to arrive at similar conclusions about effectiveness or safety will help increase the acceptance of RWE to support regulatory decisions. Establishing a standardized process for implementing RWE studies and testing it more than 30 times using the Platform is an important step forward to instill confidence in RWE.
Findings from RCT-DUPLICATE and future emulation efforts will help to gain a clearer understanding of trial parameters that are difficult to emulate in routine, real-world practice, in order to better determine and rationalize when a data source is fit-for-purpose for a research question.
If you would like to connect with one of our scientific experts at Aetion to discuss RCT-DUPLICATE or any of our other demonstration projects, please contact us here.