Both real-world data (RWD) and real-world evidence (RWE) studies continue to effectively inform regulators, payers, and clinical decision-makers in their healthcare decisions. At the same time, we also recognize the need to accurately replicate these studies in order to continue strengthening overall confidence in using evidence derived from observational studies. One strategy critical to the generation of reliable RWE is transparency at every step of the process, including data processing, study design, and analytical choices.
In a study published in Pharmacoepidemiology and Drug Safety, Aetion scientists working in collaboration with Amgen, evaluated the reproducibility of a study characterizing newly diagnosed multiple myeloma (MM) patients within an electronic health records (EHR) database. We sat down with author Shannon Reynolds to discuss the two analytical methods used in this study, as well as some key takeaways.
Methods
For our study, we replicated the results of a descriptive cohort study of newly diagnosed multiple myeloma (MM) patients with a two-phase approach. In Phase I, we used both the Aetion Evidence Platform® (now referred to as Aetion® Substantiate) and SAS statistical software to analyze data from the UK Clinical Practice Research Datalink (CPRD) – one of the most commonly used UK-based RWD sources. For Phase II, we refined the analysis based on initial comparisons and assessed the reproducibility of findings by calculating match rates and absolute differences between the Substantiate and SAS results.
Results
In Phase I, the two different analytic approaches produced nearly identical results for the number of eligible patients with newly diagnosed multiple myeloma, although there were slight discrepancies in patient demographics, comorbidities, clinical characteristics, drug exposure, and laboratory investigations. These discrepancies resulted primarily from differences in data processing and modeling between the two independent investigator teams working on each of the two analytical methods.
In Phase II, however, the discrepancies identified in Phase I were largely resolved after amendments were made to the initial statistical analysis plan (SAP), thus ensuring more consistency between the two database studies. In fact, these studies were ultimately able to achieve a 100% match in the analytic population and nearly a 100% match rate in various study parameters.
Key Takeaways
This direct replication study focusing on patients with newly diagnosed MM in the CPRD underscores the critical importance of transparency in protocol/SAP development, study design, and reporting. Stated another way, study reproducibility hinges on maximum transparency throughout all steps of the real-world evidence generation process. By adhering to standard protocols and reporting guidelines, we found that a rapid-cycle analytics tool like Aetion Substantiate could almost perfectly replicate results obtained with traditional statistical software, underscoring the importance of clear study design and shared operational decisions among independent investigator teams for successful reproducibility.
If you would like to learn more about our oncology research here at Aetion, or if you would like to talk to one of our experts, you can contact us here.