The publication of the Framework for FDA’s Real-World Evidence Program nearly a year ago launched an industry-wide exploration of methodological best practices to generate regulatory-grade real-world evidence (RWE) that would support efficacy claims of marketed drugs seeking a supplemental approval, as well as investigational new drugs seeking pivotal approval.
In the months since, academia, industry, non-profits, patient advocacy organizations, and other ecosystem players have focused on specific use cases, one of which is leveraging real-world data (RWD) to generate external control arms (ECAs) that contextualize a drug’s efficacy in single arm clinical trials by providing a real-world baseline for comparison.
As former FDA commissioner Scott Gottlieb, M.D., noted at the Bipartisan Policy Center Conference in January 2019, “In appropriate cases, we’ve also accepted RWE to support the evaluation of efficacy in product approvals — using data from registries, natural history studies, and chart reviews — to establish a comparison arm in single arm trials in oncology and rare diseases.”
From clinical development to market access, regulatory-grade ECAs go a long way
ECAs have been a common study design element in oncology and rare disease clinical development programs when randomized controlled trials (RCTs) are not ethical or feasible. In these cases, implementing principled epidemiological methods to generate ECAs that fulfill FDA’s statutory evidence requirements is critical as they enable comparative analyses of clinically-relevant time-to-event endpoints like progression free survival or overall survival, which otherwise cannot be adequately characterized in single-arm trials in oncology.
Upon approval, these efficacy outcomes shape the label, determining the medication’s clinical value proposition, medical marketing strategy, and competitive positioning. Principled real-world studies are therefore critical in generating meaningful, valid, expedited, and transparent RWE not just for regulatory approval, but beyond.
Defining “regulatory-grade RWE” via FDA precedents
While there has been an insightful discourse among RWD/RWE stakeholders on what should qualify as regulatory-grade RWE, there has been less of a systematic effort to assess why RWD did or did not qualify as such in cases where it’s been used in FDA’s decision-making to date. Doing so leads to a deeper understanding of the reasoning behind FDA’s decisions. Aligning with that reasoning when designing real-world studies means they will be more likely to meet FDA’s regulatory bar.
To close this gap in understanding, Aetion has catalogued all FDA approvals where RWD was part of the submission package and where the statistical review by FDA’s Division of Epidemiology (DEPI) was publicly available. In advance of that forthcoming full review in 2020, three recent pivotal approvals in oncology preview the value of studying precedents.
Learnings from three precedents
In each of the cases below, the ECAs did not meet FDA’s statutory evidence requirements. In all of these cases, however, the FDA went on to grant approval for the sought indications, but limited the clinical value proposition on the label to surrogate outcomes available from single arm trials instead, such as tumor overall response rate (ORR). These decisions may therefore restrict the manufacturers’ medical marketing strategy and competitive positioning — a concern if other medications are eventually approved in the same indications based on outcomes that are more clinically relevant. Similarly, a limited label also limits clinicians’ understanding of medications when making important therapeutic decisions that affect patients.
1. Rozlytrek (entrectinib) standard of care external control arm
On August 15, 2019, FDA granted accelerated approval to entrectinib for the treatment of (1) adults and adolescents with NTRK-fusion positive solid tumors, and (2) adults with ROS1-positive metastatic non-small cell lung cancer (mNSCLC), where the submission package of the latter application included RWD sourced from the Flatiron Health Analytic Database. The applicant’s RWD submission was meant to serve as a baseline for comparison of time to treatment discontinuation (TTD), progression-free survival (PFS), and overall survival (OS).
Based on DEPI’s statistical review, FDA did not consider the RWD evidentiary due to methodological limitations, such as post-hoc analysis (i.e., lack of prior review of the study protocol), selection bias (e.g., non-generalizability of real world cohorts), confounding bias (e.g., inadequate covariate adjustment), and lack of statistical power due to limited cohort size. The resulting FDA label of Rozlytrek excluded TTD, PFS, and OS outcomes, and only referenced improvements in ORR.
2. Xpovio (selinexor) natural history external control arm
On July 3, 2019, FDA granted accelerated approval to selinexor for the treatment of adult patients with relapsed or refractory multiple myeloma (RRMM). The submission package included RWD from patients identified in the Flatiron Health Analytic Database, which was meant to serve as a baseline for comparison of OS.
According to DEPI’s statistical review, “the evidence generated from the RWD analysis is not adequate to provide context or comparison for the overall survival observed in the [clinical trial] patients [due to] the lack of comparability between the [clinical trial] and [real world] treatment groups.”
DEPI highlighted post-hoc analysis, selection bias (e.g., differential eligibility criteria for clinical trial vs. external populations, immortal time bias), confounding bias, data missingness, and limited statistical power among the shortcomings of the real world study. The resulting FDA label of Xpovio excluded OS outcomes, and only referenced improvements in ORR.
3. Balversa (erdafitinib) standard of care external control arm
On April 12, 2019, FDA granted accelerated approval to erdafitinib for the treatment of adult patients with locally advanced or metastatic bladder cancer with FGFR2/3 mutations. As part of the submission, the applicant included RWD from the Flatiron-Foundation Medicine, Inc. database, which was meant to serve as a baseline for comparison of OS.
According to the statistical review, “DEPI [did] not consider the study sufficiently valid for supporting regulatory decisions pertaining to drug effectiveness [since] both internal and external validity were threatened by methodological issues.”
Among the methodological issues were confounding bias in study design (e.g., unmeasured confounders), missingness of real-world data points, selection bias and temporal bias in real-world cohort building (e.g., non-contemporaneous external cohort generation), lack of transparency in and misclassification of treatments, and lack of statistical power due to small real world cohort size. The resulting FDA label of Balversa excluded OS outcomes, and only referenced improvements in ORR.
Pitfalls to avoid
In all three RWD precedents deemed not supportive of regulatory decision-making, the following methodological issues were causes cited by FDA that could have been avoided by a principled epidemiological approach:
- Post-hoc analyses,
- Selection bias (including improper control selection),
- Immortal time bias,
- Confounding bias, and
- Lack of transparency / misclassification of treatment definitions.
Issues intrinsic to RWD itself were also at play in these decisions, and highlight the importance of finding fit-for-purpose data sources. These include:
- Data missingness, and
- Lack of statistical power due to insufficient real world cohort size.
Until FDA finalizes their detailed RWE guidance by 2021, publicly available FDA statistical reviews of RWD submissions provide a valuable resource for identifying real-world study best practices. This is why Aetion conducted a systematic review of all RWD/RWE precedents in global regulatory decisions to help inform RWE strategies of all stakeholders. We look forward to sharing those findings on a continued basis.