As researchers await comprehensive guidance on the use of real-world evidence (RWE)—and acknowledge the gaps in existing RWE best practice recommendations—many organizations, including regulators, health technology assessment (HTA) bodies, and other decision-makers, are currently developing guidance for RWE use in decision-making. For example, the U.S. Food and Drug Administration (FDA) is set to release guidance in 2021 to fulfill deliverables from its RWE Framework, and the UK’s National Institute for Health and Care Excellence (NICE) is currently working on methods recommendations to realize its five-year strategic vision. While official guidance from decision makers is forthcoming, researchers recently got a glimpse of what future comprehensive guidance may look like.
IMPACT HTA, an initiative funded from the European Union’s (EU’s) Horizon 2020 program, is a collaboration between top EU academic institutions and HTA bodies; notable members include NICE, Sweden’s Dental and Pharmaceutical Benefits Agency (TLV), and the London School of Economics. IMPACT HTA’s goal is to improve methods in HTA to inform decision-making and facilitate collaboration across the EU. IMPACT HTA recently published recommendations for the “the use of nonrandomized evidence to estimate treatment effects in health technology assessment.” Here, we summarize the key takeaways from the recommendations and postulate how future guidance may or may not follow suit.
RWE must be relevant for the research question
Randomized control trials (RCTs) are still the preferred evidence sources on treatment effects. However, decision makers understand that RCTs are not always possible or ethical, and they are open to using RWE as supplemental evidence—as long as there is sound rationale for doing so. IMPACT HTA recommends using established frameworks for justifying the use of observational studies and ensuring that the real-world data (RWD) are identified through a systematic and transparent process. The authors also recommend prospective RWE studies to reduce the risk of selection bias.
Principled database epidemiology: Recommended strategies to study design and analysis
The authors recommend numerous strategies to avoid bias, such as:
- following a “target trial” approach;
- applying new user designs with active comparators;
- adjusting on confounders, which were pre-specified and identified through scientific literature and clinical experts, using methods like propensity score matching;
- using outcomes and measures that have been previously validated or running validation studies prior to analysis; and
- pre-specifying and performing extensive sensitivity analyses, and quantifying the extent of bias and how it could influence the decisions.
Transparency is essential
IMPACT HTA recommends pre-registering the study protocol and using structured reporting templates (like ENCePP and STaRT RWE) to ensure decision makers understand all study design choices made. Putting these processes in place can help guard against concerns that researchers are “cherry picking” results, or only looking for results that support their hypothesis.
The authors also recommend that the data and analytical code be made available to decision makers to ensure replicability of the results and allow for pressure testing of the findings. However, the authors recognize the infrastructural challenges to sharing data and the potential lack of in-house resources to analyze this data. One strategy to improve transparency is using a validated RWE software program or platform which contributes to the reliability, transparency, and reproducibility of RWE.
Recommendations for HTAs: Strengthen infrastructure and invest in resources to design, analyze, and interpret RWE
With the increasing availability of RWD and applicable use cases for RWE, the authors note that decision makers need to both adapt their own data infrastructure internally, and push for updates to the larger data infrastructure ecosystem. Internally, HTAs need to strengthen the scientific advice process to include discussion of RWE studies, and increase their in-house expertise to design, analyze, and interpret RWE studies. The authors also call for a focus on conditional reimbursement with continued evidence development to facilitate the use of RWE to reduce uncertainties post launch. Another important consideration is strengthening the data infrastructure in the EU to ensure relevant data are being captured and to increase access to and interoperability of the data.
Gaps in these recommendations
These recommendations have built upon previous published guidance and recommendations on RWE generation and use, however, there are gaps that need to be filled in order to move toward comprehensive guidance. For example, the authors recommend adjusting on pre-specified confounders, but do not note what is considered good balance between treatment groups. More specific details on “what good looks like” can help researchers ensure they are meeting the decision-makers standards. IMPACT HTA also recommends justifying the use of RWE, but notes it still prefers RCT evidence for the treatment effect. Researchers would benefit from more detail on what “situations” justify the use of RWE and how this justification will be evaluated.
Foreshadowing future guidance from regulators and HTAs
We believe these recommendations are a preview of what to expect from future regulator and HTA guidance on RWE. A similar focus on principled database epidemiology (e.g., specific recommendations on study design strategies, adjusting for confounding, and sensitivity analysis) and transparency are likely to follow from decision makers. We are hopeful that additional specifics on “what good looks like” will also be included in the next iteration of recommendations or guidance.