As global regulatory and health technology assessment (HTA) bodies continue to iterate on their approaches to real-world evidence (RWE) generation and use, demonstration projects have an important role to play in informing standards. These efforts create opportunities for multi-stakeholder groups to collaborate and evaluate questions about when, how, and where RWE can support decision-making, ultimately helping to inform future guidance.
In a recent paper published in the Journal of Comparative Effectiveness Research, Ashley Jaksa, MPH, and Nirosha Mahendraratnam Lederer, PhD, summarize learnings from a range of RWE demonstration projects. Their takeaways focus on real-world data (RWD) quality, the importance of good study design, and how to broaden RWE adoption worldwide. They also propose priority areas for demonstration projects to help industry further advance RWE.
Ashley (AJ) and Nirosha (NML) expand on these topics below.
Responses have been edited for clarity and length.
Q: What is the value of a demonstration project? How do they fit into the RWE guidance development landscape?
AJ: The value is in the ability to empirically test a concept, method, or research question to help fill a gap in guidance or understanding. They’re different from case studies because the aims are broader, which allows you to iterate and define your research question. For example, while you can learn a lot from a case study of an external control arm submitted to FDA, the learnings only apply to the specific situation or approval. Demonstration projects are useful in testing gaps in methodology or recommendations, then figuring out how to fill those gaps.
NML: In addition, there are sometimes areas of innovation where sponsors may be hesitant to take risks due to the novelty of the scenario. When something is tested under a pilot project, participants can focus on the science and the problem at hand without worrying about the other considerations that can constrain innovation. Pilot projects are also often structured in a pre-competitive way, so they allow collaboration that isn’t typically part of the drug development process.
Q: What are some key demonstration projects going on now? What questions do they seek to address?
AJ: One interesting project is RCT DUPLICATE, in which FDA, Brigham and Women’s Hospital, and Aetion are replicating 30 completed randomized controlled trials (RCTs) and predicting the results of seven ongoing trials with RWE to see if they reach similar results. This is an important effort to help FDA understand where and how RWE studies can inform its decision-making—and where they can’t.
Another is Aetion’s collaboration with the UK’s National Institute for Health and Care Excellence (NICE) to explore the use of RWE for comparative effectiveness studies. It’s great to see an HTA body directly address the use of comparative effectiveness RWE studies in its decision-making to inform clinical guidance.
NML: There are a lot of ongoing demonstration projects with similar aims—for example, clinical trial emulation projects using RWE—and some may wonder if these efforts are duplicative. Do we really need multiple pilots focused on replication? The answer is yes. That’s part of the scientific process, and every type of project may take a different approach to address the problem at hand, or focus on a different disease area. Each of those learnings are important to moving towards best practices and recommendations for guidance. For example, a recent Duke-Margolis meeting reviewed three of the largest RWE replication projects. The group found that good study design and data fitness—not analytics—are essential to being able to produce valid and reliable results.
Q: What have demonstration projects revealed about the importance of data reliability?
AJ: Your data has to be fit for purpose—that’s at least 50 percent of the RWE ballgame. If your data is not fit for purpose, there’s no sense in you passing go. That in itself is one of the key learnings from a lot of these demonstration projects: we have to have the right data, and there is still a lot we can do to increase and improve the foundation of data we have.
NML: I would go so far as saying that data fitness is the ballgame. No amount of advanced analytics can fix bad data. This brings up questions of how we can improve data collection at the point of entry, because there’s a lot of interest in the ways we curate data to be fit for purpose.
One item that will help address questions related to data fitness is FDA’s recently released draft guidance: “Real-world data: Assessing electronic health records and medical claims data to support regulatory decision-making for drug and biological products.”
AJ: That type of guidance is informed by some of these demonstration projects, including RCT DUPLICATE and innovative, multi-stakeholder pilots led by groups like Duke-Margolis. This puts the idea of demonstration projects into practice: we do the projects, learn from them, then study and implement best practices. And it’s only going to increase, we’re only going to get more guidance from the European Medicines Agency (EMA), FDA, NICE, and others. I think we’re on the verge of having a lot more information about how RWE can be used in decision-making.
Q: What are some of the key study design elements that consistently led to high quality evidence in the demonstration projects you evaluated?
NML: We can’t say it enough, the key is good study design. Researchers can use high quality templates like STaRT-RWE and other tools rooted in principled database epidemiology to help them design better studies, and to mitigate reviewers’ concerns around data dredging, for example. Implementing study design best practices will go a long way.
AJ: Transparency is another huge component. It’s also important when designing RWE studies to make sure the research question can be answered in RWD as opposed to an RCT. Using frameworks like SPACE or target trials is fundamental to helping figure out the correct study design.
Q: In the paper, you proposed five areas for further investment in RWE demonstration projects to continue to advance work in this space: developing a consensus-based research agenda, building out RWD and RWE infrastructure, developing a standardized process for validating RWD, adopting current best practices for critical appraisal of RWE, and expanding on the appropriate use of RWD and RWE in decision-making. Which of those do you feel like industry is best positioned to address first? And how can they mobilize to make it happen?
AJ: Nirosha mentioned earlier that demonstration projects are a “safe space” for innovation—no one wants to take all the risk of doing an RWE study in a novel way that may not be accepted by a regulator. But that’s one of the areas where industry has a good opportunity to design demonstration projects and collaborate with smart people to execute on them. They can engage in above-brand projects, so they’re not specifically focused on their own drugs or pipelines. That’s where biopharma has a real opportunity to help set the standards and address key questions.
There are gaps in the recommendations we’ve seen from decision makers thus far, and we’re going to need all stakeholders to collaborate to fill them. Biopharma can take the gaps that are most relevant to them and design above-brand demonstration projects to push the science forward and get their name out there as helping set standards.
NML: Biopharma has a role, actually, in all of these. We need to hear from sponsors as researchers on what should go into a consensus-driven research agenda, because everyone has a different perspective. But I think the lowest hanging fruit is to assess the best practices that are available today to understand how feasible they are to address, then provide that feedback to regulators. When it’s possible to implement one of those best practices, biopharma can set up processes to embed them in their everyday research processes as well as in regulatory submissions.
Aetion has seen the value of these standard-setting initiatives through the demonstration projects we’re involved in now, including RCT DUPLICATE and our COVID-19 research with FDA, the comparative effectiveness work with NICE, and a value reassessment pilot with the Institute for Clinical and Economic Review (ICER). Overall, these efforts present a great opportunity to bring multiple perspectives together to advance the science of RWE.