As FDA intensifies its focus on real-world evidence (RWE) and the work to establish regulatory-grade study designs and methods, researchers are beginning to register their observational database studies on ClinicalTrials.gov and other registries to ensure transparency and validity equivalent to traditional randomized controlled trials (RCTs). Will RWE trials eventually require a separate registry? ISPOR has formed a working group to consider that question. We ask Richard J. Willke, Ph.D., chief science officer for ISPOR—The Professional Society for Health Economics and Outcomes Research, about the plans of this working group and why a registry specific to real-world data studies may—or may not—be needed.
Q: Why are existing registries not a good fit for real-world evidence studies?
A: There are multiple registries—ClinicalTrials.gov, the EU Post-Authorization Study Register (aka ENCePP), HSRProj, and so on. ClinicalTrials.gov, the most frequently used here in the U.S., does allow for registration of observational studies, but each were created for somewhat different purposes and thus have terminology and a level of detail that are not optimized for real-world evidence studies. That’s a deterrent to regular use by those designing and carrying out observational studies. In addition, when those studies are mixed in with clinical trials in the registry, combining the two in a single registry may confuse some consumers of the database. So, while it is being used for observational research, the registry is not a good fit for either those who registering their real-world studies in it or those who are seeking information from it.
But it’s not far off. So, representatives of our working group plan to meet with the folks at ClinicalTrials.gov see if it may be possible to modify their template so that it can be a better fit for observational data studies. The European Union Post-Authorization Studies (EU PAS) Register is probably a bit closer fit for these studies—and we will speak with that group as well as we intend this to be a global effort.
Q: So, it may not ultimately be necessary to launch a new registry?
A: At this point, we do not plan to start our own registry. We’d like to work with existing registries to see if there is a template that might work for both of them that would be more fit for purpose and more recognizable as observational data studies meant for comparative effectiveness and/or comparative safety. Early adopters who want to be ahead of the curve can go ahead and register their protocol for observational studies on existing registries. But what we’re trying to do is find something that will work well for a broad swath of users.
Q: What are the exact deterrents at, for example, ClinicalTrials.gov in registering real-world protocols?
A: Take one example: The checklist for submission includes a start date for trials, defined as when the first patient is enrolled. Translating that start date to an observational study is different, and it’s not clear how to do so. There are other instances of terms that need to be defined through the lens of an observational study.
Q: If a separate registry specific to real-world data studies is established, could it help standardize methodologies?
A: That’s a really good question. If you’re talking about analytic methodologies, I think it does become another reference for those who are thinking about how they will analyze a given data set and overcome the potential biases or other issues that may exist. It becomes another reference for people to see what’s becoming usual practice. I almost said “accepted,” but it’s maybe a stretch from saying what’s registered has been accepted. At least, however, it’s an indicator of usual practice.
There is a lot of methodological guidance already available. And the field has settled on a few basic approaches that are pretty well accepted. They each have their own advantages and disadvantages, and biases that they address or don’t address. So, we don’t yet have a standard methodology. But a registry could become another reference for those who are developing methodologies.
Q: How would a separate registry support transparency in real-world evidence?
A: In working with real-world data, the question is: How much have you looked at the data you’re going to analyze before you register your protocol? That’s where a real lack of trust happens. Most researchers want to take a quick look at their database to start with; you’ve got to know the population, have a sense of the presence of the data, what’s there and what’s not there to answer the research question. But a quick look at your database before you begin the study is quite different than actually running some regressions or otherwise taking a close look at the hypothesis you’re testing.
So, part of what the registration process needs to be—and what our working group will be very serious about—is when you register your protocol, you’ll need to be transparent about how much you have looked at your data up to the point where you register it. That will be essential to creating transparency and trust.
Q: Whom does the working group consider stakeholders in their process?
A: The evidence generators and the evidence users, of course. The evidence generators are the life sciences industry—pharma and devices—and the academics who are applying for government grants and that kind of thing. Payers or nonprofits of various sorts are also thinking about using it if they want to see their work recognized as being done at the highest standard.
The evidence users, then, are the regulators, who have their own set of standards. But we understand that they have a higher comfort level in their responsibility to the public knowing that studies have been posted publicly because doing so supports the transparency of how they make decisions and the basis for those decisions. That’s one of the driving forces for our work: The FDA and the EMA’s increasing interest in—and mandate—to incorporate real-world evidence in some of their decision-making. So, we would want to make the registry acceptable to the regulatory folks.
Payers and health technology assessment agencies are also stakeholders. They’re both generators and users; sometimes they do similar work themselves, sometimes they want to look at the work that’s been done.
Also, of course, ClinicalTrials.gov is used by patients, another stakeholder. We would want to accompany any new registry with interpretive guidance for patients who may or may not be sophisticated in looking at real-world evidence results.
Q: Are you thinking in particular of rare-disease patients?
A: They’re certainly one, yes. That’s an early use at FDA in which real-world evidence does make a difference in their regulatory decisions. And there are some cases out there where they’ve used real-patient natural history studies as some comparative evidence for their approval processes.
Q: How could a separate registry impact publication in journals?
A: Most major journals require randomized trials to have been published on ClinicalTrials.gov before they publish those studies. Not all journals do, but you do see it commonly in highly respected medical journals. So, among our stakeholder group are definitely the medical editors, and we’re considering what they’d like to see and how they would think about using it. It would certainly be easier for journal editors to tell what they’re looking at if real-world studies were not mixed in with clinical research trials.
Q: Besides the stakeholders you’ve listed, is your group working in collaboration with any other organizations?
A: We certainly are. ISPE, the International Society for Pharmacoepidemiology, is definitely one. Another is the Duke-Margolis Center for Health, which has done a lot of work in this space, and we’ve been part of their collaboration. We haven’t established a formal partnership, but we invited a lot of these folks to a summit recently, including representatives from FDA and EMA, from industry, and from researcher organizations that work a lot with industry. We had a couple of patient representatives, two or three representatives from health technology assessment agencies, and at least one person from a payer group. A representative from the National Pharmaceutical Council attended as they have been interested in this approach for a while. A number of groups with different audiences intersect and we are all trying to move forward together in this space, which we think Is critical if a registry is going to happen.
Q: What were some of the issues raised at the summit? Will there be a report summarizing it?
A: We’re planning to translate those proceedings into a white paper and have just started working on that. It will be publicly available to our membership and anybody else for general comment. Timing of publication, of course, relies on alignment from all that are involved and that can take some time. But we hope to have it available within a few months.
The paper will be a general expression of where we see this long-term effort going and what we see as next steps, such as presentations at several conferences, discussions with those responsible for other registries as I mentioned earlier, and, perhaps, creation of a task force to address some specific aspects. There’ll be some questions we need to answer. For example: What exactly should a template look like? Confidentiality will also be a consideration. There will be cases in which it is very important, at least for a period because, once a protocol for a database study is established and registered, an experienced researcher could take that protocol and, if the database is publicly available, do it in a few weeks. Maybe a few days. So, there is researcher interest and commercial interest—it works on both sides—in having a proprietary time period where you can do the study before anyone else jumps the gun on you. So, we’ve got to figure that out—and we have some thoughts about how to do it.
Q: If a registry is built, will there be a testing period?
A: Yes, we’ll do a pilot to get some folks using it before we open it up to for everybody’s use to be certain that it is fit for purpose and works as we hope. A bad start’s a bad start, and we want to avoid that.
Q: Do you have a rough expectation for a timeline for completing this process?
A: Working groups like this one—and the subsequent efforts that it will likely spawn—don’t get their work done overnight. I would like to get a pilot out there in a couple of years, but I don’t know if that’s realistic. I can see it requiring four or five years before we say, “this is ready for prime time.”
Q: Meanwhile, until it happens, do you encourage researchers working with real-world data to register their protocols on ClinicalTrials.gov?
A: Yes, it can be done. And the more experience researchers have with registering RWE studies there, the more knowledgeably they can react to and comment on any newer version that we propose—and that would be very helpful.