During the last 2 decades, there has been a significant shift towards the use of real-world evidence (RWE) to reinforce the pool of evidence for medical products aiming for marketing authorization and regulatory reimbursement. Health Technology Assessment (HTA) organizations have been pioneers in the embracement of RWE, mainly using RWE for descriptive analyses (e.g., treatment patterns), collection and interpretation of epidemiologic data, and monitoring the safety of marketed medical products [1, 2, 3, 4]. Despite this, competent authorities and decision makers still greatly rely on conventional evidence collected from randomized controlled trials (RCTs) (see Figure 1) [1, 4].
The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force differentiates RWE and real-world data (RWD) by stating :
[…]The notion was that data conjures the idea of simple factual information, whereas evidence connotes the organization of the information to inform a conclusion or judgment. Evidence is generated according to a research plan and interpreted accordingly, whereas data is but one component of the research plan. Evidence is shaped, while data simply are raw materials and alone are noninformative. […]
RWE can play an important complementary role to RCTs by [6, 7]:
However, regional differences in the definitions, scoping, and potential applications of RWE result in a vague and diverse RWE regulatory framework, as well as in delayed embedment of RWE into clinical development, regulatory processes, and health economics . For this reason, use of RWE is essentially limited to clinical development and evaluation of pharmaceuticals, assisting decision-making within pharmacovigilance and post-marketing research, and evaluating clinical treatments. Fortunately, attention is growing regarding the generation and adoption of RWD by local Competent Authorities and with regards to the incorporation of RWE in the lifecycle of medical devices (MDs). This is particularly evident under the light of the latest EU Regulations, which significantly increases the requirements for the proactive and continuous collection and evaluation of clinical data from the real-world use of a MD .
The United States (US) and Europe have been accumulating practical experience with RWD longer than other regions of the world, as reflected in their available guidance and in the growing number of regulatory approvals based on RWE. Two recent reviews have tried to quantify and assess the contribution of RWE in the approval of drug products by the Federal Drugs Administration (FDA) and the European Medicines Agency (EMA). Purpura et al. evaluated FDA public resources for New Drug Applications (NDAs) and Biological License Applications (BLAs) approved by the FDA from 2019–2021 . The authors found that out of 136 approvals, 116 (85%) included RWE, and the number of approvals that included a RWE study increased significantly between 2019 to 2021 (from 38/51 (75%) approvals in 2019 to 53/59 (90%) in 2020, and 25/26 (96%) in the first half of 2021). Of these studies, 88/136 (65%) approvals (spanning 16 therapeutic areas) used RWE studies with the intent to provide evidence of safety or effectiveness, and 83/136 (61%) used RWE studies with the intent to provide therapeutic context. Interestingly, the FDA provided publicly available feedback on the RWE studies in 37/88 (42%) approvals, identifying methodological issues (n = 23), sample size concerns (n = 8), omission of patient-level data (n = 3), amongst other limitations (n = 13). Flynn et al. performed a similar study within the European regulatory context . 158 Marketing Authorization Applications (MAAs) and 153 Extension of Indications (EOIs) applications submitted to EMA during 2018–2019 were assessed (after excluding generic products and well-established use applications). For MAAs, 63/158 products (39.9%) included RWE. For 31.7% of these products, the RWE submitted was obtained in the pre-authorization phase and was intended to support safety (87.3%) and efficacy (49.2%). The most common RWD sources were registries (60.3%) and hospital data (31.7%). For EOIs, 28/153 products (18.3%) contained RWE. For 57.1% of these products, studies were conducted prior to the EOIs to support safety (82.1%) and efficacy (53.6%), with RWE sources coming mainly from registries (35.6%) and hospital data (27.0%).
Both studies highlight the fact that, despite the vivid discussions around the use of RWD/RWE in regulatory decision-making, operational, technical, and methodological restraints prevent their actual incorporation into everyday regulatory practice. This is mainly because of the heterogeneity of data sources, low level of data quality and validity, and the likelihood of bias due to unblinded, uncontrolled, or non-randomized treatment allocation (see Figure 2) .
Consistent with the above, Arondekar et al. reported on 133 approvals for oncology NDAs and BLAs submitted to the FDA from 2015–2020. They found that 11 (8.3%) included RWE in support of efficacy, with an average time of 5.7 years from Investigational New Drug Applications (IND) submission to approval . The 11 submissions that included RWE were for avelumab, axicabtagene ciloleucel, entrectinib, erdafitinib, polatuzumab vedotin-piiq, selinexor, avapritinib, capmatinib, tafasitamab, and tazemetostat (NDAs 211723 and 213400), and the 2 NDAs/BLAs were for blinatumomab and palbociclib. Indications were often rare diseases with an underlying, high, unmet need, while the most common RWD source was chart reviews from clinical sites. Similarly to Purpura and Flynn, the FDA’s feedback pertained to inherent sources of bias in the use of historical controls and RWD, such as selection bias and residual or unobserved confounding variables (especially missing data on key covariates), as well as different outcome assessment methods and frequency of measures as compared with controlled trials, lack of comparability between external controls and trial populations, misclassification of outcomes, and insufficient statistical methods for the adjustment of differences between comparator groups [13, 14, 15, 16].
The aim of this article is to summarize the currently available guidances, frameworks, and initiatives (see Tables 1, 2 and 3) for the collection, management, and interpretation of RWD/RWE around the globe, up until September 2022. Furthermore, to underline the necessary steps for international harmonization and standardization of guidelines/frameworks, not only in the post-approval stages of a medicine’s lifecycle, but mainly in the development and lifelong post-market surveillance of MDs.
|USA||FDA||Guidance: Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drug and Biological Products|||
|Framework for FDA’s Real-World Evidence Program|||
|Guidance: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products|
|Guidance: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products|||
|Guidance: Considerations for the Use of Real-World Data and Real-World Evidence To Support Regulatory Decision-Making for Drug and Biological Products|||
|Guidance: Data Standards for Drug and Biological Product Submissions Containing Real-World Data|||
|Guidance: Submitting Documents Utilizing Real-World Data and Real-World Evidence to FDA for Drugs and Biologics|||
|Guidance: Use of Electronic Health Records in Clinical Investigations|||
|Guidance: Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices|||
|Publication 2020: Randomized, observational, interventional, and real-world- What’s in a name?|||
|Publication 2022: Real-World Evidence- Where Are We Now?|||
|Europe||EMA||Operational, Technical, and Methodological (OPTIMAL) framework for regulatory use of RWE in regulatory decision making|||
|Regulatory Science to 2025 strategic document|||
|UK||MHRA||MHRA guidance on the use of real-world data in clinical studies to support regulatory decisions|||
|MHRA guideline on randomised controlled trials using real-world data to support regulatory decisions|||
|NICE||NICE real-world evidence framework|||
|Australia||TGA||Real world evidence and patient reported outcomes|||
|Clinical evidence guidelines for medical devices|||
|An Action Plan for Medical Devices|||
|Canada||Sante Canada-Health Canada||Optimizing the Use of Real-World Evidence to Inform Regulatory Decision-Making|||
|Elements of Real-World Data/Evidence Quality throughout the Prescription Drug Product Life Cycle|||
|CADTH||Real-World Evidence for Decision-Making|||
|Greater China||NMPA||Guideline on using real-world evidence to support drug research & development and review|||
|Technical guidelines (trial) for real-world research and support for drug research and development and review of children|||
|Guideline on using real-world evidence to support medical device evaluation (Trial)|||
|Guideline on using real-world data to generate real-world evidence (trial)|||
|TFDA||Basic considerations for real-world evidence supporting drug development|||
|Japan||RWD Working Group of PMDA||Guidelines for the Conduct of Pharmacoepidemiological Studies in Drug Safety Assessment with Medical Information Databases|||
|Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Drugs|||
|Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Medical Devices|||
|Procedures for Developing Post-marketing Study Plan (originally published as “Procedures for Developing Post-marketing Study Plan|||
|Questions and Answers (Q&A) on Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Drugs|||
|Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Regenerative Medical Products|||
|Basic Principles on Utilization of Registry for Applications|||
|Points to consider for Ensuring the Reliability in Utilization of Registry Data for Applications|||
|REAL World Data In Asia for Health Technology Assessment in Reimbursement (REALISE) working group||A framework for the use of RWD and RWE in decision-making in Asia, which is designed to be adapted to users’ local needs, reflecting an awareness of the differing practical barriers occurring in different countries|||
|Duke-Margolis Center for Health Policy. Developing real-world data and evidence to support regulatory decision-making.||Cluster of stakeholders, which has released a number of whitepapers, including a suggested regulatory framework for the use of RWD and RWE in decision-making in the USA|||
|HTx Next Generation Health Technology Assessment||A European Union (Horizon 2020) funded program monitoring the RWE use for the decision-making process throughout Europe, aiming to construct the future Framework for the “Next Generation Health Technology Assessment (HTA) and to enable the decision-making process to rely on patient-centred evidence, real-time, and socially oriented reimbursement policies in Europe|||
|INNOVATIVE MEDICINES INITIATIVE’S COLLABORATIVE RESEARCH PROJECTS |
|Clinical Trials Transformation Initiative||Initiative aiming to modernize clinical trials, which has released a position paper on accelerating the use of RWD in clinical trials|||
|Europe’s Innovative Medicines Initiative’s GetReal project||Initiative aiming to incorporate data from real-life clinical settings into drug development|||
|RCT DUPLICATE (Randomized Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) initiative||Led by Brigham and Women’s Hospital in collaboration with the FDA and other academic and industry stakeholders, it is engaged in replicating large-scale RCTs using RWD sources to evaluate the latter’s ability to replicate findings from RCTs and validate findings for RWE acceptance|||
|ADAPT-SMART (Accelerated Development of Appropriate Patient Therapies: A Sustainable, Multi-Stakeholder Approach From Research to Treatment Outcomes)||Project to the EMA’s Adaptive Pathways Pilot and the Medicines Adaptive Pathway to Patients concept. ADAPT-SMART generates evidence throughout the product life cycle and develops methods for adjusting for biases|||
|Big Data for Better Outcomes initiative||European research programme aiming to develop enablers to support health care system transformation through the use of big data. The initiative has developed platforms for integrating and analysing diverse real-world data sets|||
|International Council for Harmonisation (ICH)||ICH has published a reflection paper on Good Clinical Practice and put forth plans to update the existing E8 (General Considerations for Clinical Trials) and the E6 (Guideline for Good Clinical Practice) guidelines to leverage data from more flexible study designs and a diversity of data sources. In particular, the ICH proposed to include discussion on pragmatic study designs and guidance on how RWD collection could be used to supplement or even replace traditional data collection within the E6|||
|European Health Data & Evidence Network||European consortium aiming to harmonize health records to the Observational Medical Outcomes Partnership data model and create an EU-wide architecture for federated analysis of RWD|||
|Council for International Organizations of Medical Sciences (CIOMS) - Working Group XIII – Real-World Data and Real-World Evidence in Regulatory Decision Making||The primary goal of the proposed CIOMS WG is to develop, for global use, a consensus report and recommendations on principles to be applied regarding triggers, objectives, research questions, design features, and timing of RWD and RWE as part of the regulatory process for products in the peri-approval stage of development or for authorized products|||
|International Society for Pharmacoeconomics and Outcomes Research (ISPOR); Real World Evidence Strategic Initiative||Working to improve standards and practice for the collection and analysis of RWD.
4 Joint International Society for Pharmacoepidemiology (ISPE) -ISPOR Good Practices Reports have been published
Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Healthcare Decision Making
Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0
Making Real-World Evidence More Useful for Decision Making (editorial)
All Good Practices Reports for Real-World Data
|[113, 114, 115, 116, 117, 118]|
|International Coalition of Medicines Regulatory Authorities (ICMRA)||During a 2020 ICMRA working group meeting on building international cohorts, for example, the EMA, FDA, Agencia Espanola de Medicamentos y Productos Sanitaros, and Health Canada worked together to develop criteria to help prioritize key regulatory and public health research questions for international collaboration (e.g., large sample size, regional comparisons, and development of infrastructure)|||
|International Network of Agencies for Health Technology Assessment (INAHTA)||INAHTA is a network of 50 HTA agencies that support health system decision-making, focusing on the sharing of information about producing and disseminating HTA reports for evidence-based decision making|||
|International Society for Pharmaceutical Engineering (ISPE)||The International Society for Pharmaceutical Engineering is a non-profit association serving its members by leading scientific, technical, and regulatory advancement throughout the entire pharmaceutical lifecycle and has issued a position paper on the use of RWE|||
|AUTHORITY||DEFINITION RWD||DEFINITION RWE||SOURCES OF DATA||MAJOR OUTCOMES||PERSONAL DATA CONCERNS||REIMBURSEMENT|
|FDA||Data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.||Clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.||
||RWD sources are usually developed for non-regulatory purposes (to document care in the case of EHRs or to submit insurance claims for reimbursement in administrative and claims data)
Medical Administrative Claims Data—“Claims data arise from a person’s use of the health care system and reimbursement of health care providers for that care]
collected data relating to a patient’s health
status or the delivery of health care from
a variety of sources other than traditional
|The information derived from analysis of RWD
Data that are collected outside the constraints of conventional randomised clinical trials
|MHRA||RWD is defined as data relating to patient health status or delivery of health care collected outside of a clinical study. Sources of RWD include electronic healthcare records (EHR) defined as structured, digital collections of patient level medical data, primary and secondary care records, disease registries, and administrative data on births and deaths. Other sources of RWD include patient reported outcomes (PRO) data and data which are collected outside of a clinical trial setting, such as through wearable devices, specialised/secure websites, or tablets||When such data are analysed, the information produced may be referred to as RWE.||
||The simplest endpoint to consider is all-cause mortality. Mortality is particularly suitable as an outcome for a RWD based trial||As in traditional RCTs, Patient consent is required before enrolment in RWD trials as well.||WIP|
|NICE||Evidence generated from the analysis of real-world data. It can cover a large array of evidence types including disease epidemiology, health service research or causal estimation (see use cases for real-world data in NICE guidance). It can be generated from a large range of study designs and analytical methods (including quantitative and qualitative methods) depending on the research question or use case||
||Not explicitly stated but NICE considers that collection of valid RWD must reflect a procedure that essentially overcomes limitations emerging from routine practice, e.g. lack of strict protocols on recording outcomes on certain time-points||NICE solely specifies that “data should be used in accordance with national laws and regulations on data protection and information governance”. In the UK, the Health Research Authority (HRA) provides guidance around research and use of data in accordance with the UK Policy Framework for Health and Social Care Research||WIP|
|TGA||RWD may come from the following sources:
||RWE: “clinical evidence regarding the use and potential benefits or risks of a medical product derived from analysis of Real World Data, usually collected outside of the clinical trial (for therapeutics) or investigational testing (for medical devices) setting”.
RWE is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD .
|Health Canada||RWD are data relating to patient status and/or the delivery of health care routinely collected from a variety of source.
health care decisions.
|RWE is the evidence regarding the usage, and potential benefits or risks, of a medical product derived from analysis of RWD.
RWE is evidence about the use, safety, and effectiveness of a medical product, technology, or drug that is based on data from the real-world health care setting. It is playing an increasing role in health care decisions.
|RWE leverages data collected in the routine care of patients through sources such as electronic medical records, healthcare claims data or disease registries. The data sources should be clearly described in the study design, which may include observational studies and pragmatic clinical trials.||RWD/RWE outcomes may compensate for the limited data on rare diseases or more vulnerable subpopulations, due to constraints in the conductance of RCTs. Moreover, a more coordinated and systematic approach to the generation and integration of RWE has the potential to reshape the drug regulatory approval and the reimbursement process. Full integration of RWE from pre-market to post-market, may lead to earlier approval of drugs for rare diseases, favour adequate monitoring of safety and better assessment of the economic value of therapies. Furthermore, the inclusion of RWE may allow for more robust assessments and reassessments of the effectiveness and impact of drugs.||Efforts to protect study participants should be included in this section, including confidentiality measures, safeguards of personal information, involvement and outcome of Institution Research Ethics Boards including a Data Safety Monitoring Board, as well as exemption status and other elements of data protection.||The consideration of incrementally accrued RWE into negotiated flexible pricing arrangements has the potential to reshape the final cost (of drugs). A broader strategy incorporating RWE would support the need for more progressive listing agreements, such as pay-for-performance and outcome-based reimbursement models, that have the potential to reduce drug prices.|
The aim of this article was not to conduct a systematic literature review on the use of RWE, but rather to summarize currently available guidances to facilitate understanding, especially with respect to MDs. The information discussed herein is intended to be useful as an introductory resource for readers interested or having a stake in using RWD/RWE in the regulation of medical technology or pharmaceutical interventions. For this reason, websites of National Competent Authorities were searched for available guidances in July and September 2022, with the last search conducted on September 15, 2022. To identify relevant, recent literature focusing on the use of RWD/RWE for the regulation of MDs, articles were identified using Embase and Medline electronic databases. Only published articles were included. Search terms were developed across concepts of “RWD/RWE”, “medical devices”, ‘’regulatory affairs’’, ‘’MDR – Medical Device Regulation’’ and ‘’Health Technology Assessment (HTA)’’. Filters were applied to include only peer-reviewed articles published since 2017 in English. Three (3) independent reviewers screened the title and abstracts for eligibility, including articles that discussed RWD/RWE, regulation of MDs, and information on concrete RWD/RWE examples. Reviewers examined articles identified for full-text review and disagreements concerning inclusion were resolved by joint review. Duplicate articles were excluded. Hand searches were also performed to identify and/or confirm RWD/RWE initiatives from around the world that the authors were aware of due to their professional exposure to Regulatory Affairs.
Currently, the USA is the only country where RWE is explicitly mentioned in its legislation, resulting in the publication of a formal RWE program by the FDA . The American regulatory Agency was also among the first to identify the need to provide concrete guidance for the use of RWD/RWE, not only for medicines, but for MDs as well. In 2017, the Agency released the first of as series of guidances in an attempt to define and form a new path in regulatory decision-making for MDs that would include the use and evaluation of RWD and RWE [18, 19]. The framework is divided into four main parts:
FDA defines both RWD (“data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources”) and RWE (“clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD”) (see Table 3), but does not define the planning, conducting, and reporting processes pertinent to the collection of RWD and RWE.
The submission of study protocols, including RWD on drugs and biological products and their statistical analysis prior to conducting a study, has also been addressed by the FDA. The protocols focus on the early discussion of relevant patient populations, study exposures/outcomes, study duration, the continuity of data coverage, study plan amendments, and potential confounds related to the study design and data collection processes . RWD/RWE quality is affected by the data life-cycle (e.g., data source-curated, data-transformed, data-analytic dataset) and the FDA stresses that all actions and mechanisms should be documented and evaluated accordingly throughout the study period. The importance of taking into account the “well-captured” (i.e., well-measured data) and “not well-captured” (e.g., smoking, lifestyle, disease history, nutrition, etc.) confounders that may compromise data quality remains one of the major goals for the FDA.
The FDA acknowledges various sources of RWD (see Table 4) in the MD context, including electronic health records (EHRs), claims/billing activities, product/disease registries, patient-generated data (including in home-use settings), and data gathered from other sources that can inform health status (i.e., mobile devices). The main aspects to consider when assessing a RWD source’s eligibility should be accuracy, completeness, integrity, adequacy, data consistency, and a well-tested hypothesis. Overall, data accrual and data assurance/quality control are essential to provide a stable ground for the creation of good quality RWD data. Nevertheless, according to the FDA, the quality of the supported data must be critically assessed to determine whether the RWD sources are reliable, clinically relevant, and sufficient, which also depends on data quality levels required for a given regulatory decision.
|Electronic medical records (EMRs)||Data documented and derived from digital versions of the paper charts in clinician offices, clinics, and hospitals|
|Electronic health records (EHRs)||Electronic data documented and derived from healthcare providers during real-world treatment|
|Claims and billing data||Administrative claims databases comprised of billing data collected by practices, pharmacies, or hospital systems for payor insurance adjudication.|
|Product and disease registries||An organized system that collects clinical and other data in a standardized format for a population defined by a particular disease, condition or exposure|
|Patient Reported data||Patient reported outcomes (PROs) and patient-generated health data (PGHD) deriving directly from patients and reflecting the patient experience, which are gathered by personal devices & health applications|
|Genomic databases (for medicines only)||Real-world genomic databases contain comprehensive genomic profiling from patients treated in the real-world practice setting|
The FDA recognizes two major challenges of RWD collection that may compromise data quality; one is the inherent bias included in these data sources (e.g., bias that compromises data quality and creates issues in drawing conclusions on MD exposures and outcomes), and the second is the actual data collection process. However, patient preference information, unique device identifiers (UDI), paper/electronic patient records, etc., can act as verifiable sources of documentation. With respect to registries, there is an ongoing discussion on how they can be used and the level of relevance and reliability they introduce . The relevant draft guidance from the FDA invites stakeholders to discuss the following:
On the other hand, RWD sources may include, but are not limited to, randomized trials, large simple trials, hybrid and/or pragmatic trials, and observational studies (prospective/retrospective). The FDA will consider RWE as sufficient to support a regulatory decision for an MD when, a) the methodologies used to generate RWE are scientifically sound (i.e., methodology/analysis of RWD, clinical relevance, statistical significance, etc.), and b) the RWD data used to generate the RWE data are also evaluated as sufficient, relevant, and adequate (e.g., with regards to population, source verification procedures, timelines, sources/technical methods, etc.). However, the threshold for data sufficiency is dependent upon the regulatory decision that needs to be made.
Along with the RWD/RWE, the FDA also requires the evaluation of the overall device safety and effectiveness endpoints via the Objective Performance Criterion (OPC) and the Performance Goal (PG). Both OPC and PG act as numerical target values, providing information on the technology/performance and safety of the MDs, with OPC being more robust and trustworthy. The FDA guidance provides only limited insight on data/patient privacy/confidentiality concerns. Nevertheless, it has been among the pioneers with respect to establishing patient privacy as an important factor when working with RWD/RWE.
The latest document issued by FDA on September 2022 adds a practical perspective to the use of RWD/RWE by outlining the need to include cover letters with their submission, to explicitly identify the use of RWD/RWE in support of product labelling .
The European Medicines Agency (EMA) and the European Commission have been discussing the use of RWD and RWE in relation to medicinal products for the last ten years. As early as 2015, the use of RWE for the regulation of medicines was explored in the context of the EMA Adaptive Pathways Pilot . A retrospective analysis of clinical trials submissions to the EMA during 2018 and 2019 revealed that RWE was included in two-fifths of initial marketing authorization applications, and in one-fifth of indication extension applications . A more recent analysis by Eskola et al. revealed that nearly all European Public Assessment Reports (EPARs) included RWE in drug discovery (98%) and life-cycle management (100%), and nearly half included it in the development phase (49%) or to support regulatory decisions at registration (47%) .
In 2019, the EMA published the Operational, Technical, and Methodological (OPTIMAL) framework for regulatory use of RWE in regulatory decision-making, which aimed to set out the appropriate use of valid RWE for regulatory purposes (e.g, safety, efficacy, benefit–risk monitoring), to highlight the operational, technical, and methodological challenges for use of RWD to generate acceptable RWE, and to propose potential solutions .
In 2020, the EMA published the Regulatory Science to 2025 strategic document, which vividly promotes the use of high-quality RWD in regulatory decision-making to generate complementary evidence across a medicinal product’s life-cycle. This includes the development of a framework for rapid and safe access to the latest RWD, and the incorporation of training into the regulatory processes based on EHRs and other routinely collected health data . EMA’s Regulatory Science to 2025 also emphasizes the key role of harmonization across international stakeholders to enhance data sharing and sets the stage for the embedment of data analytics and digital tools for everyday analysis of healthcare data, aiming to promote standardization of RWD. Furthermore, the EMA and FDA reinforced their collaboration on the regulation of medicines with the use of RWE in 2018 .
More recently, in August 2021, the EU put RWE in the wider context of big data and revisited their approach, guided by the priority recommendations of the Big Data Task Force, implemented through the Big Data Steering Group . In November 2021, EMA published a document describing their vision for the use of RWE in the European Union, aiming to establish the use and value of RWE across a spectrum of regulatory use cases .
As part of the EMA-HMA Big Data Steering Group workplan, EMA has recently launched the Data Analytics and Real World Interrogation Network (DARWIN EU), which is part of a wider 11-workstreams EU policy context, most notably the European Commission’s plans for a European Health Data Space [29, 30]. Darwin EU will operate through the co-establishment of a coordination centre, currently under supervision by the Erasmus University Medical Centre. It aims to support the roles of EMA and the European medicines regulatory network in their regulatory decision-making by conducting studies to access and analyze healthcare data from across the EU, to increase the use and receptiveness of RWE in regulation of medicines, and to reduce the time and cost of drug development. At the same time, Darwin EU will become the managing depository of RWD by maintaining a catalogue of data sources along with their metadata for their use in regulatory processes.
DARWIN will deliver 4 types of observational analyses and studies:
During the project’s last meeting in July 2022, the DARWIN EU advisory board aimed to increase the number of acceptable RWD sources and to propose a set of guidelines to improve the efficiency of RWE studies.  A technical workshop has been planned to identify common “use-cases” that could provide regulatory insight for all involved stakeholders, including EMA, HTA bodies, and payers .
The RWD sources for DARWIN EU will include data on primary and specialist care, hospital care electronic health records, claims databases, disease registries, patient-reported outcomes, and drug prescription and dispensing data. The European stakeholders acknowledge that RWD may arise from registries, e-wearables, and EHRs, all of which are susceptible to challenges related to quality, introduction of bias, and heterogeneity, due to non-standardized collection practices. Up until the time this article was written, EMA had not published a comprehensive clear guidance with best practices on RWE collection. However, there are 6 ongoing pilot studies under Darwin EU and their results will be used to inform the forthcoming decisions of data harmonization and metadata collection and subsequent reporting, as well as regulatory decision-making .
On November 2022, EMA announced the first data partners to collaborate with DARWIN EU. The Data Analysis and Real-World Interrogation Network aims to provide timely and reliable evidence on the use, safety, and effectiveness of medicines for human (including vaccines) from real-world healthcare databases across the European Union . In 2024, the first year of operation, DARWIN EU will increase its capacity to routinely support EMA and NCA National Competent Authorities (NCAs) by delivering studies and maintaining data sources. The network will be fully operational by end of 2025- beginning of 2026, when it will be integrated with the European Health Data Space.
As part of its efforts to support the use of RWE, EMA also established the Patient Registry Initiative in 2015, which aids data harmonization within different disease areas and across different national registries [1, 34]. Europe is privileged with an affluence of healthcare data, and for this reason patient registries are often used (see Table 2). However, heterogeneity, restriction in access due to national laws, as well as data protection laws, such as the General Data Protection Regulation (GDRP), challenge the potential for data integration and harmonization. EMA has repeatedly underlined the need for data standardisation and expansion of RWD data sources, proposing the use of legally acceptable consent and data anonymization techniques to conform with data privacy requirements. However, so far, the impact of GDPR has hampered the expansion of the use of RWD/RWE for regulatory purposes.
While EMA has stretched its intention to ‘promote use of high-quality RWD in decision-making’ as published in “EMA regulatory science to 2025”, it has yet to develop a framework that includes concrete guidelines regarding the generation, reporting, and handling of RWE big data, including statistical considerations. It is important to note that EMA has been working with RWE only for medicines and pharmaceuticals and, to the best of our knowledge, there is no mention of RWD/RWE use for MDs.
In December 2021, the United Kingdom’s Medicine and Healthcare products Regulatory Agency (MHRA) published a consultation document on RCTs generating RWE to support regulatory decisions . The guidance covers simple and hybrid trials and sets out the factors that need to be considered when collecting RWD as part of a clinical trial. More specifically, it highlights the importance of accuracy, validity, variability, reliability, and provenance of the data source. Confirmation that the data quality is considered sufficient for the intended use should be included in the protocol. According to the British Agency, there is no barrier to using RWE to gain an initial approval or approval of a new indication, providing the data quality is “robust” and the trial is “designed in a way which allows it to provide the evidence required to answer the regulatory question”.
A second guidance specific to clinical trials is currently under development and will be incorporated in the future . Although the principles are applicable to clinical trials in any area, this guideline will refer only to pharmaceuticals. The guideline does not cover the clinical trials using RWD as a control arm. The RCTs using RWD should be of the same standard that would be expected for a traditional RCT, including pre-specification of the objectives, data to be collected, primary and secondary endpoints, and analysis methods. Evidence generated from RCTs using an RWD source is not generally considered of more or less value for regulatory decision-making than evidence from traditional RCTs, provided the data quality is robust and the trial is well-designed. Data from randomized trials using an RWD source are currently most used for:
RCTs using RWD should be sensitive and structured in a way that encounters the challenges arising from collecting data outside the controlled environment of a traditional clinical trial. In a real-world setting, differences in how the study is conducted could introduce noise or extra variability, which could be a result of clinicians not following the same protocol, differences in background care, or other factors.
Regarding the general framework for the use of RWD in clinical studies, MHRA considers it important to demonstrate that the data source is of sufficient quality for the intended use. Care should be taken to understand the origin of the source database, along with any transformation or manipulation that may have occurred during its processing. Users must be able to define the provenance of the source data, explain the mechanisms used to link data points, manage discrepancies, and describe any limitations or considerations associated with the data. With respect to endpoints to be tested, MHRA prefers the ones that are well recorded in the UK (overall mortality is the most representative example, but other examples are available). There is no difference in reporting between the traditional clinical trials with those using RWD. However, some flexibility is possible around the requirements for safety reporting, with flexibility depending on the proposed population and expected AE profile compared to what is already known about the safety profile and this would be considered on a case-by-case basis. MHRA adopts the same data quality concerns that most authorities express with respect to credibility, assurance, and replicability, and requires that data quality processes and checks are in place to ensure accurate reporting, interpretation, and verification. According to the Agency, RCTs using RWD should follow the same rules for privacy/confidentiality as the traditional RCTs; patient consent is required before enrolment and it should be clearly stated that the privacy and security policies apply to the use of all databases as well as the restrictions related to the transfer, storage, use, publication, and retention of data.
In June 2022, the UK’s HTA appraisal body, the National Institute for Health and Care Excellence (NICE), published its own framework on RWE, reviewing the methods and processes applicable for HTA programs . The framework is one of the most comprehensive documents available so far and provides in-depth guidance and tools to support the use of RWE for HTA-related processes and decision-making. While describing NICE’s expectations for planning, conducting, and reporting RWE studies, it assesses data suitability, describing the information needed to assess data provenance, quality, and relevance for addressing specific research questions. It also provides more specific recommendations for conducting non-randomized studies. An overview of the NICE framework is tabulated in the Supplementary material.
In line with the increasing realization of the importance of RWD and RWE use in regulatory decision-making, the Therapeutics Goods Administration (TGA) has recently requested a review from selected stakeholders to identify how RWE is perceived and utilised by both manufacturers (pharmaceuticals/biologicals/MDs) and TGA assessors . The review focuses on the use of RWE/RWD and patient-reported outcomes (PROs), thereby separating these sources of evidence. However, in a guidance document by TGA on the collection of clinical evidence for MDs, RWD is considered to cover all types of clinical experience data “generated through any clinical use of the device that is not related to clinical investigation”, including EHRs, registries and PROs (“patient-generated data including in home-use settings”). Other sources of RWD identified in the guidance are claims and billing activities, as well as data gathered by Digital Health Technologies (DHTs) (e.g., mobile devices and applications collecting data on health status). Nonetheless, the review established that RWE and PROs are currently not commonly utilised in pre-market submissions and approvals, certainly in comparison to post-market submissions. Additionally, TGA identified that, whereas “critical use of RWE for emerging technologies, such as gene, cell and tissue therapies, and software-based medical devices is a critical and necessary component to understanding and enhancing the performance of such products”, there was a gap in the communication to manufacturers on the acceptance of RWE and PROs in regulatory submissions .
To overturn this, TGA initiated several activities, including promotion of the use of RWE for regulatory pre-market submissions, and the introduction of a requirement for clear communication on the contribution of RWE in the application or decision-making. Pre-submission meetings are also encouraged to discuss the types of RWE included, as well as their quality and acceptability for regulatory purposes. Additionally, for MDs (particularly for software-based devices), in accordance with the 2019 Action Plan for Medical Devices, TGA is committed to publishing more information on how RWE and PROs are used in the regulatory submission and how they contribute to decision-making, at least for higher risk devices .
Implementation of UDIs for MDs is anticipated to enable better traceability and, therefore, support the collection of higher quality RWD that could push forward the use of RWE for MD regulatory submissions. In support of higher data quality, TGA advises the establishment of registries with a high number of included patients and device types. Finally, the collection of RWD is expected to follow protocols to ensure elimination of bias and Good Clinical Practice (GCP) principles.
TGA acknowledges that pharmaceuticals and MDs have different development pathways; formal clinical studies (clinical investigations) are mainly conducted for higher risk MDs. Additionally, clinical investigations are usually smaller and often “simpler” compared to RCTs for pharmaceuticals, as in some cases it is not feasible to use randomisation and/or blinding in the study design. Within this context, the Australian Agency deems necessary to develop different guidelines for the generation and use of RWE in regulatory submissions for medicine and MDs. Constraints in resources, however, appear to preclude the generation of guidance documents by the TGA. Therefore, TGA tends to follow other regulatory authorities (such as FDA, EU, Health Canada) or the International Medical Device Regulators Forum (IMDRF) for MDs and to adopt already developed guidelines.
Health Canada has recently expressed their intention to optimize the use of RWD/RWE in the regulatory decision-making process as part of the R2D2 (Regulatory Review of Drugs and Devices) project and respective follow-on initiatives. Health Canada and the Canadian Agency for Drugs and Technologies in Health (CADTH) held a joint workshop in 2018, launching an initiative to integrate RWE across the life cycle of drugs . The 2 stakeholders co-decided that the full integration of RWE was necessary to help overcome marketing authorization and reimbursement challenges in the country [42, 43, 44].
Following this preliminary framework, Health Canada published a guidance document on April 16, 2019, acknowledging that the use of RWE in regulatory decision is increasing globally for the assessment of drug safety, efficacy, and effectiveness . An enclosed document identified that certain diseases/disorders (such as rare diseases) could not be sufficiently studied with the conduct of RCT and, as such, RWE studies could be the answer to the unmet medical/regulatory need .
Health Canada currently limits discussion of RWE primarily to medicines. However, CADTH participates in several international initiatives such as CIOMS, INAHTA, and ISPE/RWE that consider MDs as well (see Table 3). The proposed framework requires that a protocol should clearly describe the data sources utilized and the appropriateness of these data to capture all relevant exposures, outcomes, and covariates of interest. RWE leverages data collected in the routine care of patients through sources such as electronic medical records, healthcare claims data, or disease registries. The study population should be well-defined, and the protocol should provide information on the sampling framework, demographic characteristics, clinical exposures, and duration of follow-up. Inclusion and exclusion criteria and a description of the generalizability of the study may also be included. For both prospective and retrospective collection of RWE, the protocol should describe how the outcomes are relevant to the claim and the approaches taken to demonstrate the robustness of the findings to any sources of bias. Outcomes should be clearly defined and measured. The protocol should not only address the validity of any measures but predict the collection of items that could modify the effect and how they will be included in the analysis.
Expanding data and evidence sources to include RWD/RWE may facilitate generalizability by incorporating data from rare diseases and/or sensitive patient populations. The Canadian authorities point out the need for clear data collection methods that include pretesting and identify the need to ensure data quality and consistent data management as major challenges.
Despite the growing interest in RWD/RWE in Latin America, significant gaps exist compared to other regions in the world. To the best of the authors’ knowledge, the regional local authorities have not issued formal guidances, therefore there are no official definitions for RWD and RWE, and the terms are used based, mostly, on FDA’s definitions. Moreover, the collection and use of RWE for regulatory and healthcare decision-making is inconsistent across countries, providers, and insurance stakeholders, while RWE initiatives are unfavoured due to lack of resources and concrete guidance on how to collect and manage RWD. Cultural and socioeconomic diversity, and the uneven access to technology and health resources across the continent, also hamper data quality and the ability to ensure representativeness of data.
The ongoing discussion about the impact of digitalization aims to generate a framework for the facilitation of access to RWE. Yet, the emergence of various HTA agencies across the continent has not been able to promote incorporation of RWE in regulatory processes until now, mainly because the available sources of RWD are limited and lack reliability, thus rendering the use of data from RCTs more attractive and realistic .
A recent study by Justo et al. identified that the commonest RWD sources in Argentina, Brazil, Chile, and Colombia are clinical databases, including EMRs patient registry databases of observational cohort studies, and health information systems, including surveillance systems and administrative databases . The main uses of RWE in Argentina, Brazil, Chile, and Colombia align with those of most countries around the world (i.e., for pharmacovigilance purposes, post-marketing surveillance of pharmaceuticals, HTA decisions related to reimbursement and economical evaluations, and academic research). Interestingly, there seems to be limited use of RWD/RWE to support access of orphan medicines into the market and RCTs are still considered the main source of data for marketing authorization.
Various initiatives are currently ongoing aiming to increase awareness and understanding of how RWE can be used in Latin American countries to boost healthcare decision-making (see Table 2). ANVISA, Brazil’s Regulatory Agency, is also working to revise Resolução da Diretoria Colegiada 200/2017, which sets outs the requirements for marketing authorization of medicines containing synthetic and semisynthetic active ingredients. The goal is to enable access to effective and tolerable treatments in the absence of traditional RCTs and it is expected that incorporation of RWE in this approach will play an important role .
The development of RWE in China has been promoted significantly in the past few years and has taken a distance from the evaluation of outcomes and comparative effectiveness of the traditional Chinese medicine interventions. During 2020, the Chinese National Medical Product Administration (NMPA) published a series of preliminary guidance documents including a technical guideline for RWE supporting drug development, and an interim technical guidance for using RWE to support research, development, and regulatory review of pediatric drugs . Chinese Authorities were among the first to issue a guidance for the use of RWE to support the evaluation of MDs, which was later supported by a more generic guidance on how to use RWD to generate RWE . Overall, China scopes the use of RWE around drug development and support of post-marketing studies. Different case studies are discussed in which RWE may support drug development and regulatory decisions, including the use of external controls for approval of rare disease treatments and orphan drug assessments where the conduct of an RCT is neither feasible nor ethical, as well as the use of RWE for expansion of indications of currently marketed medicinal products. The Chinese approach focuses on medicines for pediatric use and how RWE can support pediatric drug development. In particular, the NMPA specifies circumstances for which RWE could introduce unique benefits both for product development and approval processes, specifically:
Given the size and complex infrastructure of China, one of the major challenges for Chinese Authorities is to determine reliable, reproducible data sources that can be used for epidemiology and health economics purposes, as well as research outcomes that are comparable and acceptable by the international community. Within this context, Xie et al.  grouped the majority of RWD used in China into 6 types: a) administrative claims databases (including national and regional levels), b) EHR (including regional EHR databases, multicenter, and single-center EHR databases), c) databases linking EHR with claims, d) cohort or registry data, e) medical chart review studies, and f) surveys of the general population and patients. The first 2 types are relatively new in China and their use is subject to limitations, such as limited data access/sharing potentials, lack of longitudinal follow-up data, and sub-optimal data quality. In addition, lack of regulations related to patient privacy protection is also hindering the use of RWD/RWE, as bureaucratic requirements, and lack of a central ethics review board in China complicate the use of the above sources. On the other hand, the potential to collect data from large-size cohorts creates research opportunities for general and rare diseases.
In July 2020, the Taiwan Food and Drug Administration (TFDA) finalized a guidance document on the points to consider for using RWE to support the research and development of drugs. This was followed by the November 2020 guidance on the use of EMR data in clinical investigations to improve data accuracy, and to promote clinical trial efficiency and increased interoperability . TFDA generally tends to align with FDA’s rationale and promotes early engagement and regular communication with regulatory authorities to establish transparency in real-world study design. The agency is investigating acceptable uses of RWE to support a wide range of regulatory decisions, including label changes and premarketing safety profile evaluations, the use of periodic safety update reports and periodic benefit-risk evaluation reports of pharmaceuticals from other countries, and premarketing effectiveness assessments (e.g., clinical evidence of orphan medicines) as RWD sources of safety data.
Japan has a long history of initiatives related to the use of RWD/RWE and the RWD Working Group of the Pharmaceuticals and Medical Devices Agency (PMDA) has issued a series of relevant documents (see Table 1) . In previous years, focus was mainly on drug safety assessment at the post-marketing stage and the efficacy evaluation of orphan drugs, but recently there is ongoing remediation regarding the acceptability of RWD and RWE for regulatory submission in earlier stages in the lifetime of a medical product. The goal of the Agency’s Clinical Innovation Network Working Group is to develop a framework that will incorporate appropriate registry data and use of EMRs to ensure data reliability for new and supplemental drug applications . The 2 most recent guidelines address the basic principles for the use of registries for approval purposes and the points to consider for ensuring reliability during their use [55, 56]. PMDA emphasizes the importance of reliability and quality of the RWD in terms of accuracy, consistency and comprehensiveness, and argues that inclusion of RWD/RWE in regulatory decision-making should be determined by the appropriateness of the analytical methods used in a study. This is because unless both reliability and appropriateness are present, the interpretability of results cannot be guaranteed. The Japanese authorities point toward sufficient framing and specification of the regulatory purpose before deployment of any activity involving RWD/RWE, to ensure that the collected data are sufficient for the regulatory setting they are being used in. Currently, PMDA receives feedback from multidisciplinary working groups on all issues relating to RWD/RWE, including, but not limited to, data reliability standards, methodological approaches, and the potential to accept new technologies, such as data collection by wearable devices and data handling and screening by artificial intelligence.
Collection of local RWD is of high interest to South-East Asian countries as it can be used to close the gap in data collected in RCTs or observational studies conducted in a foreign-country context without being able to inform policy-making in the local Asian context, and often, without considering differences in regional clinical practices, reimbursement practices, ethics, and judicial systems. Indonesia, Malaysia, Philippines, Singapore, South Korea and Thailand all accept RWD/RWE as part of regulatory submissions, while Malaysia and Philippines do not even require justification for their use [57, 58]. The REALISE guidance , a collaboration between global experts and leaders from HTA agencies in Asia, indicates that primary RWD sources in South-East Asia include registries, EMRs, claims databases, and hospital administration documents (e.g., discharge records, prescirption archives and health surveys). Within this context, reimbursement decisions in many South-East Asian health systems (e.g., Indonesia, Malaysia, Philippines, Singapore, Thailand), may take years to issue, which renders the need for ‘’imported’’ clinical feedback even more critical.
However, despite the vivid interest by various stakeholders in the use of RWD/RWE, there is still no clear consensus, neither on national, nor regional level, about how to incorporate them into existing HTA processes . Although there are a few available frameworks , HTA systems in South East Asia remain extremely variable. Alignment of practices within a region on how to generate and use RWD/RWE is hampering effective implementation. HTA institutionalization has progressed well in Thailand, where it has been formally integrated into reimbursement decisions, including the development of the National List of Essential Medicines and the Universal Health Coverage Scheme benefits package. [60, 61] Similarly, Philippines have moved to a major reform in 2019 by signing the Universal Health Care Act, which also included the establishment of a HTA unit, which is supported by the International Decision Support Initiative (iDSI) to develop capacities to generate and use relevant HTA evidence .
Indian Authorities are currently performing an extensive reform of healthcare services and regulation of medical devices/medicines. Although there is not yet any concrete regulation or guidance of the field, progress is made in digitalization of the healthcare system and regulation of patient data with the Digital Information Security in Healthcare Act (DISHA) .
DISHA defines Real-World Data as […] information concerning the physical or mental health of individuals; information around health service provision; information concerning organ or bodily substance donations; information on testing or examination of organs and bodily substances; information collected during the course of health service delivery; and information on the clinical establishment accessed by the individual […]. The Indian Chapter of ISOPR has already proposed a set of guidelines for the country, while various HTA initiatives are promoted. [57, 59, 64, 65, 66] According to a recent survey , the main actors/user of HTA in India is the government itself, while public organisations including autonomous, research institutions, are considered as the main generator. The same survey revealed that there is limited or no access to data, such as input from pharmaceutical use, cost of service delivery, hospital-level data and health outcomes, which is consistent with missing infrasturctures, the diverging profiling of the country and the nascent leveraging of digital healthcare.
Currently, Middle East and Gulf countries are falling behind in generating and using RWD/RWE for regulatory purposes, while grappling to transform local healthcare systems. However, a limited number of case studies showcasing the use of RWE is indeed available . To address the challenges hindering effective use of RWE in the region, the Gulf Corporation Council (GCC) came together in December 2020 to identify the inherent regional restrictions and propose actionable solutions. Based on the working assumption that major challenges include the public’s fear and mistrust to share personal health data, the lack of collaboration and partnership between institutions, variability in data sources also due to ethnic differences, and suboptimal technological expertise , GCC stakeholders identified the need to develop a consistent governance framework on RWD/RWE taking into account ISPOR-ISPE’s work in progress and proposed a set of 7 actionable points for leveraging RWE, namely:
Currently, the focus lies heavily on digitalization of healthcare systems via the incorporation of HTA applications but since requirements for registration, pricing & procurement stakeholders are highly variable across countries of the region, infrastructural changes in delivery of healthcare services must precede local RWD generation.
To the best of the authors’ knowledge, the major HTA actions on the region include
Similar to the Middle-East/Gulf regions, low-and middle-income (LMICs) African countries are significantly lagging in the generation and use of RWE in regulatory and reimbursement decision-making for practically the same reasons. Due to the particularities of the regional healthcare systems related to limited budgets and prioritization of healthcare needs based on prevalence of medical diseases, focus has been on the establishment of pharmacovigilance systems aiming to promote epidemiology-related services. For example, the African Union NEPAD Smart Safety Surveillance (AU-3S programme)  has adapted the principles of Habitat International Coalition (HIC) to fit local needs and capabilities in sub-Saharan African nations, while in 2020 and 2021, Ghana, Nigeria, South Africa, and Ethiopia leveraged pharmacovigilance technology provisioned by MHRA to establish an ICH-compliant RWD data repository in a secure UK cloud environment . Nonetheless, there is a growing interest in the use of HTAs even though there are currently no examples of independent, established HTA  institutions. South Africa and Ghana seem to be pioneers in the field. South Africa is the only African country, which has introduced a legal framework to protect the country’s residents from harm by protecting their personal information (South Africa’s Protection of Personal Information Act (POPIA)) . Ghana has already conducted multi-criteria decision analysis studies to set up the feasibility of various interventions to be used for the country’s healthcare sector reform .
Non-randomized, evidence-related confounding factors and biases are discussed in all available guidances and frameworks since they add challenges to the statistical evaluation, mainly due to uncertainties and limitations for assessment and interpretation of causal inferences [6, 75, 76]. A concrete statistical analysis plan is essential when using RWD to estimate treatment effects and clinically meaningful outcomes. The statistical analysis should be appropriate to anticipate all challenges that may occur. A broad spectrum of statistical tools may be used for the analysis depending on the nature of data and the corresponding type of study (RCT, registry, prospective cohort study, survey, etc.). When conducting an RCT using RWD, the statistical analysis plan and protocol should follow that of a traditional clinical trial. Where available, most guidelines focus on techniques (traditional or more elaborative such as machine-learning) for managing bias and confounding factors that are likely to occur when using RWD, due to discrepancies across the diverse sources of data. Another aspect often mentioned is the use of a power calculation analysis to obtain a representative sample size for the study. Only FDA mentions that RWD can be used in later analyses to conduct an informative prior probability distribution using Bayesian statistics (see Table 5).
|MHRA||It is recommended that statistical power calculations are used to assess whether the potential number of patients would enable a clinically important treatment effect to be detected.|
|Health Canada||A description and justification for the chosen approach for statistical analyses should be well-described, including methods of estimation, a rationale for the study size and/or statistical precision, descriptive analyses, stratified analyses, defined point estimates and confidence intervals, types of comparators, plans to control for confounding, outcome misclassification, sensitivity analyses, type I error control, and plans for handling missing data.|
|NICE||An essential aspect repeating throughout the NICE framework is that applied statistical methods should be tailored to the research question and take into consideration the key risks of bias that emerge from the study’s design (e.g. time varying confounders). Bias can also result from the statistical analysis itself (e.g. model misspecification). The appropriateness of the data analysis model should be scrutinized via diagnostic checks, without the framework elaborating on the type of those verification methods. Within the study report, the statistical methods section should adequately describe the chosen models and substantiate their validity. A statistical analysis plan should be in place prior to performing final analysis and fundamentally integrate within the published study protocol. A presentation of statistical methods for analysing observational data is provided in NICE Decision support unit Technical support document (TSD)17. Finally, during analysis, advanced computational approaches (e.g. machine learning) can be utilized for the identification of the covariates. Selection of covariates based on statistical significance should be avoided according to the NICE framework.|
Statistical analysis techniques may vary according to the corresponding type of study and should be planned on a case-per-case scenario. More detailed guidelines are expected from authorities to provide a more thorough framework around statistical analyses and the use of RWD in different types of studies.
The new Medical Devices Regulation in Europe, and the profound revisions in the regulations of Australia, Canada and elsewhere, have significantly increased the requirements for substantiative clinical evidence throughout the lifecycle of MDs (see Figure 3). [9, 77, 78] In particular, EU Regulation 2017/745 requires that manufacturers who wish to market/continue to market their MDs within Europe must develop and regularly update a process that will enable the monitoring of the real-world performance and safety of their MD. This should not only be via clinical investigations but also through the proactive, dynamic monitoring of feedback collected from patients, end-users, and healthcare professionals. Post-market surveillance requirements under EU Regulation 2017/745 are risk-based and part of a continuous process that updates the substantiative clinical evidence of clinical evaluation reports, aiming to confirm the positive benefit-risk ratio and the clinical benefit of any given MD. Within this context, the Post-Market Clinical Follow-up (PMCF) is defined as the proactive collection and evaluation of clinical data from the use in, or on, humans of a marketed device, with the aim to confirm the device’s clinical safety and performance throughout its expected lifetime, to ensure the continued acceptability of identified risks, and to detect emerging risks on the basis of factual evidence . Without explicitly stating it, EU Regulation 2017/745 necessitates the inclusion of RWD for the regulatory approval of MDs. MDCG 2020-6, a European guidance on the amount and type of clinical data needed for MDs that were marketed before the coming into force of EU Regulation 2017/745 (‘legacy devices’) to demonstrate conformity with the relevant General Safety and Performance Requirements (GSPR), states that RWD, such as registries and information deriving from insurance database records, can be used to demonstrate indirect clinical benefits .
In contrast to pharmaceuticals, where RCTs seem to remain the standard option for substantiation of claims, ‘’traditional’’ clinical data may not always be able to serve the requirements of PMCF under EU Regulation 2017/745. This is primarily due to the extended scope of PMCF, which aims not only to confirm the approved indications, targeted population, and potential restrictions of use (contraindications, warnings, precautions), but also to identify the real-world use of a device and therefore to highlight potential off-label uses, either in terms of clinical applications or the targeted population. RWE that feeds into the clinical evaluation of a MD is drawn from the entire population exposed to the device. For this reason, RWE, by definition, is non-comparative and focuses on the performance of the device in scope, rather than making comparisons to other devices or treatment options. Since PMCF is intended for devices already on the market, there is no experimental exposure involved, no upper limit on included subjects, and no eligibility criteria. Furthermore, PMCF is not necessarily conducted in clinical settings, as data can be collected directly from patients and/or end-users. Vigilance data i.e., data collected from databases managed by Competent Authorities for the reporting of (serious) incidents, field safety corrective actions (FSCAs) and recalls, are among the sources of data used in PMCF, underlying the growing role of RWD in the regulation of MDs around the globe.
In this context, development of medical device registries is significantly favoured. The IMDRF defines a medical device registry as an […] an organized system with a primary aim to increase the knowledge on medical devices contributing to improve the quality of patient care that continuously collects relevant data, evaluates meaningful outcomes and comprehensively covers the population defined by exposure to particular device(s) at a reasonably generalizable scale (e.g., international, national, regional, and health system) [80, 81]. […] In the previous years, patient registries have been restrictively used for the marketing authorization of medical products with low-disease prevalence or high-disease severity intended for small and heterogeneous patient populations, especially when the level of clinical evidence submitted were not deemed sufficient. Lately, under the generalized reform of healthcare by the introduction of innovative new technologies, registries are considered a major source of RWD/RWE, receiving global attention. The International Medical Device Regulators Forum (IMDRF) is exploring the possibilities of prospective national data collection in multiple countries, with the potential for subsequent data linkage to provide information from very large numbers of patients longitudinally. [80, 81]
Typically, registries  become an option for MDs on a post-market surveillance level for the collection of long-term safety data. In case of implantable MDs, the flexibility of registries to accommodate RWD in a dynamic manner is gradually establishing their role as a key supplementary method for the evaluation of long-term safety. Indeed, registration of all implantable devices under the EU Regulation 2017/745 is compulsory and both notified bodies and manufacturers are required to consider registry data, not only to ensure traceability but also as part of their strategy for continuous collection of data about patient characteristics and clinical outcomes. Registries that collect a predefined, limited set of variables with an easy and quick procedure may be more effective in motivating data providers and, therefore may be triggering a higher level of data quality. However, when assessing the level of evidence collected through a medical device registry, the overall design and rationale of device categorisation, the type of selected patient-reported outcomes, and the ability to link and/or verify the registry’s data with other data sources play a decisive role. The International Medical Device Regulator Forum Registry Working Group has defined 15 registry requirements , grouped into six elements, to assess the suitability of registry data for regulatory submissions, namely:
Therefore, when considering the use of registry data for regulatory submissions, it is necessary to provide evidence of systematic and complete coverage transparency of quality assurance, clear policies for data access and sharing, registry sustainability and conformity with ISO 14155:2020 requirements.
From a PMCF perspective, a concrete sustainability plan laying down short and long-term strategies for the collection of RWD, along with monitoring and interpretation is critical to establish the continued acceptability of the benefit-risk ratio as well as to monitor the emergence of previously unidentified risks for MDs through a MD registry.
International efforts to establish frameworks for the use of RWD/RWE for regulatory purposes are emerging around the world, underlining a universal interest to use RWE throughout the lifecycle of both pharmaceuticals and MDs. As expected, the USA and European Stakeholders have been more heavily engaged, partially due to easier access to funding resources but also due to the need to compile bodies of evidence to justify reimbursement decision-making. Meanwhile, efforts in countries such as China have been boosted by big data initiatives as well as by their goal to demonstrate the efficacy and safety of new products in the Chinese population, which is a challenging pool of subjects for RCTs.
Common limitations to the use of RWE are discussed within the available guidances and include issues of credibility, quality, validity, data collection, statistical analysis planning and interpretation, and bias due to unblinded, uncontrolled, or non-randomized treatment allocation (see Table 6).
Leading countries of the RWE ecosystem have already identified the damaging role of ambiguity within the design of RWD studies and are now shifting their focus to the generation of transparent, high-quality, interpretable RWD studies, and the identification of best practices to ensure consistency and bias minimization (see Table 6). A growing number of advocates for the harmonization and standardization of RWD/RWE practices recognize the importance of collecting, managing and analysing RWD in a reproducible manner, emulating the controlled micro-environment of a RCT and, as such, are collaborating to develop templates to reduce inconsistency when planning and reporting RWE studies .
Interestingly, available guidances point to a shift from the traditional RWE uses (i.e., the post-marketing effectiveness assessments of pharmaceuticals) towards the pre-marketing stages of regulatory approvals and the support of preclinical and clinical evaluations of MDs. [1, 3, 4, 58, 85, 86] In fact, the directionality of clinical research used to support regulatory affairs seems to be shifting away from tightly controlled clinical settings, which is the case for RCTs, heading towards a more inclusive model which is the case for RWD. Although, the secondary synthesis of RWD-based information introduces non-negligible bias and has limitations to the extractable outcomes, which must be addressed in a continuous and structural manner, clinical experience-based feedback has the potential to counterbalance the longer time needed to go from RCT conceptualization to materialization of its results and turning them into substantiative evidence of a regulatory submission. For this reason, all currently available guidances point to the need to centralize, keep track of, and ensure public access to the advances of the field. The variety and complexity of RWD also calls for the development of more sophisticated data processing and analysis tools to stand up to the well-established infrastructures and processes of traditional clinical research that do not require justification in front of a board of regulators.
The authors of this article interpret RWD as a cluster of data sources ranging from routine clinical practice to systematic use of medical products by lay persons aiming to inform regulatory and clinical quivers decisions with input that is not available through controlled clinical research activities. In that sense, RWE feeds the healthcare ecosystem with raw data that, once, refined and analysed, can provide insight into the clinical performance and safety of MDs and medicines, therefore, bridging the gap between clinical evaluation, accountability and transparency in both clinical practice and regulatory affairs (see Figure 4).
RWE has the potential to cross the translational gap between closed-type clinical research, which often lacks equity and diversity, and real-world clinical practice. Sustainable access to healthcare is nothing less than a complex ecosystem. Its primary biotic components -patients and healthcare professionals- are in continuous need of abiotic elements such as medical technology innovations, which in turn, rely to a network of access-to-healthcare facilitators. By ensuring the latter, we practically trigger continuity of care by accelerating access to market and reimbursement options. That being the case, capturing accurate, reproducible clinical data becomes a one-way street for established and developing healthcare ecosystems, which must accommodate both ‘’traditional’’ clinical data and RWD and allow them to become interoperable functional pieces of regulatory and clinical decision-making.
The generation of accurate, reliable, and transparent RWE platforms sets the scene for expedited access to care and new treatment options, not only for previously undertreated populations but also for those in need of personalized therapies. However, the operational, technical, and methodological gaps that remain unaddressed by the currently available guidances and frameworks render the harmonization and standardization efforts urgent. More than being a matter of RWD/RWE dominating over traditional clinical research, it is a matter of convergence. RCTs used in parallel with RWD/RWE can result in improvement of data quality, enhancement of data linkage to ensure better data flow and sharing, clinically significant diversity and representativeness, optimized healthcare planning, facilitation of reimbursement schemes, and practical betterment of clinical development for both pharmaceuticals and MDs. The use of RWE must be driven by forward-thinking practices aiming to conform with the new, stringent requirements for dynamic post-market surveillance of MDs, as well as the delivery and evaluation of targeted pharmaceuticals.
The additional file for this article can be found as follows:Supplementary Table
Overview of NICE real-world evidence framework. DOI: https://doi.org/10.29337/ijdh.50.s1
The authors have no competing interests to declare.
Burns L, et al. Real-World Evidence for Regulatory Decision-Making: Guidance From Around the World. Clin Ther, 2022; 44(3): 420–437. DOI: https://doi.org/10.1016/j.clinthera.2022.01.012
Lewis A, et al. Digital Health Technologies for Medical Devices – Real World Evidence Collection – Challenges and Solutions Towards Clinical Evidence. International Journal of Digital Health, 2022; 2(1). DOI: https://doi.org/10.29337/ijdh.49
Maruszczyk K, et al. Systematic review of guidance for the collection and use of patient-reported outcomes in real-world evidence generation to support regulation, reimbursement and health policy. J Patient Rep Outcomes, 2022; 6(1): 57. DOI: https://doi.org/10.1186/s41687-022-00466-7
Kamusheva M, et al. Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers. J Comp Eff Res, 2022; 11(12): 905–913. DOI: https://doi.org/10.2217/cer-2022-0065
Garrison LP, Jr., et al. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health, 2007; 10(5): 326–35. DOI: https://doi.org/10.1111/j.1524-4733.2007.00186.x
Di Maio M, Perrone F, Conte P. Real-World Evidence in Oncology: Opportunities and Limitations. Oncologist, 2020; 25(5): e746–e752. DOI: https://doi.org/10.1634/theoncologist.2019-0647
Nazha B, Yang JC, Owonikoko TK. Benefits and limitations of real-world evidence: lessons from EGFR mutation-positive non-small-cell lung cancer. Future Oncol, 2021; 17(8): 965–977. DOI: https://doi.org/10.2217/fon-2020-0951
European Medicines Agency. ICH guideline E8 (R1) on general considerations for clinical studies; 2021.
European Commission. REGULATION (EU) 2017/745 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC; 2017.
Purpura CA, et al. The Role of Real-World Evidence in FDA-Approved New Drug and Biologics License Applications. Clin Pharmacol Ther, 2022; 111(1): 135–144. DOI: https://doi.org/10.1002/cpt.2474
Flynn R, et al. Marketing Authorization Applications Made to the European Medicines Agency in 2018-2019: What was the Contribution of Real-World Evidence? Clin Pharmacol Ther, 2022; 111(1): 90–97. DOI: https://doi.org/10.1002/cpt.2461
Cave A, Kurz X, Arlett P. Real-World Data for Regulatory Decision Making: Challenges and Possible Solutions for Europe. Clin Pharmacol Ther, 2019; 106(1): 36–39. DOI: https://doi.org/10.1002/cpt.1426
Arondekar B, et al. Real-World Evidence in Support of Oncology Product Registration: A Systematic Review of New Drug Application and Biologics License Application Approvals from 2015–2020. Clin Cancer Res, 2022; 28(1): 27–35. DOI: https://doi.org/10.1158/1078-0432.CCR-21-2639
Derman BA, et al. Reality check: Real-world evidence to support therapeutic development in hematologic malignancies. Blood Rev, 2022; 53: 100913. DOI: https://doi.org/10.1016/j.blre.2021.100913
Batra A, Cheung WY. Role of real-world evidence in informing cancer care: lessons from colorectal cancer. Curr Oncol, 2019; 26(Suppl 1): S53–S56. DOI: https://doi.org/10.3747/co.26.5625
Greshock J, et al. Harnessing Real-World Evidence for the Development of Novel Cancer Therapies. Trends Cancer, 2020; 6(11): 907–909. DOI: https://doi.org/10.1016/j.trecan.2020.08.006
Law UP. SHORT TITLE.—This Act may be cited as the ‘‘21st Century Cures Act’’. 2016; Available from: https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf.
FDA. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices - Guidance for Industry and Food and Drug Administration Staff 2017; Available from: https://www.fda.gov/media/99447/download.
FDA. FRAMEWORK FOR FDA’S REAL-WORLD EVIDENCE PROGRAM. 2018; Available from: https://www.fda.gov/media/120060/download.
FDA. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory DecisionMaking for Drug and Biological Products – Guidance for Industry. 2021; Available from: https://www.fda.gov/media/152503/download.
FDA. Real-World Data: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry. 2021; Available from: https://www.fda.gov/media/154449/download.
FDA. Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drug and Biological Products Guidance for Industry; 2022.
European Medicines Agency. Adaptive pathways. 2016; Available from: https://www.ema.europa.eu/en/human-regulatory/research-development/adaptive-pathways.
Eskola SM, et al. Use of Real-World Data and Evidence in Drug Development of Medicinal Products Centrally Authorized in Europe in 2018–2019. Clin Pharmacol Ther, 2022; 111(1): 310–320. DOI: https://doi.org/10.1002/cpt.2462
European Medicines Agency. Regulatory Science to 2025 strategic document. 2022; Available from: https://www.ema.europa.eu/en/about-us/how-we-work/regulatory-science-strategy#regulatory-science-strategy-to-2025-section.
European Medicines Agency. Reinforced EU/US Collaboration on Medicines; 2018.
HMA. European union medicines agencies network strategy 2025 protecting public health time rapid change; 2020.
European Medicines Agency. A vision for use of real-world evidence in EU medicines regulation. 2021; Available from: https://www.ema.europa.eu/en/news/vision-use-real-world-evidence-eu-medicines-regulation.
Commission E. European Health Data Space. 2022; Available from: https://health.ec.europa.eu/ehealth-digital-health-and-care/european-health-data-space_en.
European Medicines Agency. Data Analysis and Real World Interrogation Network (DARWIN EU). 2022; Available from: https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-eu.
Scholfield I. EU DARWIN Project To Catalog Real-World Data Sources; 2022, Pink Sheet.
Sharma V. EMA, HTA Bodies & Payers Explore Common Ground On Use Of RWE; 2022, Pink Sheet.
European Medicines Agency. DARWIN EU® welcomes first data partners. 2022; Available from: https://www.ema.europa.eu/en/news/darwin-eur-welcomes-first-data-partners.
Varnai P, et al. The Evidence REVEAL Study: Exploring the Use of Real-World Evidence and Complex Clinical Trial Design by the European Pharmaceutical Industry. Clin Pharmacol Ther, 2021; 110(5): 1180–1189. DOI: https://doi.org/10.1002/cpt.2103
MHRA. MHRA guidance on the use of real-world data in clinical studies to support regulatory decisions; 2021.
MHRA. Consultation document: MHRA draft guidance on randomised controlled trials generating real-world evidence to support regulatory decisions. 2021; Available from: https://www.gov.uk/government/consultations/mhra-draft-guidance-on-randomised-controlled-trials-generating-real-world-evidence-to-support-regulatory-decisions/consultation-document-mhra-draft-guidance-on-randomised-controlled-trials-generating-real-world-evidence-to-support-regulatory-decisions.
NICE. NICE real-world evidence framework. 2022; Available from: https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837.
TGA. Clinical evidence guidelines for medical devices. 2022; Available from: https://www.tga.gov.au/sites/default/files/clinical-evidence-guidelines-medical-devices.pdf.
TGA. Review of real world evidence and patient reported outcomes. 2021; Available from: https://www.tga.gov.au/review-real-world-evidence-and-patient-reported-outcomes.
TGA. An Action Plan for Medical Devices. 2019; Available from: https://www.tga.gov.au/resources/publication/publications/action-plan-medical-devices.
Tadrous M, et al. Developing a Canadian Real-World Evidence Action Plan across the Drug Life Cycle. Healthc Policy, 2020; 15(4): 41–47. DOI: https://doi.org/10.12927/hcpol.2020.26225
Foundation S-TSM. Outcomes-based reimbursement of medicines; 2017. Available from: https://www.smf.co.uk/publications/outcomes-based-reimbursement-medicines/.
Vlaanderen FP, et al. Design and effects of outcome-based payment models in healthcare: a systematic review. Eur J Health Econ, 2019; 20(2): 217–232. DOI: https://doi.org/10.1007/s10198-018-0989-8
Lau C, Jamali F, Loebenberg R. Health Canada Usage of Real World Evidence (RWE) in Regulatory Decision Making compared with FDA/EMA usage based on publicly available information. J Pharm Pharm Sci, 2022; 25: 227–236. DOI: https://doi.org/10.18433/jpps32715
Health Canada. Optimizing the Use of Real World Evidence to Inform Regulatory Decision-Making - Health Products and Food Branch Notice. 2019; Available from: https://www.canada.ca/en/health-canada/services/drugs-health-products/drug-products/announcements/optimizing-real-world-evidence-regulatory-decisions.html.
Health Canada. Elements of Real World Data/Evidence Quality throughout the Prescription Drug Product Life Cycle. 2019; Available from: https://www.canada.ca/en/services/health/publications/drugs-health-products/real-world-data-evidence-drug-lifecycle-report.html.
Justo N, et al. Real-World Evidence in Healthcare Decision Making: Global Trends and Case Studies From Latin America. Value Health, 2019; 22(6): 739–749. DOI: https://doi.org/10.1016/j.jval.2019.01.014
(Anvisa) BHRA. Anvisa Working Toward New Regulatory Framework for Innovative Synthetic and Semi-Synthetic Drugs in Brazil; 2019. Available from: https://globalforum.diaglobal.org/issue/december-2019/anvisa-working-toward-new-regulatory-framework-for-innovative-synthetic-and-semi-synthetic-drugs-in-brazil/.
Evaluation NMPACCfD. Guideline on using real-world evidence to support drug research & development and evaluation; 2020. Available from: https://www.nmpa.gov.cn/yaopin/ypggtg/ypqtgg/20200107151901190.html.
Evaluation CfD. Guideline on using real-world data to generate real-world evidence (trial); 2021. Available from: https://redica.com/wp-content/uploads/NMPA_-Attachment_-_Guiding-Principles-of-Real-World-Data-Used-to-Generate-Real-World-Evidence-Trial_.pdf.
Xie J, et al. Real-World Data for Healthcare Research in China: Call for Actions. Value Health Reg Issues, 2022; 27: 72–81. DOI: https://doi.org/10.1016/j.vhri.2021.05.002
TFDA. Basic considerations for real-world evidence supporting drug development. Available from: https://www.fda.gov.tw/TC/siteLamp;id=33972%20.
Nishioka K, et al. Evolving Acceptance and Use of RWE for Regulatory Decision Making on the Benefit/Risk Assessment of a Drug in Japan. Clin Pharmacol Ther, 2022; 111(1): 35–43. DOI: https://doi.org/10.1002/cpt.2410
Agency P-PaMD. PMDA Updates. 2019; Available from: https://www.pmda.go.jp/files/000231353.pdf.
Agency P-PaMD. Basic principles on Utilization of Registry for Applications; 2021; Available from: https://www.pmda.go.jp/files/000240810.pdf.
Agency P-PaMD. Points to Consider for Ensuring the Reliability in Utilization of Registry Data for Applications. 2021; Available from: https://www.pmda.go.jp/files/000240811.pdf.
Lou J, et al. Real-world data for health technology assessment for reimbursement decisions in Asia: current landscape and a way forward. Int J Technol Assess Health Care, 2020; 36(5): 474–480. DOI: https://doi.org/10.1017/S0266462320000628
Berger ML, et al. Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Value Health, 2017; 20(8): 1003–1008. DOI: https://doi.org/10.1016/j.jval.2017.08.3019
REALISE Working Group. USE OF REAL-WORLD DATA AND REAL-WORLD EVIDENCE TO SUPPORT DRUG REIMBURSEMENT DECISION-MAKING IN ASIA 2020. 2021; Available from: https://hiper.nus.edu.sg/wp-content/uploads/2021/05/REALISE-Abridged-guidance-for-users-of-HTA_20201104-version-1.0-2.pdf.
Leelahavarong P, et al. Health Technology Assessment in Thailand: Institutionalization and Contribution to Healthcare Decision Making: Review of Literature. Int J Technol Assess Health Care, 2019; 35(6): 467–473. DOI: https://doi.org/10.1017/S0266462319000321
Trerayapiwat K, et al. Using real world evidence to generate cost-effectiveness analysis of fibrinolytic therapy in patients with ST-segment elevation myocardial infarction in Thailand. Lancet Reg Health West Pac, 2022; 26: 100503. DOI: https://doi.org/10.1016/j.lanwpc.2022.100503
iDSI Website. 2022; Available from: https://www.idsihealth.org/our-impact/philippines/.
Department of Health Research Ministry of Health and Family Welfare Government of India. Health Technology Assessment in India. Available from: https://nhm.gov.in/New_Updates_2018/Innovation_summit/6th/Health_Technology_Assessment_in_India_Deptt_of_Health_Research.pptx.
Dang A, Vallish BN. Real world evidence: An Indian perspective. Perspect Clin Res, 2016; 7(4): 156–160. DOI: https://doi.org/10.4103/2229-3485.192030
Dabak SV, et al. Budgeting for a billion: applying health technology assessment (HTA) for universal health coverage in India. Health Research Policy and Systems, 2018; 16(1): 115. DOI: https://doi.org/10.1186/s12961-018-0378-x
McNair D, et al. Use of RWE to Inform Regulatory, Public Health Policy, and Intervention Priorities for the Developing World. Clinical Pharmacology & Therapeutics, 2022; 111(1): 44–51. DOI: https://doi.org/10.1002/cpt.2449
Farghaly M, et al. Promise of Real-World Evidence for Patient Centricity in Gulf Cooperation Council Countries: Call to Action. Drugs Real World Outcomes; 2022. DOI: https://doi.org/10.1007/s40801-022-00336-0
Akhras KS, Alsheikh-Ali AA, Kabbani S. Use of real-world evidence for healthcare decision-making in the Middle East: practical considerations and future directions. Expert Rev Pharmacoecon Outcomes Res, 2019; 19(3): 245–250. DOI: https://doi.org/10.1080/14737167.2019.1568243
Elsisi GH, et al. Recommendations for Reporting Pharmacoeconomic Evaluations in Egypt. Value in Health Regional Issues, 2013; 2(2): 319–327. DOI: https://doi.org/10.1016/j.vhri.2013.06.014
INAHTA. Health Technology Assessment (HTA) Glossary. 2006; Available from: http://aaz.hr/resources/pages/55/INAHTA%20Health%20Technology%20Assessment%20(HTA)%20Glossary.pdf.
International labour organization. Extending Social Health Protection in Lebanon; 2020. Available from: https://www.ilo.org/wcmsp5/groups/public/---arabstates/---ro-beirut/documents/publication/wcms_780987.pdf.
Iessa N, et al. Smart Safety Surveillance (3S): Multi-Country Experience of Implementing the 3S Concepts and Principles. Drug Saf, 2021; 44(10): 1085–1098. DOI: https://doi.org/10.1007/s40264-021-01100-z
Hollingworth S, et al. Health technology assessment in sub-Saharan Africa: a descriptive analysis and narrative synthesis. Cost Eff Resour Alloc, 2021; 19(1): p. 39. DOI: https://doi.org/10.1186/s12962-021-00293-5
Usercentrics. South Africa’s Protection of Personal Information Act (POPIA).
Liu M, et al. Toward a better understanding about real-world evidence. European Journal of Hospital Pharmacy, 2022; 29(1): 8. DOI: https://doi.org/10.1136/ejhpharm-2021-003081
Gray CM, et al. A Framework for Methodological Choice and Evidence Assessment for Studies Using External Comparators from Real-World Data. Drug Saf, 2020; 43(7): 623–633. DOI: https://doi.org/10.1007/s40264-020-00944-1
ARGMD. Australian regulatory guidelines for medical devices. 2022; Available from: https://www.tga.gov.au/resources/resource/guidance/australian-regulatory-guidelines-medical-devices-argmd.
Health Canada. Canada Medical Device Regulations. 2022; Available from: https://laws-lois.justice.gc.ca/eng/regulations/sor-98-282/.
MDCG. MDCG 2020-6: Regulation (EU) 2017/745: Clinical evidence needed for medical devices previously CE marked under Directives 93/42/EEC or 90/385/EEC A guide for manufacturers and notified bodies.
IMDRF. Methodological Principles in the Use of International Medical Device Registry Data; 2016. Available from: https://www.imdrf.org/consultations/methodological-principles-use-international-medical-device-registry-data.
IMDRF. Patient Registry: Essential Principles. 2016; Available from: https://www.imdrf.org/consultations/patient-registry-essential-principles.
Sedrakyan A, et al. Advancing the Real-World Evidence for Medical Devices through Coordinated Registry Networks. BMJ Surgery, Interventions, & Health Technologies, 2022; 4(Suppl 1): e000123. DOI: https://doi.org/10.1136/bmjsit-2022-000148
IMDRF. Tools for Assessing the Usability of Registries in Support of Regulatory Decision-Making. 2018; Available from: https://www.imdrf.org/documents/tools-assessing-usability-registries-support-regulatory-decision-making.
Wang SV, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. Bmj, 2021; 372: m4856. DOI: https://doi.org/10.1136/bmj.m4856
Grimberg F, et al. The Real-World Data Challenges Radar: A Review on the Challenges and Risks regarding the Use of Real-World Data. Digit Biomark, 2021; 5(2): 148–157. DOI: https://doi.org/10.1159/000516178
Li M, et al. Integrating Real-World Evidence in the Regulatory Decision-Making Process: A Systematic Analysis of Experiences in the US, EU, and China Using a Logic Model. Front Med (Lausanne), 2021; 8: 669509. DOI: https://doi.org/10.3389/fmed.2021.669509
FDA. Considerations for the Use of Real-World Data and RealWorld Evidence to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry. 2021; Available from: https://www.fda.gov/media/154714/download.
FDA. Data Standards for Drug and Biological Product Submissions Containing Real-World Data Guidance for Industry. 2021; Available from: https://www.fda.gov/media/153341/download.
FDA. Use of Electronic Health Record Data in Clinical Investigations Guidance for Industry. 2018; Available from: https://www.fda.gov/media/97567/download.
FDA. Randomized, observational, interventional, and real-world—What’s in a name? 2020; Available from: https://www.fda.gov/media/155273/download.
FDA. Real-World Evidence — Where Are We Now? 2022; Available from: https://www.fda.gov/media/158581/download.
CADTH. Real-World Evidence for Decision-Making. 2022; Available from: https://www.cadth.ca/real-world-evidence-decision-making.
Administration N-CNMP. Notice of the State Food and Drug Administration on Issuing the Guiding Principles (Trial) for Real-World Evidence Supporting Drug Development and Review (2020 No. 1); 2020. Available from: https://www.nmpa.gov.cn/yaopin/ypggtg/ypqtgg/20200107151901190.html.
Administration N-CNMP. Technical Guidelines for Real-World Research to Support the Development and Evaluation of Children’s Drugs (Trial) (No. 22, 2020); 2020. Available from: https://www.nmpa.gov.cn/directory/web/nmpa/xxgk/ggtg/qtggtg/20200901104448101.html.
Administration N-CNMP. Notice of the State Food and Drug Administration on Issuing the Technical Guiding Principles (Trial Implementation) of Real World Data for Clinical Evaluation of Medical Devices (2020 No. 77); 2020. Available from: https://www.nmpa.gov.cn/xxgk/ggtg/qtggtg/20201126090030150.html.
Administration N-CNMP. Attachment: “Guiding Principles of Real World Data Used to Generate Real World Evidence (Trial)”; 2021. Available from: https://redica.com/wp-content/uploads/NMPA_-Attachment_-_Guiding-Principles-of-Real-World-Data-Used-to-Generate-Real-World-Evidence-Trial_.pdf.
PMDA RWGo. Guidelines for the Conduct of Pharmacoepidemiological Studies in Drug Safety Assessment with Medical Information Databases; 2014. Available from: https://www.pmda.go.jp/files/000240951.pdf.
PMDA RWGo. Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Drugs; 2018. Available from: https://www.pmda.go.jp/files/000243761.pdf.
PMDA RWGo. Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Medical Devices; 2018. Available from: https://www.pmda.go.jp/files/000243762.pdf.
PMDA RWGo. Procedures for Developing Post-marketing Study Plan (originaly published as “Procedures for Developing Post-marketing Study Plan”. 2018; Available from: https://www.pmda.go.jp/files/000226080.pdf.
PMDA RWGo. Questions and Answers (Q&A) on Points to Consider for Ensuring the Reliability of Post-marketing Database Study for Drugs. 2019; Available from: https://www.pmda.go.jp/files/000243763.pdf.
group RWDIAfHTAiRRw. Use of Real-World Data and Real-World Evidence to Support Drug Reimbursement Decision-Making in Asia; 2020. Available from: https://hiper.nus.edu.sg/wp-content/uploads/2020/12/REALISE-Full-guidance_updated-20201101.pdf.
Policy D-MCfH. Real-World Evidence Collaborative. Available from: https://healthpolicy.duke.edu/projects/real-world-evidence-collaborative.
Assessment HNGHT. Framework for the Next Generation Health Technology Assessment (HTA) to support patient-centered, societally oriented, real-time decision-making on access to and reimbursement for health technologies throughout Europe. Available from: https://www.htx-h2020.eu/about-htx-project/.
ΙΜΙ. Innovative Medicines Initiative’s collaborative research projects. Available from: https://www.imi.europa.eu/about-imi.
Initiative CTT. REAL-WORLD DATA. Available from: https://ctti-clinicaltrials.org/our-work/novel-clinical-trial-designs/real-world-data/.
DUPLICATE, R. Randomized Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology initiative. Available from: https://www.rctduplicate.org/.
ADAPT-SMART. Accelerated Development of Appropriate Patient Therapies: A Sustainable, Multi-Stakeholder Approach From Research to Treatment Outcomes. Available from: https://www.infographic.adaptsmart.eu/.
initiative, BDfBO. European research programme aiming to develop key enablers to support health care system transformation through the use of big data. Available from: https://bd4bo.eu/index.php/about-the-programme/about-bd4bo/.
Harmonisation ICf. ICH Reflection on “GCP Renovation”: Modernization of ICH E8 and Subsequent Renovation of ICH E6. 2017; Available from: https://admin.ich.org/sites/default/files/2019-04/ICH_Reflection_paper_GCP_Renovation_Jan_2017_Final.pdf.
Network EHDE. European Health Data & Evidence Network (EHDEN), an IMI 2 consortium with 23 partners operating in Europe. Available from: https://www.ehden.eu/.
(CIOMS) CfIOoMS. Working Group XIII – Defining Intent, and Guiding Harmonization and Ethics Standards for Real-World Data and Real-World Evidence in Regulatory Decision-Making; 2022. Available from: https://cioms.ch/wp-content/uploads/2020/03/CIOMS_WG_XIII_14thMeeting_minutes_4July2022_web.pdf.
(ISPOR) ISfPaOR. Real World Evidence Strategic Initiative Available from: https://www.ispor.org/strategic-initiatives/real-world-evidence.
(ISPOR) ISfPaOR. Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness. Available from: https://www.ispor.org/heor-resources/good-practices/article/good-practices-for-real-world-data-studies-of-treatment-and-or-comparative-effectiveness-recommendations-from-the-joint-ispor-ispe-special-task-force-on-real-world-evidence-in-health-care-decision-making.
(ISPOR) ISfPaOR. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Available from: https://www.ispor.org/heor-resources/good-practices/article/reporting-to-improve-reproducibility-and-facilitate-validity-assessment-for-healthcare-database-studies-v1.0.
(ISPOR) ISfPaOR. Making Real-World Evidence More Useful for Decision Making. 2017; Available from: https://www.ispor.org/docs/default-source/strategic-initiatives/rwe-healthcare-decision-making-guideline.pdf?sfvrsn=56d9b92_2.
(ISPOR) ISfPaOR. Real World Data Good Practice for Outcomes Research Reports are listed by date published (most recent first). Available from: https://www.ispor.org/heor-resources/good-practices/report/-in-category/categories/real-world-data-information-systems.
O’Neill T, et al. ISPOR, the FDA, and the Evolving Regulatory Science of Medical Device Products. Value Health, 2019; 22(7): 754–761. DOI: https://doi.org/10.1016/j.jval.2019.03.020
(ICMRA) ICoMRA. ICMRA meeting: COVID-19 Real-World Evidence and Observational studies; 2020. Available from: https://www.icmra.info/drupal/en/news/22july2020/summary.
(INAHTA) INoAfHTA. a network of 50 HTA agencies that support health system decision-making. Available from: https://www.inahta.org/.
(ISPE) ISfPE. ISPE’S POSITION ON REAL-WORLD EVIDENCE (RWE); 2020. Available from: https://pharmacoepi.org/pub/?id=136DECF1-C559-BA4F-92C4-CF6E3ED16BB6.
Miclăuş T, et al. Impact of Design on Medical Device Safety. Therapeutic Innovation & Regulatory Science, 2020; 54(4): 839–849. DOI: https://doi.org/10.1007/s43441-019-00022-4