1.1. Clinical data collection

Collection of high-quality clinical data is vital in clinical research and evidence collection. To this end, identification of clinical research gaps from all involved stakeholders is required, and the entire processes of data collection must be thoroughly discussed in the related Standard Operating Procedures [3]. Additionally, sponsors are obliged to monitor (either on-site or remotely) all research activities, including clinical data collection [4]. For example, on-site or remote monitoring should evaluate and identify potential data collection errors, compliance with protocols, completeness of collected data, unusual data distribution, and review of data collection in real-time (data supervision) [4].

Potential clinical data collection sources may include (but are not limited to) clinical trials (CTs), clinical investigations (CIs), electronic health records/medical records, administrative data (e.g., hospital records, pharmaceutical prescriptions), health and post-market clinical follow-up surveys, and patient registries [5, 6, 7, 8, 9] (See Figure 2). The MDR introduces post-market surveillance (PMS), and in particular, post-market clinical follow-up (PMCF) as a clinical data source [1]. Clinical data analysed and reported at a PMCF level are part of a continuous process that updates the clinical evidence substantiated in the clinical evaluation report (according to EU MDR 2017/745 [1]). This requirement for continuous monitoring and collection of safety and performance data is fulfilled by the collection of real-world data (RWD) [10]. RWD, according to the FDA Framework for Real-World Evidence Program, is the routinely collected data relating to patient health status and/or the delivery of health care, and is applicable to medical products in general (not just medical devices) [11]. Real world evidence (RWE) is the result of the analysis of RWD and provides evidence on the performance and the potential benefits or risks of the product during real-world use.

The type of data to be collected will determine the methods and processes used for data collection and data analysis. Paper-based, as well as electronic and hybrid methods (combination of paper-based and electronic means), are the most common methods used for data collection activities [8, 12]. Depending on the design of the clinical study and the related endpoints, clinical data are generated and/or collected by the investigator(s), research staff, and/or participants/patients [13]. Clinical research studies (CTs and CIs included) must be conducted in accordance with appropriate scientific principles, good manufacturing practices (GMP), and ethical considerations (Helsinki declaration, good clinical practice (GCP)) to ensure that meaningful data is generated [1, 14, 15, 16]. From the early stages of participant/patient recruitment to the final stages of data analysis and reporting, clinical data management (CDM) plays a vital role in data handling by monitoring and ensuring compliance with regulations and standards [13, 17, 18]. CDM processes are essential for the design of all clinical data collection steps, providing an “error-free, valid, and statistically sound” data collection agenda [17].

The process of clinical data collection from CTs/CIs can be both time-consuming and costly and can therefore hinder the development of new health-related innovations [19]. Costs can arise both internally from the sponsor and externally from participant/patient recruitment, data management and monitoring, and regulatory planning and fees, to name a few [19, 20]. Clinical data collection in drug development studies is a great example of cost inefficiency, as multiple factors (e.g., variation of drug type, regulatory policies, therapies, manufacturer’s company size) create ongoing and substantial study costs [19, 21, 22, 23]. An overview of the elements involved in clinical data collection and the associated costs is shown in Figure 3.

1.2. Sufficient clinical evidence and real-world evidence (RWE)

Sufficient clinical evidence refers to the “amount and quality of the clinical data and the clinical evaluation results which allow a qualified assessment of whether the device is safe and achieves the intended clinical benefits when used as intended by the manufacturer” [24]. Sufficient clinical evidence is required to evaluate potential (new) risks associated with the device under normal conditions of use, as well as to identify and substantiate the device’s clinical benefit (i.e., the positive impact of a device on the health of an individual or the public [1]). Sufficient clinical evidence is necessary to support a positive benefit-risk profile of a device.

One of the major challenges for manufacturers under the EU MDR is to be able to provide sufficient clinical evidence to support conformity with the General Safety and Performance Requirements (GSPRs). This is due to the more strict criteria for what constitutes an acceptable source of clinical data (i.e. some sources previously accepted under MDD are no longer accepted under MDR), the up-classification of devices (e.g. stand-alone software can now potentially be categorized as a Class III medical device [25]) and the strict requirements around equivalent devices. For example, for a manufacturer to claim equivalence, the device in question needs not only to meet requirements for similarity on technical, biological and clinical characteristics to the device to which they are claiming equivalence, but they must also be able to prove that they have sufficient levels of access to the reference device’s technical documentation [1].

One of the sources of clinical data, as suggested in the MDR, is from the real-world use of the device (clinical experience). Although MDR doesn’t use the term ‘real-world data’ (RWD) or ‘real-world evidence’ (RWE), RWD appears as a term in MDCG 2020–6 [10]. The term RWE has been used more recently by the FDA, such as in the 2017 FDA guidance on “Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices” [26], and the 2018 “Framework for FDA’s Real-World Evidence Program” [11]. Based on this, we use the term RWE. Sources of RWE can be seen in Figure 4.

Collection of RWE can be particularly useful to substantiate the continued acceptability of the benefit-risk profile throughout the lifetime of a medical device. For example, under the EU MDR there is a strict requirement to conduct a post-market clinical follow-up (PMCF) (e.g. with the collection of RWE) in order to proactively monitor, collect and analyse performance and safety of a device throughout its entire lifetime, regardless of device class, legacy and technology [1].

In the USA, medical devices are regulated by the FDA based on the Medical Device Amendment of 1976 [27]. The vast majority of submissions are via the 510(k) pathway, with an average of 2,982 510(k) approvals between 2015–2020, compared to only 38 via Pre Market Approvals (PMA) [28]. The difference is indicative of the requirements of the two regulatory pathways; whereas a 510(k) submission is based mainly on substantial equivalence of an already marketed device, with minimal additional pre-market clinical data generated for the target device, the PMA pathway is based on the generation of clinical data through clinal investigations. Although clinical data on the device in scope is rarely requested by the FDA for 510(k) submissions [29], the need for device-specific clinical data arises from PMS requirements, which dictate the monitoring and collection of RWE on the safety and performance of the device (Class I devices are exempt from this requirement) [30].

The collection of RWE has become of greater focus under the FDA due to the FDA RWE program. Within the drug sector, the FDA’s RWE program aims to optimize the use of RWE for regulatory decisions, specifically the use of RWE to support drug effectiveness and approval of a new indication for an already approved drug, or to support post-approval study requirements [11]. In particular, the program aims to evaluate the use of RWE for supporting labelling changes on drug effectiveness, adding/modifying an indication, or changing dose/regimen/administration, amongst others [11]. Although the RWE program is currently focused on drugs, recent guidance from the FDA on the use of RWE to support with regulatory decision-making within the medical devices sector has also been made available [26]. For example, the FDA proposes that RWE may be able to provide information on a broader patient population compared to a clinical trial or investigation, and has potential implications for reimbursement decisions [26]. As such, the collection of clinical data, particularly RWE, is at the forefront of medical device and drug regulations.

3.1. Value of DHT in clinical research

There are numerous beneficial uses of DHT in clinical research. In 2016, a think-tank involving various stakeholders (academia, industry, and regulatory bodies) identified numerous potential uses of DHT that span the entire process of a clinical study, from fundraising and recruitment to data management (see Figure 6) [42].

DHTs can aid substantially with the recruitment process, leading to a reduction in costs and reduced timeframes [42, 43, 44]. Challenges with recruitment include costs of advertising, lack of enrolment and difficult-to-reach populations (e.g. rare medical conditions) [43]. Evidence suggests that social media platforms can help researchers reach segments of the population that may otherwise be difficult to access, or to identify participants from their desired demographic and eligibility criteria [43]. Additionally, the use of DHT makes participation easier by removing potential issues such as travel, time restraints and access to the investigation site, as data can be collected remotely through a device or application [32, 42]. This can aid with recruitment, as shown by a recent survey that found participants preferred a mobile study compared to a conventional one due to negating the need to travel (amongst other reasons) [45]. Furthermore, evidence suggests that a significant proportion of patients face substantial travel burdens with regards to cost and distance when taking part in clinical trials [46].

Recently, DHTs have also been used to gain informed consent from participants, for example via apps and video [42]. The idea behind these methods is to make the consent procedure easier to understand so that participants are fully aware of the potential risks and benefits involved. However further development is needed to prevent potential issues with competency, identity (i.e., ensuring the participant is the one giving the consent) and consenting without actually reading the information (i.e. ticking the relevant boxes but not reading the information) [42].

The use of DHTs enables the remote collection of data, which not only aids with recruitment (as previously discussed) but also allows for continuous and more frequent data collection (e.g., during daily activity and even sleep) and from numerous environments (e.g., at home, work, outdoors) [32, 44]. Costs related to the investigational site will also be removed/reduced, as a physical investigational site may not be needed if all data can be collected remotely. More frequent data collection with increased participation means that potentially the duration of clinical studies can be reduced, as the required data can be collected over shorter periods of time. This reduction in duration may in turn lead to reduced costs, as the shorter the investigation the less operating costs involved. However, whether shorter durations are appropriate is entirely dependent upon the purpose of the clinical study, as if the aim is to assess the safety and performance of a device over a specific time-period, then the length of the clinical study cannot be reduced.

Another potential benefit of DHTs is that they enable the collection of data from novel endpoints and from participants that would otherwise not normally be able to provide data [32, 42]. For example, wearable devices can collect physiological data (such as heart rate) from pre-verbal infants and from those that may struggle to communicate or are cognitively impaired (subject to ethical considerations, such as informed consent) [32]. This is particularly useful for devices that are intended to be used in these types of populations, as it provides external validity to the findings. It’s also possible to use multiple DHTs to collect a range of information, including clinical, physiological, psychological, behavioural, and functional data [32]. This data may provide information on endpoints that traditionally are not captured. For example, a think-tank from 2016 [42] identified a clinical trial called the NEATHFpEF (Nitrate’s Effect on Activity Tolerance in Heart Failure With Preserved Ejection Fraction) which evaluated patients with heart failure when using a 6-week dose-escalation regimen of isosorbide mononitrate or a placebo [47]. Daily activity level was the primary endpoint, as measured by an activity monitor, and was found to be lower in the interventional group compared to the placebo. However, when activity was measured using the more traditional 6-min walking distance, no significant difference was found, suggesting that the endpoint obtained via the DHT (activity monitor) was more sensitive than that of the more traditional one [47]. As such, endpoints that can be collected from DHTs may provide additional, and even superior, data when used alongside more conventional endpoints [42].

Regarding data management, all data captured from DHTs can be transmitted directly to the relevant authorized party (e.g., investigators, sponsors etc), making a more efficient way of sharing data. This can also be done in a way that keeps all parties involved blinded/masked to the clinical study conditions [32]. This is important to ensure potential bias is minimized when interpreting the data (e.g., interpreting data in a more favourable way for the device under investigation). Data can also be shared directly with the participants, which may keep patients engaged and motivated in the study, particularly as recent research indicates that access to data is something that participants want [45].

As such, there are numerous beneficial uses for DHT within clinical research, that ultimately will aid with reducing the time and costs involved in clinical studies, whilst also removing potential barriers to participant participation.

3.2. Considerations and challenges for selecting and using DHT in clinical research

As previously outlined, there are numerous types of DHTs that can be used to aid with the collection of clinical data, from software to hardware and combination devices. The choice of DHT will mostly depend on the characteristics of the clinical study to be conducted. In particular, the clinical characteristics of the disease/condition of interest, the population and the design of the clinical study will influence which DHT to use [32]. For example, a clinical study that aims to assess the performance of a biopsy needle in paediatrics will need a DHT (e.g., digital microscopy) that is able to measure the appropriate endpoints of biopsy procedures (such as quality of specimen) in a paediatric population (which may entail a different DHT for use in an adult population). In some cases, it may be that a participant can use their own DHT (e.g., a commercial activity tracker) and/or general-purpose computing platform (e.g., mobile phone) to collect relevant data. The benefit of this is that the participant is already using the device and is familiar with how to use it and, as such, are more likely to use it consistently and correctly. However, they may not be appropriate for clinical studies that require highly specialized or customized measurements [32], moreover, the data collected from different devices may not be comparable [48]. For example, a recent survey conducted with academic, government, regulatory, and health care experts with experience with RWE raised concerns about the credibility, accuracy and reliability of algorithms that are not clinically tested [48]. Furthermore, algorithms that differ across vendors may not produce data that is comparable (i.e. algorithms across devices may measure endpoints differently) [48].

Regardless of the choice of DHT, all DHTs must be fit-for-purpose, that is, they should have sufficient validation and verification to support their use and interpretability of the data they collect [32, 49]. Verification is needed to ensure the DHT accurately measures what it intends to measure, whereas validation is needed to ensure it assesses the clinical event/characteristic in the proposed population [49]. All DHTs should be able to yield accurate, consistent results and be reliable over time [50].

As well as being fit-for-purpose, an ideal DHT should consider various additional elements, such as design features, intended population, and risks. For example, the Clinical Trials Transformation Initiative (CTTI) have developed a framework of elements to consider when selecting mobile health technology [49], which includes measurement performance, data access, and human factor specifications, amongst others [49]. Table 2 provides an overview of various elements that should be considered when selecting the most appropriate DHT.

With regards to design and population considerations, usability studies can be used to obtain feedback on ease of use from populations similar to the intended population and subsequently be used to improve the design of the DHT and/or ensure that the most suitable DHT is used for the intended population [32]. Additionally, feasibility studies can investigate the tolerability of the DHT with the intended population, which in turn will affect the willingness of the participant to use the DHT for the intended duration [49]. For DHTs that are wearable, mobile or contain biosensors, specific issues regarding technological literacy, anxiety regarding the tracking of data and retention issues over the long-term require additional consideration [44, 51]. For example, evidence suggests that there is an educational, wealth and age division with regards to the use of digital technology, with younger, wealthier, and higher educated individuals more likely to use digital technologies [44, 52]. This may result in the exclusion of specific sub-populations, causing population bias and loss of data from potentially relevant participants [44].

A key aspect that must be considered for all DHTs is the ability to provide meaningful data that can be statistically analysed. This can be a challenge if a DHT is able to collect a large amount of various behavioural and physiological data. To this end, Taylor et al (2020) [53] have developed a framework for the development of digital endpoints based on measurements from DHTs. They propose two types of endpoints: data-centric endpoints and patient-centric endpoints. Data-centric endpoints are intended to measure the outcome of interest (e.g., disease progression), whereas patient-centric endpoints measure how the patient feels/is functioning. The framework for both types of endpoints is built upon ensuring statistical robustness and validity to ensure that meaningful data is measured (see [53] for details). Likewise, the Digital Medicine Society (DiMe) have also developed a framework to help with the development and selection of digital endpoints [54]. For example, they provide guidance on developing meaningful and measurable endpoints using a three-step approach: 1) Meaningful Aspect of Health (MAH), 2) Concept of Interest (COI), and 3) Outcome to be measured. An example provided is that of Parkinson’s disease, in which MAH may be defined as the ability to walk, COI as an activity (e.g., walking ability) and the Outcome to be measured as the number of walking bouts per day. The endpoint would be the percentage of patients that had an increase in walking bouts from baseline [54].

Importantly, as with any medical device, potential risks to the patient/user using the DHT must be considered. The FDA emphasizes that DHT-related risks need to be reviewed in conjunction with risk management planning and to address any potential problems that may occur when using the DHT, and furthermore, address how these problems may be managed/resolved [32]. Potential risks may be in relation to physical properties of the device (e.g., the material may cause injury/irritation) or to erroneous data (e.g., inflated readings). For instance, a DHT that measures glucose levels has the potential to give inaccurate readings, leading to the potential risk of hypoglycaemia events or inappropriate treatment. Other potential risks include breach of data, which is one of the biggest concerns surrounding DHT. Recent data hacking scandals have highlighted the issues with keeping data secure (see [42] for an overview) and as such, DHTs need to be able to protect patient’s data from cybersecurity threats.

Lastly, the costs of using the DHT should be assessed, which include not only the unit cost of the DHT (plus surplus devices in case of malfunctions), but training of both researchers and participants on how to use the device correctly [32, 51]. Appropriate training for both research personnel and participants on how to use the DHT is needed, as without a clear understanding of how to use the device, inaccurate data may be collected. Participants should also understand exactly what data will be collected and the relevant privacy and data protection processes that are in place and provide consent for the collection and analysis of their data. In addition, a plan should be in place to provide technical assistance when necessary to avoid data loss/error [32, 51].

To help address these practical considerations and challenges, various guidelines and regulations concerning the use of DHTs have recently been published. For example, the CTTI has published recommendations and various resources for those that wish to conduct digital health trials, with resources dedicated to developing novel endpoints that accurately reflect meaningful outcomes to patients [55] and testing a DHT via feasibility, verification and validation processes [56], amongst others. An overview of the current regulatory frameworks and guidelines surrounding DHT are presented below.

3.3. Current regulatory frameworks and guidelines

3.3.1. DHT guidelines and frameworks

The most recent guidance to be published on DHT is the FDA draft guidance on “Digital Health Technologies for Remote Data Acquisition in Clinical Investigations” [32]. This draft guidance provides recommendations for all relevant stakeholders about the potential use of DHTs for the remote collection of clinical data. The guidance covers the benefits of using DHT (as discussed in Section 3.1) as well as all aspects that should be considered, such as technology selection, potential risks, and how to choose an appropriate DHT (as discussed in Section 3.2) [32]. Furthermore, examples of different types of DHT and their uses in different scenarios are provided, along with an example of how to select a DHT for use in a clinical investigation [32]. As such, the guidance provides a comprehensive approach for stakeholders to assess the value that DHT can bring to data collection in CIs and the processes and considerations required to implement DHT.

In addition, the guidance touches upon the use of DHTS as Drug Development Tools (DDT) or Medical Device Development Tools (MDDT) [32]. In the US, manufacturers can chose to get their DHTs qualified as a DDT or MDDT for use in a specific context [32]. Qualified DHTs can then be used in multiple CIs with the same context of use (without needing to go through qualification each time), to support premarket submissions of drugs or medical devices (DDT or MDDR, respectively) [32]. Of further note is that some DHTs themselves may qualify as a medical device under the FDA definition (section 201(h) of the FD&C Act) [32]. Although the guidance does not cover whether a DHT meets the definition of a medical device, it does note that medical devices that are intended to be used in CIs are exempt from most of the requirements, provided that the CI itself complies with applicable requirements under 21 CFR part 812 [32].

Other FDA guidance of relevance to the use of DHT in CIs includes the Computerized Systems used in Clinical Investigations [57], the draft guidance on Electronic Records and Electronic Signatures in Clinical Investigations [58] and the Electronic Source Data in Clinical Investigations [59]. For example, the later aims to help with ensuring reliability, quality, integrity, and traceability of electronic source data that is captured and submitted into an electronic case report form (eCRF) [59]. This electronic source data may come from a DHT and can be directly transmitted to the eCRF.

Within Europe, the process of incorporating DHTs in RW clinical data acquisition is less advanced [60]. Although the use of DHTs has been considered a strategic EU health priority already since 2012 [61], there is currently no harmonised regulatory framework for use of DHTs in clinical research. Furthermore, the EU Clinical Trials Regulation (EU 536/2014) does not provide any provision for clinical data acquisition via DHTs. In a recent Reflection paper by the European Federation of Pharmaceutical Industries and Associations (EFPIA), it is recognised that a manufacturer/sponsor will need to navigate through complex regulatory pathways in order to use DHTs in clinical development or as a combination product with (investigational) medicinal products [62]. In this Reflection paper, EFPIA suggests a number of initiatives to support the routine use of DHTs, which are expected to “help to improve patient care” [62]. Additionally, the European Commission is planning the establishment of a European Health Data Space to “promote better exchange and access to different types of health data” [63]. Despite this, a central European regulatory harmonisation for DHT application in healthcare is still in its infancy, with individual countries, such as Germany, already making progress.

Within Germany, the use of DHTs (specifically, digital health applications) is part of the 2019 Digital Healthcare Act (Digitale-Versorgung-Gesetz, DVG). This act introduced the “app on prescription”, which entails that all individuals on the German statutory health insurance have access to digital health applications (DiGa) [64]. The DiGa directory provides a list of all digital health applications that are reimbursable. Any DiGa application is subject to a “Fast-Track” process in which the German regulatory authority (BfArM) has 3-months to assess the application in terms of its positive healthcare effect,² security, functionality, quality, data protection and usability [64]. To enter the Fast-Track pathway, a digital health application should meet certain provisions, including being classified as low-risk device (class I or IIa under MDR) that is used primarily by the patient to support collection of either patient-reported outcome measures (PROMs) or patient-reported experience measures (PREMs) [60].

The Fast-Track pathway supports generation of clinical evidence to support DHT approval by other approaches (e.g., collection of RWE). For example, the reSET from Pear Therapeutics was granted marketing clearance with use of RWE (collected from observational studies) on the device’s performance and usability [60]. However regulatory uncertainty precludes manufacturers from using this approach, with RCTs still considered the “safest” approach for regulatory approval [60]. Nonetheless, experience from Germany’s Fast-Track program suggests that further study and ongoing discussions should focus on reaching a consensus for best practices in the following areas: missing data, study endpoints, selection of a comparator group, multimodal interventions, hypothesis testing standardisation, equity in the included population(s), generalisability, confounders and Fit for purpose use of RWE [60].

In the UK, the NICE Evidence standards framework for DHTs describes the type and level of evidence needed to show the value that a DHT brings to the UK health and social care system [65]. The standard of evidence required is based upon the tier of the DHT (Tier A: system impact; Tier B: Understanding and communicating; Tier C: Interventions), which specifies the function and level of risk. Each tier requires different types of evidence, with both minimum evidence standards and best practice standards presented [65]. This framework thus ensures that all DHTs used with the UK system are sufficiently supported by appropriate evidence. Figure 7 provides an overview of the current implementation of DHTs by country.

Figure 7 

Overview of DHT implementation by country. * See Lovell [66]. ** See Kanta [67].

3.4. Implication of DHT within regulatory decision-making processes

As previously outlined, clinical research designs are evolving to incorporate the use of DHTs (Table 1), which in turn is enhancing the way data is collected, analysed and shared, from the use of remote data collection in real-world settings, to the generation of novel and patient-centred endpoints (as discussed in Section 3.1).

The recent regulations and guidance focusing on RWE collection via DHTs (see section 3.3) paves the way for the wider implementation of DHTs in clinical research, and subsequently, within regulatory decision-making processes. For example, RWE is of particular value for reimbursement decisions in Asia, due to a low number of clinical trials being conducted as a result of the financial, ethical and operational barriers, amongst others [74]. This means that the Asian population may be under-represented in clinical trials, resulting in potential differences between populations (e.g., Caucasians and Asians) being overlooked, such as biological (e.g., genetics) and societal differences (e.g. practice guidelines) [74]. RWE collected via DHTs could help inform reimbursement decisions where clinical trial data is lacking, and moreover, provide data on the long-term safety and performance which cannot be addressed in the limited duration of a clinical trial. [74] Indeed, a survey by Lou et al (2020) established that Health Technology Assessment (HTA) agencies in Asia already accept RWE to inform reimbursement, both as standalone evidence or supplementary [74], thus the groundwork for DHTs to aid with this is already laid.

More generally, RWE can provide data in situations where it may not be ethical or practical to conduct a clinical trial or investigation, such as for rare or life-threatening diseases, or for situations where recruitment or financial costs prevent the ability to run a clinical trial [75]. Moreover, RWE can provide information for HTA beyond that which clinical trials or investigations can provide. The strict and controlled environment in which clinical trials are conducted means that data obtained likely does not reflect the real-world use of the device. RWE, in theory, is thus more applicable to HTA agencies for reimbursement decisions, but in practice, is limited in use by potential bias introduced with the lack of experimental controls [76]. However, the recent guidelines on RWE and the use of DHTs may help to reduce potential bias in RWE collection, translating the use of RWE from theory to practice.