Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices - Guidance for Industry and Food and Drug Administration Staff

Contains Nonbinding Recommendations

Use of Real-World Evidence to

Support Regulatory Decision-Making

for Medical Devices

Guidance for Industry and

Food and Drug Administration Staff

Document issued on August 31, 2017.

The draft of this document was issued on July 27, 2016

For questions about this document regarding CDRH-regulated devices, contact the Office of

Surveillance and Biometrics (OSB) at 301-796-5997 or CDRHClinic[email protected].

For questions about this document regarding CBER-regulated devices, contact the Office of

Communication, Outreach, and Development (OCOD) at 1-800-835-4709 or 240-402-8010.

U.S. Department of Health and Human Services

Food and Drug Administration

Center for Devices and Radiological Health

Center for Biologics Evaluation and Research

Contains Nonbinding Recommendations

Preface

Public Comment

You may submit electronic comments and suggestions at any time for Agency consideration to

http://www.regulations.gov . Submit written comments to the Dockets Management Staff, Food

and Drug Administration, 5630 Fishers Lane, Room 1061, (HFA-305), Rockville, MD 20852.

Identify all comments with the docket number FDA-2016-D-2153. Comments may not be acted

upon by the Agency until the document is next revised or updated.

Additional Copies

CDRH

Additional copies are available from the Internet. You may also send an e-mail request to CDRH-

[email protected] to receive a copy of the guidance. Please use the document number

1500012 to identify the guidance you are requesting.

CBER

Additional copies are available from the Center for Biologics Evaluation and Research (CBER),

Office of Communication, Outreach, and Development (OCOD), 10903 New Hampshire Ave.,

Bldg. 71, Room 3128, Silver Spring, MD 20993-0002, or by calling 1-800-835-4709 or 240-402-

8010, by email, [email protected] or from the Internet at

http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guida

nces/default.htm.

Contains Nonbinding Recommendations

Table of Contents

I. Introduction and Scope ................................

......................................................................... 4

II. Background .......................................................................................................................... 5

III. Real-World Evidence ........................................................................................................... 8

IV. Regulatory Context in Which RWE May be Used ............................................................... 9

A. General considerations for the use of RWE ......................................................................... 9

B. Application of Investigational Device Exemption (IDE) Requirements in 21 CFR 812 to

the Collection of RWD .............................................................................................................. 11

V. Characteristics of RWD ...................................................................................................... 12

A. Relevance ........................................................................................................................... 13

B. Reliability ........................................................................................................................... 15

(1) Data accrual ................................................................................................................ 15

(2) Data assurance - Quality Control ................................................................................ 16

VI. Examples Where RWE is Used .......................................................................................... 17

A. Expanded Indications for Use ............................................................................................ 17

B. Postmarket Surveillance Studies (Section 522) ................................................................. 18

C. Post-Approval Device Surveillance as Condition of Approval ......................................... 19

D. Control Group .................................................................................................................... 19

E. Supplementary Data ........................................................................................................... 20

F. Objective Performance Criteria and Performance Goals ................................................... 20

VII. Glossary .............................................................................................................................. 21

Contains Nonbinding Recommendations

Use of Real-World Evidence to

Support Regulatory Decision-Making

for Medical Devices

Guidance for Industry and

Food and Drug Administration Staff

This guidance represents the Food and Drug Administration’s (FDA’s) current thinking on

this topic. It does not create or confer any rights for or on any person and does not operate to

bind FDA or the public. You can use an alternative approach if the approach satisfies the

requirements of the applicable statutes and regulations. If you want to discuss an alternative

approach, contact the FDA staff responsible for implementing this guidance. If you cannot

identify the appropriate FDA staff, call the appropriate number listed on the title page of this

guidance.

I. Introduction and Scope

FDA is issuing this guidance to clarify how we evaluate real-world data to determine whether

they are sufficient for generating the types of real-world evidence that can be used in FDA

regulatory decision-making for medical devices. This guidance is applicable to all devices, as

that term is defined under section 201(h) of the Federal Food, Drug, and Cosmetic Act (the

FD&C Act), including software that meets the definition of a device.

Real-World Data (RWD) are data relating to patient health status and/or the delivery of health

care routinely collected from a variety of sources.

Examples of RWD include data derived from electronic health records (EHRs), claims and

billing data, data from product and disease registries, patient-generated data including in home-

use settings, and data gathered from other sources that can inform on health status, such as

mobile devices. RWD sources (e.g., registries, collections of EHRs, and administrative and

healthcare claims databases) can be used as data collection and analysis infrastructure to support

many types of trial designs, including, but not limited to, randomized trials, such as large simple

trials, pragmatic clinical trials, and observational studies (prospective and/or retrospective ).

Real-World Evidence (RWE) is the clinical evidence regarding the usage, and potential

benefits or risks, of a medical product derived from analysis of RWD.

Contains Nonbinding Recommendations

Under the right conditions, data derived from real world sources can be used to support

regulatory decisions. RWD and associated RWE may constitute valid scientific evidence

depending on the characteristics of the data. This guidance should not be construed to alter, or

change in any way, the existing evidentiary standards applicable to FDA’s regulatory decision-

making; rather, it describes the circumstances under which RWD may be used to support a

variety of FDA decisions based on the existing evidentiary standards. While FDA encourages the

use of relevant and reliable RWD, this guidance neither mandates its use nor restricts other

means of providing evidence to support regulatory decision-making. This guidance highlights

some of the potential uses of RWD, and describes the factors that FDA considers when

evaluating whether specific RWD is of sufficient quality to inform or support a regulatory

decision. It also clarifies when an Investigational Device Exemption (IDE) may be needed to

prospectively collect and use RWD for purposes of determining the safety and effectiveness of a

device.

This document does not address the use of non-clinical data, adverse event reports, secondary

use of clinical trial data (e.g., post hoc analyses), or systematic literature reviews. Nor does it

address study design/conduct or analytical methodologies. While it does describe the factors that

FDA considers when evaluating RWD or RWE, it does not provide a specific set of pass/fail

criteria or other scoring tools for making a determination about the suitability of RWD or RWE

for a particular regulatory decision.

This guidance does not affect any federal, state or local laws or regulations, or foreign laws or

regulations that may be applicable to the use or collection of RWD, or that provide protections

for human subjects (including informed consent requirements) or patient privacy. This guidance

should be used to complement, but not supersede, other device-specific and good clinical

practice guidance documents.

FDA's guidance documents, including this guidance, do not establish legally enforceable

responsibilities. Instead, guidance documents describe the Agency’s current thinking on a topic

and should be viewed only as recommendations, unless specific regulatory or statutory

requirements are cited. The use of the word should in Agency guidance means that something is

suggested or recommended, but not required.

II. Background

To protect and promote the public health, FDA needs to understand and evaluate the available

evidence related to regulated products.

For medical devices, available evidence is traditionally

comprised of non-clinical and, in some cases, clinical studies conducted and provided to FDA by

the device manufacturer or sponsor. However, FDA recognizes that a wealth of RWD covering

medical device experience exists and is routinely collected in the course of treatment and

management of patients. Data collected during clinical care or in the home setting may not have

the same quality controls as data collected within a clinical trial setting. Even so, under certain

FDA’s What We Do (http://www.fda.gov/AboutFDA/WhatWeDo/default.htm).

Contains Nonbinding Recommendations

circumstances RWD may be of sufficient quality to help inform or augment FDA’s

understanding of the benefit-risk profile of devices at various points in their life cycle. RWD,

which are typically collected for non-regulatory purposes in EHRs, registries, and administrative

and claims data, may provide new insights into the performance and clinical outcomes associated

with medical device use. This information can potentially be used by sponsors to demonstrate

compliance with regulatory requirements and to aid FDA in our regulatory decision-making.

FDA has issued guidance documents on premarket and postmarket data collection,

benefit-risk

determinations,

patient preference information,

and expedited access to medical devices for

unmet medical needs in order to streamline the process for bringing new technologies to market

while maintaining the assessment of reasonable assurance of safety and effectiveness of medical

devices. FDA has also issued plans for, and has begun implementation of, the National

Evaluation System for health Technology (NEST)

to leverage RWD in order to more

quickly identify safety problems and to better understand the benefit-risk profile of devices used

in clinical care. FDA believes that, if leveraged correctly, the NEST may also help to reduce the

time and cost of generating the types of evidence used to support the marketing authorization of

FDA-regulated products and to meet postmarket study and reporting requirements.

Devices are often used in routine clinical practice for uses that are not within their cleared or

approved indications for use. However, the knowledge gained from all uses of a device in

medical practice is often not realized because the data collected are not systematically

characterized, aggregated, and analyzed in a way that can be relied upon to inform regulatory

decision-making. By recognizing the value of RWE as an important contributing factor for

understanding and regulating medical devices, we hope to encourage the medical community to

learn more from routine clinical care than we do today.

Balancing Premarket and Postmarket Data Collection for Devices Subject to Premarket Approval

(http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddev-gen/documents/document/ucm393994.pdf).

Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De

Novo Classifications (http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddev-

gen/documents/document/ucm517504.pdf)

Patient Preference Information – Voluntary Submission, Review in Premarket Approval Applications,

Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and

Device Labeling (Patient Preference Information – Voluntary Submission, Review in Premarket Approval

Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision

Summaries and Device Labeling

http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm446680.pdf).

Strengthening Our National System for Medical Device Postmarket Surveillance

(http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe

ports/UCM301924.pdf).

Strengthening Our National System for Medical Device Postmarket Surveillance: Update and Next Steps - April

2013 (http://www.fda.gov/downloads/MedicalDevices/Safety/CDRHPostmarketSurveillance/UCM348845.pdf).

Strengthening Patient Care: Building a National Postmarket Medical Device Surveillance System

(http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe

ports/UCM435112.pdf).

Recommendations for a National Medical Device Evaluation System: Strategically Coordinated Registry Networks

to Bridge the Clinical Care and Research - August 2015

(http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe

ports/UCM459368.pdf).

Contains Nonbinding Recommendations

FDA will use the criteria described in this guidance to evaluate whether RWD are of sufficient

quality to support regulatory decision-making, including potentially generating valid scientific

evidence.

FDA relies only upon valid scientific evidence to determine whether there is a

reasonable assurance that a device is safe and effective. However, it is possible that RWE can

meet this threshold under conditions where the underlying RWD were accurately and reliably

captured at clinically relevant time intervals throughout the device lifecycle. Under the right

conditions, RWE may be suitable to support the clearance or approval of a new device, or the

expansion of the indications for use of devices that are already on the market. RWE may also be

used to supplement the total evidence required for such clearances or approvals. Other

applications of RWE in premarket decision-making may also be possible, particularly as RWD

systems and analysis methodology advance.

Additionally, aggregation of RWD (e.g., in medical device registries) may prove useful as a

postmarket control suitable for providing ongoing device safety surveillance and additional

evidence for effectiveness. FDA has long applied postmarket controls as a way to reduce

premarket data collection, while still ensuring that the statutory standard of reasonable assurance

of safety and effectiveness is met.

FDA believes that applying postmarket controls to reduce

premarket data collection, when appropriate, can help improve patient access to safe and

effective medical devices.

In some cases, a “traditional” clinical trial may be impractical or excessively challenging to

conduct. Ethical issues regarding treatment assignment, and other similar challenges, may

present themselves when developing and attempting to execute a high quality clinical trial.

Analyses of RWD, using appropriate methods, may in some cases provide similar information

with comparable or even superior characteristics to information collected and analyzed through a

traditional clinical trial. For example, RWD collected using a randomized exposure assignment

within a registry can provide a sufficient number of patients for powered subgroup analyses,

which could be used to expand the device’s indications for use. However, not all RWD are

collected and maintained in a way that provides sufficient reliability. As such, the use of RWE

for specific regulatory purposes will be evaluated based on criteria that assess their overall

“Valid scientific evidence is evidence from well-controlled investigations, partially controlled studies, studies and

objective trials without matched controls, well-documented case histories conducted by qualified experts, and

reports of significant human experience with a marketed device, from which it can fairly and responsibly be

concluded by qualified experts that there is reasonable assurance of the safety and effectiveness of a device under its

conditions of use. The evidence required may vary according to the characteristics of the device, its conditions of

use, the existence and adequacy of warnings and other restrictions, and the extent of experience with its use.” [21

CFR 860.7(c)(2)]

(http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?FR=860.7)

The Least Burdensome Provisions of the FDA modernization Act of 1997: Concept and Principles,

(http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm085999.p

df)

Balancing Premarket and Postmarket Data Collection for Devices Subject to Premarket Approval,

(http://www.fda.gov/ucm/groups/fdagov-public/@fdagov-meddev-gen/documents/document/ucm393994.pdf).

Contains Nonbinding Recommendations

relevance and reliability. If a sponsor is considering using RWE to satisfy a particular FDA

regulatory requirement, the sponsor should contact FDA through the pre-submission process.

III. Real-World Evidence

RWE can exist across a wide spectrum, ranging from observational studies within an existing

dataset to studies that incorporate planned interventions with or without randomization at the

point of care.

Because of the rapidly advancing methodology for generating and interpreting

RWD, this guidance will not elaborate on the methodological approaches that can be used.

However, when reviewing the use of RWE to support a regulatory decision, FDA will rely on

scientifically robust methods and approaches to determine whether submitted RWE is of

sufficient quality to support a particular regulatory decision.

Clinical trials are designed to control variability through detailed eligibility criteria and carefully

designed clinical protocols performed by specialized research personnel. They require intensive

monitoring and data auditing to demonstrate that use of a device produces the expected results.

Although useful in establishing a baseline for device performance, clinical trials may be narrow

in scope due to practical challenges. In contrast, studies leveraging RWD can potentially provide

information on a wider patient population, thus providing information that cannot be obtained

through a traditional clinical trial alone. An existing RWD source, however, may have some

inherent bias that could limit its value for drawing causal inferences between medical device

exposures and outcomes. Therefore, to mitigate potential bias, careful study design is needed,

and a study protocol and analysis plan should be created prior to accessing, retrieving, and

analyzing RWD, regardless of whether the RWD are already collected (retrospective) or if they

are to be collected in the future (prospective design). Protocols and analysis plans for RWD

should address the same elements that a traditional clinical trial protocol and statistical analysis

plan would cover. FDA recommends use of the pre-submission process when considering the

development of a study using RWD in a regulatory submission.

When considering a prospective study design, one should consider whether RWD collection

instruments and analysis infrastructure are sufficient to serve as the mechanism for conducting

the study, and if they are not, whether it is possible to modify them for such a purpose.

Ultimately, if the sources of bias can be mitigated, RWD collected using a prospective study

design may be used to generate or contribute to the totality of evidence needed to assess medical

device performance.

Because of its nature, the quality of RWE can vary greatly across different data types and

sources. Likewise, there are many types of FDA regulatory decisions spanning the Total Product

Life Cycle (TPLC) that necessitate different levels of evidence. This guidance does not change

Requests for Feedback on Medical Device Submissions: The Pre-Submission Program and Meetings with Food

and Drug Administration Staff

(http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM311176.

pdf)

Sherman RE, Steven A, Dal Pan GJ, et.al.; Real-World Evidence – What Is It and What Can it Tell Us?; NEJM,

Dec 2016; 375; 23

Contains Nonbinding Recommendations

FDA’s evidentiary standards for regulatory decision-making, and in each context we will

evaluate whether the available RWE is of sufficient quality to address the specific regulatory

decision being considered. FDA believes that the increased use of electronic data systems in the

healthcare setting has the potential to generate substantial amounts of RWD. Because these

systems can vary greatly in terms of quality, not all RWD can by itself generate sufficient

evidence to support an FDA regulatory decision. Nevertheless, these RWD may still provide a

valuable contribution to the totality of evidence considered for the decision. Furthermore, in

order to use RWD, the device must be sufficiently identified with the level of detail necessary to

address the regulatory question. For example, to evaluate a particular device, the Unique Device

Identifier (UDI) or serial/model number may be necessary to identify the device within a RWD

source that contains data on many similar devices.

When RWE is intended to be used for purposes of evaluating a regulatory issue, it is important

that the RWD not only follow the criteria described in section V, but are also presented to FDA

according to recognized data standards (i.e. in standardized file formats and data structures,

utilizing standardized variables and definitions, etc.) when applicable. This includes discussions

of the methodology used to analyze RWD and assess clinically relevant differences as well as

statistical significance.

IV. Regulatory Context in Which RWE May be Used

A. General considerations for the use of RWE

FDA will consider the use of RWE to support regulatory decision-making for medical devices

when it concludes that the RWD used to generate the RWE are of sufficient quality to inform or

support a particular regulatory decision. The threshold for sufficient quality will depend on the

specific regulatory use of the evidence. For example, a specific registry might be leveraged for

postmarket surveillance, but not be adequate to support a premarket determination of reasonable

assurance of safety and effectiveness or substantial equivalence.

The collection or aggregation of RWD sources outside of medical records is usually performed

for specific pre-determined non-regulatory purposes. For example, medical administrative claims

data are typically kept for purposes of billing/payment for medical care. Disease-specific RWD

sources sponsored by patient advocacy organizations may be useful for tracking progression or

outcomes of specific rare or poorly understood diseases. Treatment-specific RWD sources

coordinated by one or more professional societies may have several purposes, including

assessment and tracking of overall outcomes, providing data for quality assessment (QA),

informing performance improvement (PI) initiatives, or providing risk prediction and

benchmarking data for specific therapies. Therefore, the use of RWD for specific regulatory

decisions necessitates an understanding of the strengths and limitations of the RWD, and how

these qualities impact the relevance and reliability factors described below.

RWD may potentially be used as some or all of the evidence necessary for understanding

medical device performance at different points in the TPLC. Some purposes for which RWD

may potentially be used include the following:

Contains Nonbinding Recommendations

· for generating hypotheses to be tested in a prospective clinical study;

· as a historical control, a prior in a Bayesian trial

, or as one source of data in a

hierarchical model or a hybrid data synthesis;

· as a concurrent control group or as a mechanism for collecting data related to a clinical

study to support device approval or clearance in a setting where a registry or some other

systematic data collection mechanism exists;

· as evidence to identify, demonstrate, or support the clinical validity of a biomarker;

· as evidence to support approval or granting of an Humanitarian Device Exemption,

Premarket Approval Application (PMA), or De Novo request;

· as support for a petition for reclassification of a medical device under section 513(e) or

(f)(3) of the FD&C Act;

· as evidence for expanding the labeling of a device to include additional indications for

use or to update the labeling to include new information on safety and effectiveness

· for public health surveillance efforts. Through ongoing surveillance, signals are at times

identified that suggest there may be a safety issue with a medical device. RWE may be

used to refine these signals for purposes of informing appropriate corrective actions and

communication;

· to conduct post-approval studies that are imposed as a condition of device approval or to

potentially preclude the need for postmarket surveillance studies ordered under section

522 of the FD&C Act;

Guidance for Industry and FDA Staff: Guidance for the Use of Bayesian Statistics in Medical Device Clinical

Trials,

(https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071121.

pdf)

What We Mean When We Talk About EvGen Part II: Building Out a National System for Evidence Generation,

(http://blogs.fda.gov/fdavoice/index.php/2016/04/what-we-mean-when-we-talk-about-evgen-part-i-laying-the-

foundation-for-a-national-system-for-evidence-generation/)

Guidance for Industry: General/Specific Intended Use

(http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm073945.p

df).

Public Notification of Emerging Postmarket Medical Device Signals (“Emerging Signals”) Guidance for Industry

and Food and Drug Administration Staff

(http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm479248.pdf).

Strengthening Patient Care: Building an Effective National Medical Device Surveillance System

(http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHRe

ports/UCM435112.pdf).

Contains Nonbinding Recommendations

· in certain circumstances, for use in generating summary reports of Medical Device

Reports (MDRs); and

· to provide postmarket data in lieu of some premarket data.

B. Application of Investigational Device Exemption (IDE)

Requirements in 21 CFR 812 to the Collection of RWD

An approved IDE permits a device to be shipped lawfully for the purpose of conducting

investigations of the device without complying with certain other requirements of the FD&C Act

that would apply to devices in commercial distribution. The purpose of this, per 21 CFR 812.1,

“is to encourage, to the extent consistent with the protection of public health and safety and with

ethical standards, the discovery and development of useful devices intended for human use, and

to that end to maintain optimum freedom for scientific investigators in their pursuit of this

purpose.” As explained in Part 812, the IDE regulations apply to all clinical investigations of

devices to determine safety and effectiveness, with certain limited exceptions. In many cases, an

approved IDE is required before initiating a clinical investigation. An investigation is defined as

“a clinical investigation or research involving one or more subjects to determine the safety or

effectiveness of a device.

”

Whether the collection of RWD for a legally-marketed device requires an IDE depends on the

particular facts of the situation. Specifically, if the device is being used in the normal course of

medical practice, an IDE would likely not be required. Because FDA does not regulate the

practice of medicine, this could include administration of a legally marketed device for uncleared

or unapproved uses as long as the device is being administered under the authority of a

healthcare practitioner within a legitimate practitioner-patient relationship.

However, if data

are being gathered to determine the safety and effectiveness of the device, and the process for

gathering the data would influence treatment decisions, such administration would likely not be

within the normal course of medical practice, and an IDE may be required. For example, a

registry designed to determine the safety and effectiveness of an approved device for a new

intended use would likely be subject to IDE requirements if physicians are instructed to treat

specific patients or otherwise administer the device in a prescribed way for purposes of data

generation, or when certain follow-up activities are performed for the purpose of research.

Because the gathering of RWD is different from traditional investigations, we recommend that

you contact FDA if you have questions about whether an IDE is required.

As stated above, FDA does not regulate the practice of medicine, and recognizes that RWD may

be collected during the routine care of patients that provides information related to the actual use

of an approved or cleared device. Such observations may include RWD from a use of a medical

device that was not within the cleared or approved indications for use. If such RWD collection

See 21 CFR 812.3(h)

FDA will not limit or interfere with the authority of a health care practitioner to prescribe or administer any

legally marketed device to a patient for any condition or disease within a legitimate health care practitioner-patient

relationship. Section 1006 of the FD&C Act, 21 USC 396.

Contains Nonbinding Recommendations

does not impact how the device is administered, and the administration is within the normal

course of medical care, an IDE would likely not be required. For example, retrospective

analyses of existing RWD involving the use of a device that was not within the cleared or

approved indications for use would generally not be subject to IDE regulations. In such cases,

treatment decisions were made in the best interest of patients according to their clinician’s

medical judgment at that time. However, if the plan to conduct such analyses impacts patient

care, then the study may be subject to IDE requirements.

Should a sponsor or Institutional Review Board (IRB) be unclear regarding the applicability of

the IDE regulations to a particular RWD collection activity or use, the sponsor or IRB should

contact FDA. If an IDE is determined to be required, FDA will work with the IDE sponsor to

develop the least burdensome approach to facilitate the efficient generation of high-quality

RWE. Note that regardless of the applicability of 21 CFR 812, FDA regulations at 21 CFR 56

(IRB review), 21 CFR 50 (Informed Consent) and 21 CFR 54 (Financial Disclosure) may apply

to RWE generation activities, as may other federal, state, and local laws regarding human subject

protections.

V. Characteristics of RWD

FDA does not endorse one type of RWD over another. Sponsors should select appropriate RWD

sources based on their suitability to address specific regulatory questions. Whether the RWD

resides within paper or electronic medical records, is collected by administrative databases, is

abstracted, aggregated and stored in disease- or treatment-specific databases (i.e., registries), or

collected and aggregated through other means, accuracy when compared to verifiable source

documentation is essential. Verifiable source documentation for RWD elements includes, but is

not limited to: paper or electronic inpatient and outpatient medical records and case histories,

diagnostic laboratory and imaging data, patient preference information, patient-reported outcome

measures, UDI and other device identifiers, and performance data that exist within the device

such as self-diagnostics, error codes, and patient diagnoses/treatments delivered. Requirements

and needs for individual source data verification will vary with specific regulatory questions, and

will contribute to the overall understanding of data quality for that source.

In order to determine the suitability of RWD for regulatory decision-making, FDA will assess

the relevance and reliability of the source and its specific elements. This assessment will be used

to determine whether the RWD source(s) and the proposed analysis can generate evidence that is

sufficiently robust to be used for a given regulatory purpose. Whether evidence is sufficiently

relevant and reliable for use will, in part, depend on the level of quality necessary to make a

particular regulatory decision. FDA will evaluate the same factors to assess RWD across all data

sources and regulatory decisions. In cases where RWE is derived from multiple RWD sources,

each RWD source will be evaluated individually and together in the aggregate to determine the

relevance and reliability of the RWD. When developing a new RWD source, consultation with

FDA and other stakeholders is recommended to ensure that relevance and reliability are

addressed in the initial design.

Contains Nonbinding Recommendations

A. Relevance

The relevance of RWD, RWD sources, and resultant analysis is assessed by evaluating several

factors as outlined below. These factors can help determine if the data adequately address the

applicable regulatory question or requirement, in part or in whole. Questions about the

applicability of RWD to a specific case should be discussed with FDA through the pre-

submission process.

Relevance of RWD for regulatory decision-making can be assessed prior

to a regulatory submission, such as via the pre-submission process, or during the regulatory

review process. The overall assessment of relevance should determine whether the existing

RWD source is adequate for evaluating the performance of a device in the identified regulatory

context (as a sole source or partial source of evidence).

Since RWD sources are usually developed for non-regulatory purposes (e.g., to document care in

the case of EHRs or to submit insurance claims for reimbursement in administrative and claims

data), FDA will assess whether the individual data elements contained within an existing RWD

source are sufficient to be used for a regulatory purpose. The data should be accurate, as

complete as possible, and have an appropriate scope to address the question at hand (i.e., data

adequacy). The need for review or adjudication of specific outcomes of interest (e.g. stroke or

major bleeding) at the patient level may also be assessed. For analysis and interpretation of

RWD, it is important to have a pre-defined common set of data elements, a common definitional

framework (i.e., data dictionary), and pre-specified time intervals for data element collection and

outcome analyses. In assessing the relevance of RWD, FDA will also consider, if warranted, the

ability to supplement the available RWD through linkage with other data sources to provide

additional or confirmatory data, e.g., with EHR and/or administrative claims data.

Important relevance factors that FDA will assess to determine if the RWD are suitable for

regulatory use include, but are not limited to, whether:

· the RWD contain sufficient detail to capture the use of the device, exposures, and the

outcomes of interest in the appropriate population (i.e. the data apply to the question at

hand);

· the data elements available for analysis are capable of addressing the specified question

when valid and appropriate analytical methods are applied (i.e. the data are amenable to

sound clinical and statistical analysis); and

· the RWD and RWE they provide are interpretable using informed clinical/scientific

judgment. Important considerations for the assessment of this factor include:

Requests for Feedback on Medical Device Submissions: The Pre-Submission Program and Meetings with Food

and Drug Administration Staff: Guidance for Industry and Food and Drug Administration Staff

(http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM311176.

pdf)

Contains Nonbinding Recommendations

o whether the use of the device in a real-world population is representative as

captured within the data source, and is generalizable to the relevant population

being evaluated;

o whether the RWD source is used regionally, nationally and/or internationally;

o the overall percentage of patient exposure to the device that are captured in the

RWD source;

o the validation protocols and resultant data that are used to evaluate how well the

RWD source reflects the patient population’s experience;

o whether the RWD study design, study protocol, and/or analysis plan is

appropriate to address the regulatory question and capable of being accomplished

in a sufficiently timely manner;

o whether the RWD contains elements to capture specific device identification

information (e.g., unique device identifier);

o whether the RWD adequately captures patient medical history and preexisting

conditions, as well as follow-up information needed evaluate the question being

addressed (e.g., whether administrative claims data have adequate continuity of

coverage);

o whether sufficient data elements are collected to adjust for confounding factors

that may impact the exposure or outcomes of interest;

o whether any linkages performed are scientifically appropriate and account for

differences in coding and reporting across sources;

o the RWD source reporting schedule, including time interval between database

close and release, and length of reporting periods;

o the prior documented (e.g., peer reviewed publications or practice guidelines) use

of the RWD source for determining outcomes-based quality assessments,

validated predictive risk modeling, signal detection, performance improvement,

benchmarking, and other clinically-meaningful uses;

o whether the data elements collected are sufficient for assessing outcomes

(including adjudication, if necessary); and

o whether supplemental data sources are available and sufficient to provide any

missing information or evidence required for an informed decision.

Contains Nonbinding Recommendations

B. Reliability

The reliability of RWD, RWD sources, and resultant analyses is assessed by evaluating several

factors as outlined below. The primary factors FDA considers for assessing the reliability of

RWD include how the data were collected (data accrual), and whether the people and processes

in place during data collection and analysis provide adequate assurance that errors are minimized

and that data quality and integrity are sufficient (data assurance). Additionally, the RWD

analysis protocol should be prospectively defined as described in section III above. FDA will

consider existing data accrual, data assurance, and analysis approaches in its assessment of the

fitness of a given RWD source and its data.

(1) Data accrual

To ensure the reliability of RWD, the RWD source should have an operational manual or other

documentation that pre-specifies the data elements to be collected, data element definitions (i.e.,

data dictionary to provide a common definitional framework), methods for data aggregation and

documentation (e.g., common case report form, abstraction from verifiable sources), and the

relevant time windows for data element collection (i.e., common temporal framework). Some

RWD sources such as EHRs or claims data may not fulfill all of these characteristics, but may

still demonstrate sufficient reliability to support regulatory decision-making. Factors FDA will

consider in making this determination include (but are not limited to):

o the preparedness of individual sites for complete and accurate collection of RWD

(e.g., whether there are defined processes, site training and support, and qualified

personnel);

o whether a common data capture form was used;

o whether a common definitional framework (i.e., data dictionary) was used;

o adherence to a common temporal framework for collection of key data points;

o the timing of establishing the study plan, protocol, and/or analysis plan relative to

collection or retrieval of the RWD;

o the sources and technical methods used for data element capture (e.g., chart

abstraction, point of care entry, EHR integration, UDI capture, data records from

the device, and linkage to claims data);

o whether patient selection and enrollment criteria minimize bias and ensure a

representative real-world population (e.g., all-comer’s design, consecutive patient

enrollment);

o the timeliness of data entry, transmission, and availability; and

Contains Nonbinding Recommendations

o whether necessary and adequate patient protections were in place (e.g., methods

to protect patient privacy, and need for informed consent as determined by the

reviewing IRB and in compliance with FDA regulations).

(2) Data assurance - Quality Control

Data quality control is essential for providing confidence in the reliability of RWD and RWE

sources. RWD quality can generally be improved by following published recommendations

concerning registries, such as those by the Agency for Health Care Quality, Patient-Centered

Outcomes Research Institute,

the National Medical Device Registry Task Force,

and the

Regulators Forum (IMDRF) Registry Working Group.

However, certain sources of RWD,

such as some administrative and healthcare claims databases or EHRs, may not have established

data quality control processes and may not be capable of fully implementing or following the

above recommendations. When considering a source of RWD for regulatory purposes, it is

important to consider any methods and systems used to help ensure sufficient data quality.

Potential RWD sources should be evaluated in accordance with the data QA plan and procedures

developed for the data source itself. Since evaluation of RWD sources may not always permit

specific line item source verification, important factors for consideration include:

o the quality of data element population (e.g., whether abstracted from a verifiable

source to assess transcription errors or automatically populated through a data

extraction algorithm);

o adherence to source verification procedures and data collection and recording

procedures for completeness and consistency;

o completeness (i.e., minimized missing or out of range values) of data necessary

for specified analyses, including adjustment for confounding factors;

o data consistency across sites and over time;

AHRQ Registries for Evaluating Patient Outcomes: A User’s Guide 3

Edition, Volume 1 Chapter 11, Data

Collection and Quality Assurance and Volume 2 Chapter 25, Assessing Quality.

(http://www.effectivehealthcare.ahrq.gov/search-for-guides-reviews-and-

reports/?pageaction=displayproduct&mp=1&productID=1897)

A Report from the Medical Device Registry Task Force & the Medical Devices Epidemiology Network.

Recommendations for a National Medical Device Evaluation System. Strategically Coordinated Registry Networks

to Bridge Clinical Care and Research, August 2015. Available at:

http://www.fda.gov/downloads/aboutfda/centersoffices/officeofmedicalproductsandtobacco/cdrh/cdrhreports/ucm45

9368.pdf. Accessed August 6, 2016.

IMDRF Registry Essential Principles. Available at:

http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-essential-principles-151124.pdf. Accessed August

3, 2016

PCORI Conduct of Registry Studies

http://www.pcori.org/sites/default/files/Standards-in-the-Conduct-of-Registry-Studies-for-Patient-Centered-

Outcomes-Research1.pdf

Contains Nonbinding Recommendations

o evaluation of on-going training programs for data collection and use of data

dictionaries at participating sites;

o evaluation of site and data monitoring practices; and

o the use of data quality audit programs.

The use of routine medical care data for additional analyses often relies on data cleaning and

cross-referencing. These techniques can confirm the data’s internal consistency and identify

missing values, but cannot fully determine data accuracy and authenticity. In traditional clinical

research, an audit that compares source documents to entered data is often an essential step to

verify the accuracy and completeness of the data. Study monitoring through various methods,

such as remote monitoring, also plays an important role.

These types of data verification

activities are equally important for RWD that is intended to be used for regulatory purposes.

Regardless of the original purpose for collecting the RWD, procedures for data collection and

quality assurance should be put into place during the data source design and development stages

(when applicable) to optimize the reliability, quality and usefulness of the data. The data

collection procedures used for data platforms such as registries should be clearly defined and

described in a detailed data management standard operating procedures (SOP) manual. When

implementing a retrospective study using RWD, standardizing procedures to ensure the use of

uniform and systematic methods for collecting and cleaning data are vital to ensuring data

quality. For example, a “quality system approach” with a risk-based quality assurance and

monitoring plan is a practical strategy for data platforms such as registries that can be

challenging to audit. The RWD source organization or entity considering using the RWD for

regulatory purposes should retain records regarding the assessment of adherence to the RWD

source’s established data quality assurance and quality control policies and procedures.

VI. Examples Where RWE is Used

The following examples are generalized from actual uses of RWE in support of regulatory

decision making. These examples do not represent a comprehensive list of all potential uses or

sources of RWD, but do describe some situations where RWE might be used to support

regulatory decision-making.

A. Expanded Indications for Use

The National Cardiovascular Data Registry (NCDR) was created in 1997 by the American

College of Cardiology (ACC) as “an exploration into strategies for improving cardiovascular

care through the use and application of clinical data.” These registries are designed to help

participants measure, benchmark, and improve cardiovascular care. In particular, the Registry

for diagnostic cardiac CATHeterization and Percutaneous Coronary Intervention (Cath-PCI

Oversight of Clinical Investigations: A Risk-Based Approach to Monitoring (PCORI Conduct of Registry Studies

(http://www.pcori.org/sites/default/files/Standards-in-the-Conduct-of-Registry-Studies-for-Patient-Centered-

Outcomes-Research1.pdf)

Contains Nonbinding Recommendations

Registry) “assesses the characteristics, treatments and outcomes of cardiac disease patients who

receive diagnostic catheterization and/or percutaneous coronary intervention (PCI) procedures,

measuring adherence to ACC/AHA clinical practice guideline recommendations, procedure

performance standards and appropriate use criteria for coronary revascularization.” As a registry

collecting RWD on patients treated in routine clinical practice with an approved or cleared

device, an IDE is not required even though a substantial volume of RWD is generated for uses

outside of the device’s cleared or approved indications for use. RWD from this registry could be

used to identify opportunities to expand the labeling of devices captured in the registry. Should a

manufacturer wish to expand indications, this type of RWD might provide sufficient evidentiary

support, depending on the specific devices, indications, and analyses.

Another example is a Class III device that, due to technological advances in its design, has seen

an expansion of clinically acceptable use that is outside of the device’s approved indications for

use. There is little published data to support a reasonable assurance of safety and effectiveness

of the new use. To address the lack of data to support new indications for use, relevant medical

societies have established a national registry to collect safety and effectiveness information for

all patients implanted with this specific device at participating institutions. This registry also

uses a validated matching algorithm to link registry records with administrative healthcare claims

as a supplemental dataset for capturing long-term outcomes. A study using the registry data

collection and analysis infrastructure was initiated with an approved IDE application since the

study focused on a use of this device that was not within the approved indications for use and

imposed changes to the collection regiment of specific follow-up data that might not otherwise

have occurred as part of routine medical care. FDA is hopeful that the RWD may be of

sufficient quality to address critical safety questions and to potentially support labeling changes

or other regulatory decisions for this device.

B. Postmarket Surveillance Studies (Section 522)

FDA has issued a series of postmarket surveillance orders, related to investigating patient safety

issues for a type of class II device, under the authority of section 522 of the FD&C Act. These

522 orders covered multiple devices from different manufacturers that are similar in intended

use, design, and other characteristics, such that the surveillance questions were identical. To

comply with the orders, many manufacturers decided to collaborate with a clinical professional

society in this field and with FDA to develop a patient registry that would collect the needed data

to address the public health questions. An IDE was not required because the 522 order was

focused on device uses consistent with the labeling. The resultant registry was designed to

collect RWD on all patients with the relevant condition, including those treated with the devices

of interest, with other devices, and through medical management. Manufacturers are able to

share the comparator group consisting of treatments that do not use the devices of interest. In

addition, because the registry was designed at the outset to produce regulatory-quality RWD in

addition to meeting research and quality improvement purposes, appropriate data quality checks

and electronic controls were a part of the initial design and implementation. Since this registry

development process took a substantial amount of time, FDA was willing to grant extensions to

manufacturers to respond to the 522 orders as long as progress was being made. The registry

was also designed to allow for its use (with additional protocols and other traditional study

Contains Nonbinding Recommendations

operational elements) in conducting premarket studies that could support future premarket

submissions.

C. Post-Approval Device Surveillance as Condition of Approval

Permanent implants are typically designed to serve patients for a time period that is much longer

than what can reasonably be captured in a premarket clinical trial. For example, a trial that

follows patients for two years after implantation would not produce data for the designed life

span of 7 to10 years for that implanted device. Traditionally, FDA would require extended

follow-up of the premarket patient cohort and an additional new-enrollment study designed to

capture hundreds to thousands of patients with follow-up for the life of the implanted device.

Some clinical professional societies have developed registries that collect RWD on patients

receiving these devices. FDA has worked with manufacturers and professional societies to

evaluate the registries and has found that they can be reliable for certain health outcomes of

interest. Should sufficient RWD exist that are capable of addressing the questions for which a

Post-Approval Study (PAS) requirement may be issued, FDA may instead issue a condition of

approval that requires collection/reporting of these RWD on the device.

For example, a new breakthrough Class III medical device was approved based on prospective,

randomized, and controlled clinical trial data. Early in the PMA review process, the

manufacturer began to consider postmarket commitments, and began discussions with FDA and

other stakeholders. A registry was launched that generated RWD that could meet FDA’s data

requirements, as well as others. Because the new registry was constructed early enough to collect

information about all patients receiving this device upon approval, FDA could provide an earlier

device approval conditioned on further robust RWD collection and reporting in the postmarket

setting. This registry has since been used to a) collect surveillance data on subsequent devices

with similar designs and indications, b) collect and retrospectively analyze RWD on all uses of

the devices to support new expanded indications for use, and c) support embedded prospective

clinical investigations under IDE for new devices and new generations of approved devices. No

IDE is necessary for the general data collection activities of the registry, as it collects RWD on

all uses of otherwise approved medical devices and it does not influence the treatment decisions

and/or follow-up care that patients receive. The retrospective analysis of RWD for uses that are

outside the approved indications for use did not require an IDE because treatment decisions were

not influenced by the expectation of conducting the future analysis, but was still reviewed by an

IRB for human subject protection issues. However, prospective enrollment in a clinical

investigation using the registry infrastructure to study a new, unapproved significant risk device

would require an IDE. Similarly, a prospective, non-observational clinical investigation of a new

indication for an approved device may require an IDE, depending on the risk determination.

D. Control Group

A manufacturer approached FDA during the development of a next generation medical device

that had substantial technological changes from previous iterations of that specific device and

other similar devices from other manufacturers. FDA determined that clinical evidence was

needed to support an approval decision for this device modification. A registry exists that

captures RWD on all uses of medical devices with a similar intended use. The manufacturer

Contains Nonbinding Recommendations

designed a clinical study that compared the use of the new device to a non-randomized

concurrent control group derived from the registry. The existing registry was evaluated by FDA

and the manufacturer according to the factors cited in this guidance and was found to provide

sufficiently relevant and reliable RWD on the control population, such that the manufacturer did

not have to collect additional data from these patients or influence the course of their clinical

care in any way. The patients who received the investigational device were enrolled under an

approved IDE. However, the patients who contributed to the control group were not considered

part of the IDE because they were enrolled in a national registry that collected RWD on FDA

approved devices, and their treatment was not influenced by the existence of the study.

E. Supplementary Data

Periodically, FDA identifies an issue related to the safety of a marketed medical device that was

not detected in premarket trials. In the case where RWD has been systematically collected, FDA

has used these RWD, in combination with case reports, publications, adverse event reports,

engineering and nonclinical test data, and other sources of information available to FDA to gain

a better understanding of the severity of the issue, precipitating factors, affected populations, and

alternative therapies. The addition of RWD has proven extremely valuable as a means to

develop a course of action that best protects public health in these instances.

For example, a class III device was under review for a new indication. The manufacturer

provided data from a prospective clinical trial with limited follow-up information and inadequate

data from the control group, which made interpretation of results difficult. However, a pre-

existing registry was already collecting and reporting RWD on the control therapies. The

registry data were used to supplement and help interpret the clinical trial data, allowing FDA to

come to an appropriate regulatory decision without requiring additional clinical trial data.

Without the RWE, additional study subjects could have been exposed to a device with a

questionable risk-benefit balance. Coming to a final decision more quickly in this case protected

subjects’ health while also spurring development of new designs for the medical device.

F. Objective Performance Criteria and Performance Goals

An Objective Performance Criterion (OPC) refers to a numerical target value derived from

historical data from clinical studies and/or registries and may be used in a dichotomous

(pass/fail) manner for the review and comparison of safety or effectiveness endpoints.

An OPC

is usually developed when device technology has sufficiently matured and can be based on

publicly available information or on information pooled from all available studies on a particular

kind of device. Similar to OPC, a performance goal (PG) refers to a numerical value that is

considered sufficient for use in the evaluation of an investigational device regarding a safety

and/or effectiveness endpoint. But, generally, the device technology is not as well-developed or

Design Considerations for Pivotal Clinical Investigations for Medical Devices - Guidance for Industry, Clinical

Investigators, Institutional Review Boards and Food and Drug Administration Staff for more information on OPCs

and PGs.

http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM373766.

pdf).

Contains Nonbinding Recommendations

mature for use of a PG as for an OPC, and the data used to generate a PG are not considered as

robust as those used to develop an OPC. A PG might be considered for patient populations that

are difficult to study or if there is no clinical equipoise for any control. From a sufficiently

relevant and reliable RWD source, a PG can be constructed using appropriate statistical methods,

such as a subject-level meta-analysis. As technology evolves over time, an OPC or PG could be

updated using RWD.

VII. Glossary

The following definitions are supplied to provide the reader with an understanding of the specific

terms used in this guidance. These definitions should not be construed to be new interpretations

or clarification of the use of similar words or phrases in the FD&C Act, related code or

regulation, other federal, state, or local laws, or other guidance documents.

· Bias—“Bias is any systematic error in the design, conduct, analysis, interpretation,

publication, or review of a study and its data that results in a mistaken estimate of a

treatment’s effect on disease. This systematic error results from flaws in the method of

selecting study participants, in the procedures for gathering data, and in the decision of

how and whether to publish the results. These flaws can lead to observed study results

that tend to be different from the “true” results. Bias can be minimized by ensuring that

the study design is appropriate for addressing the study hypotheses and establishing and

carefully monitoring procedures of data collection that are valid and reliable.”

· Confounding—“A situation in which a non-causal association between a given exposure

or treatment and an outcome is observed as a result of the influence of a third variable

designated as a confounder. The confounding variable needs to be related to both the

treatment and the outcome under study. Confounding is distinct from bias because this

association, while not causal, is real.”

· Electronic Health Record (EHR)—“An electronic record of health-related information

on an individual that conforms to nationally recognized/utilized interoperability standards

and that can be created, managed, and consulted by authorized clinicians and staff across

more than one health care organization.”

· Electronic Medical Record (EMR)—“An electronic record of health-related

information on an individual that can be created, gathered, managed, and consulted by

authorized clinicians and staff within one health care organization.”

JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ

Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000

L Gordis. Epidemiology. Philadelphia: WB Saunders, Co., 1996

The National Alliance for Health Information Technology Report to the Office of the National Coordinator for

Health Information Technology on Defining Key Health Information Technology Terms April 28, 2008.

(http://www.himss.org/national-alliance-health-information-technology-report-office-national-coordinator-health).

The National Alliance for Health Information Technology Report to the Office of the National Coordinator for

Health Information Technology on Defining Key Health Information Technology Terms April 28, 2008.

(http://www.himss.org/national-alliance-health-information-technology-report-office-national-coordinator-health)

Contains Nonbinding Recommendations

· Interventional Study—“A clinical study in which participants are assigned to receive

one or more interventions (or no intervention) so that researchers can evaluate the effects

of the interventions on biomedical or health-related outcomes. The assignments are

determined by the study protocol. Participants may receive diagnostic, therapeutic, or

other types of interventions.”

· Large Simple Trial – “A large simple trial (LST) is a type of randomized clinical trial

(RCT) ideally suited to answer many important clinical questions and because it typically

answers only one or two questions in a broader patient population, is generally more

efficient and less expensive than other large RCTs. LSTs have a large sample size and

statistical power to detect clinically relevant treatment effects, providing unambiguous

results and minimizing the effects of random errors.”

· Medical Administrative Claims Data—“Claims data arise from a person’s use of the

health care system [and reimbursement of health care providers for that care].”

· Medical Device Registry – “Organized system that continuously and consistently

collects relevant data in conjunction with routine clinical care, evaluates meaningful

outcomes and comprehensively covers the population defined by exposure to particular

device(s) at a reasonably generalizable scale (e.g. international, national, regional, and

health system) with a primary aim to improve the quality of patient care.”

· Medically Recognized Standards of Care—“Medically recognized standards of care

are treatments or procedures that have been accepted by medical experts as appropriate

treatments or procedures for a given type of disease or condition and are commonly used

by health care professionals. The medical recognition of standards of care is typically

represented by publication in a peer-reviewed journal or some form of recognition by a

professional medical society. The evidentiary bases for these recognized standards of

care vary.”

· Observational Study— “A study that does not involve any intervention (experimental or

otherwise) on the part of the investigator; e.g., a population study in which changes in

health status are studied in relation to changes in other characteristics. Most analytical

epidemiological designs (notably, case-control and cohort studies) are properly called

observational because investigators observe without intervening other than to record,

classify, count, and analyze results.”

https://www.clinicaltrials.gov/ct2/about-studies/glossary

Project: Large Simple Trials; Clinical Trials Transformation Initiative (https://www.ctti-

clinicaltrials.org/projects/large-simple-trials)

Strom, Brian. Pharmacoepidemiology. Chichester, England: John Wiley and Sons, 2005.

IMDRF Registry Essential Principles. Available at:

http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-essential-principles-151124.pdf. Accessed August

3, 2016

Ethical Review and Oversight Issues in Research Involving Standard of Care Interventions: Workshop in Brief

2015, Institute of Medicine (https://www.nap.edu/read/21668/chapter/1)

A Dictionary of Epidemiology (6 ed.) Oxford Reference http://irea.ir/files/site1/pages/dictionary.pdf

Contains Nonbinding Recommendations

· Postmarket surveillance— “Postmarket surveillance is the active, systematic,

scientifically valid collection, analysis and interpretation of data or other information

about a marketed device.”

· Pragmatic clinical trial (PCT) - A clinical trial “designed for the primary purpose of

informing decision-makers regarding the comparative balance of benefits, burdens and

risks of a biomedical or behavioral health intervention at the individual or population

level.”

· Prospective Study—“A prospective study design (also called a concurrent cohort study)

defines the original population of interest at the start of the study and collects

exposure/treatment and outcome data from that time point forward. The start of the study

is defined as the time the research protocol for the specific study question was

initiated.”

· Real-World Data (RWD) are data relating to patient health status and/or the delivery of

health care routinely collected from a variety of sources.

· Real-World Evidence (RWE) is the clinical evidence regarding the usage and potential

benefits or risks of a medical product derived from analysis of RWD.

· Registry—“An organized system that uses observational study methods to collect

uniform data (clinical and other) to evaluate specified outcomes for a population defined

by a particular disease, condition, or exposure, and that serves one or more predetermined

scientific, clinical or policy purposes.”

· Retrospective Study—“A retrospective study design (also called a retrospective cohort

study, a historical cohort, or non-concurrent prospective study) defines the population

and determines the exposure/treatment from historical data (i.e., data generated prior to

the initiation of the study). The variables and outcomes of interest are determined at the

time the study is initiated. Some studies are a combination of concurrent and

retrospective cohort designs where the exposure/treatment is ascertained from existing

objective records (e.g., medical records, claims data), and follow up and measurement of

the outcome continues into the future.”

21 CFR 822.3

Adapted from Califf R and Sugarman J; Exploring the Ethical and Regulatory Issues in Pragmatic Clinical Trials;

Clin Trials, 2015 Oct;12(5):436-41. doi: 10.1177/1740774515598334. Epub 2015 Sep 15

JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ

Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000

Registries for Evaluating Patient Outcomes: A User's Guide

(https://www.effectivehealthcare.ahrq.gov/ehc/products/420/1897/registries-guide-3rd-edition-vol-1-140430.pdf)

JM Last. A dictionary of Epidemiology (3rd edition). New York: Oxford University Press, 1995) (M Szklo & FJ

Nieto. Epidemiology: Beyond the basics. Gaithersburg, MD: Aspen Publishers, Inc., 2000

Contains Nonbinding Recommendations

· Surveillance—“Surveillance is a continuous and systematic process of collection,

analysis, interpretation, and dissemination of descriptive information for monitoring

health problems.”

· Traditional clinical trial—Traditional clinical trials are typically conducted in

specialized research settings with specific populations. They often utilize measures

designed to control variability and ensure data quality, such as detailed eligibility criteria,

detailed case report forms that exist apart from ordinary medical records, and intensive

monitoring and auditing designed to ensure precise adherence to study procedures and

rigorous precision in data collection. They typically also include substantial efforts to

ensure compliance with treatments and to avoid concomitant treatments that might

influence the randomized treatment effect.

JW Buehler. Surveillance (Ch. 22) pages 435-458 in KJ Rothman & S Greenland (editors) Modern Epidemiology

2nd edition. Philadelphia: Lippincott-Raven Publishers, 1998