Advances in the Use of Patient Reported Outcome Measures in
Electronic Health Records
Including Case Studies
November 7, 2013
In support of the PCORI National Workshop to Advance the Use of PRO measures in
Electronic Health Records
Atlanta, GA. November 19-20, 2013
Contact Information:
Albert W. Wu, MD, MPH
Center for Health Services and Outcomes Research
Johns Hopkins Bloomberg School of Public Health
624 North Broadway
Baltimore, Maryland 20105
410-955-6567
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
1
This technical report was written by
Albert W. Wu, MD, MPH
Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
Roxanne E. Jensen, PhD
Lombardi Comprehensive Cancer Center
Georgetown University, Washington, DC
Claudia Salzberg, MS
Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
Claire Snyder, PhD
Johns Hopkins University School of Medicine, Baltimore, MD
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
2
TABLE OF CONTENTS
Executive Summary……………………………………….…..……………………. 3
Introduction…………………………………………….……….……………………. 7
What is a Patient Reported Outcome? RO…………………………………………9
Taxonomy of PROs…………………………………….……………………………. 9
Model for use of PROs for care, quality and research………….……………...... 10
Electronic Health Records……………………………….…………………………. 14
Methods………………………………………………………………………………. 16
Case Studies……………………………..…………..……………………………… 19
Synthesizing Across the Case Studies …………………………………………… 41
Summary of Systems …………………………………………….……….………… 41
Consideration of Cases by System Features…………………………………….. 42
Major Themes…………………………………………………….………………….. 46
Moving Ahead: Standardization……………………………………………………. 50
Conceptual System Architecture…………………………………………………… 51
Unanswered Questions……………………………………………………………... 52
Key Barriers and Enabling Factors………………………………………………… 58
Conclusion……………………………………………………………………………. 64
References…………………………………………………………………………… 65
Figures…………………………………………….………………………………….. 78
Acknowledgments……………………………………………………………………. 81
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
3
EXECUTIVE SUMMARY
The goal of health care systems is to obtain optimal patient outcomes, decrease risk
and disease, and improve or maintain functioning for individual and populations.
Incorporating the patient perspective through patient reported outcome (PRO)
measures is a crucial element for clinical care, quality performance management and
clinical research. PROs are any report coming directly from patients regarding their
health condition and treatment, including symptoms, functional status and health-related
quality of life. Some PRO measures are generic and appropriate for use in a wide
range of conditions, while others focus on the specific symptoms and side effects of a
given disease, condition or treatment.
The use of PROs as outcome measures in research studies dates back to the 1980s.
Since then, PRO data collection has increasingly integrated into health care. There is
now a convergence in the evolution of PRO measurement, medical record keeping and
comparative effectiveness research into an increasingly electronic and patient-centered
space. Electronic health records (EHRs) began as an electronic version of the patient
record for hospitals and clinics, and have evolved to serve a broader purpose of giving
multiple stakeholders, including providers, managers and patients, access to a patient’s
medical information across different facilities. Systems have been developed recently
that link EHRs to the collection of PRO data. One advantage of this linkage is that data
collected for one purpose can potentially be used for multiple different tasks, including
clinical care, quality assessment and improvement, research, and public reporting. Pilot
studies, implementation efforts integrating PROs into EHRs and development of PRO
research methods have received major federal support from the National Institutes of
Health (NIH), Agency for Healthcare Research and Quality (AHRQ), Patient Centered
Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services
(CMS), Food and Drug Administration, and the Office of the National Coordinator for
Health Information Technology (ONC), as well as private and professional organizations.
PCORI has organized a National Workshop to Advance the Use of PRO Measures in
Electronic Health Records (EHRs), to be held on November 19-20, 2013 in Atlanta, GA.
This paper provides a landscape review of the current state of use of PRO measures in
EHRs, focusing on themes in the implementation and integrations of PROs within EHRs.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
4
We focused on PRO data collection systems that were linked to an EHR system. We
did not include efforts to measure patient satisfaction independent of patient health
status, electronic reviews of systems, family history, health behaviors or health care
utilization.
The review includes 11 case studies from the US that illustrate the range of what has
been done in diverse health care settings for clinical care, quality improvement and
research. To supplement our review, we interviewed the principle developers and users
of the systems. Many of the cases used the Epic Corporation EHR with its MyChart
tethered patient portal for PRO data collection. The systems were based at Dartmouth-
Hitchcock Medical Center, Cleveland Clinic, Group Health Cooperative, Cincinnati
Children’s Hospital, Kaiser Permanente Colorado (KPCO), Essentia Health/Minnesota
Community Measurement, University of Pittsburgh Medical Center (UPMC), Duke
University Medical Center, the University of California Los Angeles Medical Center
(UCLA) and the University of Michigan Medical Center, and the University of
Washington. The systems represent a broad range of functionality, patient populations
and applications. Efforts at three health care plans (KPCO, Group Health and Essentia
Health) illustrate applications of PRO collection at the plan level for clinical care,
population based screening and quality of care evaluation. Hospital based efforts at
Cincinnati Children’s Hospital, the Cleveland Clinic and Dartmouth show how the use of
PROs in specialty care can expand within a hospital. The review also includes
examples of clinic-based, disease specific PRO collection (UCLA/Michigan’s My GI
Health, University of Washington Center for AIDS Research Networks of Clinical
Systems (CNICS), Duke’s Patient Care Monitor for cancer, and UPMC for primary care),
some of which show how efforts may spread to specialists outside of the originating
institution. While clinical care efforts focus primarily on providing the physician
information to use during a patient visit, some systems also elicit information for follow-
up evaluations. Essentia, through Minnesota Community Measurement, reports their
scores as part of a statewide public reporting effort. A few organizations have
integrated their systems into clinical and comparative effectiveness research
(UCLA/Michigan, Cleveland Clinic, Cincinnati Children’s, Duke), while others, like
UPMC’s system was designed to focus exclusively on clinical utility.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
5
System features were examined across the 5 following categories: system design and
implementation, measure selection, administration and data collection, reporting and
interpretation, and analysis. While some of these system features are upstream from
the technical aspects of EHR integration, each element builds the foundation for the
focus, validity, interpretation and usefulness of the PRO data available in an EHR
system. Overall the largest variations across these features were seen between
systems that were designed using EHR-based PRO collection (e.g., Epic’s MyChart
feature) and outside collection that then sought to integrate information after collection
and reporting. Systems that designed and collected PROs independently from the EHR,
presented a much different approach allowing greater freedom in PRO content selection,
more flexibility in patient access and score use, and limited ability to integrate PRO data
with other clinical care markers. These trade-offs have implications at the person-,
provider- and national-level which can be guided at this early point through
standardization, developing a broad conceptual system architecture to encourage PRO
collection compatible with larger research and evaluation efforts.
In our analysis, four major themes emerged regarding the integration and use of PROs
in EHRs: (1) Necessity of System Customization, (2) Balancing Research and Practice
Goals, (3) Demonstrating Value and (4) EHR Integration and Limitations. These
themes were important considerations for all case studies leading to key decisions
ranging from design (e.g., PRO content and selection) to intended use (e.g., clinical
care, research or quality improvement). Systems did not necessarily make similar
choices. For example the University of Washington chose to scale-up PRO collection to
implement standardized PRO collection in a national research data network
infrastructure, while Kaiser Permanente Colorado’s effort has focused on patient
screening and quality improvement. However each of these systems has considered
these themes with respect to the feasibility to sustain and expanding PRO collection
efforts into other patient populations and/or clinical settings.
There are a number of knowledge gaps identified in this report. These gaps center on
the optimal system design features for PRO collection and integration. While these
span PRO selection, administration analysis and security they all center around two
main questions: the accuracy and accessibility of PRO data in an EHR. Ultimately these
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
6
current gaps point to the necessity of multidisciplinary teams and identifying “teachable
moments” to educate clinicians, staff, researchers, and patients on PRO use.
There are a number of remaining barriers to sustainable PRO integration. These fall
under three broad categories: patient, clinician, and system functionality. The most
common barriers were related “hidden” elements linked to electronic PRO collection that
are found even in the most developed systems. Regardless of the number of system
features and staff expertise, system awareness, response rate, clinician use, and
consistent system access (e.g., enough tablets available in clinic, Wi-Fi access) all rely
on engagement from patients, clinicians, and staff. Fortunately, most barriers identified
have been shown to be somewhat modifiable, with enablers that may be scalable.
To date, the perceived benefits of using PROs in clinical care have driven the
implementation of in-clinic PRO data collection and EHR integration. Recent efforts to
support PRO collection through patient portals offer a platform to further coordinate and
develop PRO collection beyond the clinical encounter, further enhancing patient PRO
monitoring for clinic, research and QI purposes. However, other options for collecting
PROs are still necessary, including interactive voice response and in-clinic reporting.
There is a diversity of approaches to PRO integration, and coordinated efforts are
needed to increase the capacity to use them within EHRs for comparative effectiveness
research. Barriers at the level of the patient, clinician and health system seem to be
modifiable. There are considerable knowledge gaps regarding many scientific and
practical aspects of implementing PRO measures into EHRs. Funding agencies and
government bodies can support targeted research, infrastructure recommendations,
education recommendations and methods development to help overcome current
barriers, and ensure PROs can support the delivery high quality, patient centered care.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
7
INTRODUCTION
The goal of health care systems is to obtain optimal patient health outcomes, decrease
risk and disease, and improve or maintain functioning for individual people and
populations, by efficiently delivering services of the highest quality – that is, services
that are safe, timely, equitable, effective, efficient and patient centered (IOM 2001).
Health care systems should learn from experience, by collecting and converting data
about care and outcomes into knowledge. And such knowledge should be implemented
into evidence-based clinical practice, driving improvements and the process of
discovery as a natural outgrowth of patient care (IOM 2012). Health care choices made
in collaboration between individual patients and their providers are central to this
process. Incorporating the patient voice and perspective through patient reported
outcomes (PRO) measures is critical for clinical care, quality performance management,
and clinical research.
In recent years there has been a convergence of trends in the measurement of health,
the evolution of medical records, and the development of comparative effectiveness
research (Wu 2013). Figure 1 depicts the evolution and convergence these trends. The
vertical axis indicates increasing patient-centeredness, and the horizontal axis indicates
increased digitization. As the measurement of health outcomes has come to more
consistently include the patient’s own assessment of his or her overall health and well-
being, the science of PRO assessment has advanced, and the electronic collection and
storage of health data has become routine. Paper-based medical records have been
converted into electronic health records (EHRs), which can include customizable, built-
in or “tethered” patient portals. Comparative effectiveness research has become more
patient centered, with increased emphasis on stakeholder participation and capturing
the patient perspective on treatments and outcomes. Consequently, PROs, EHRs, and
comparative effectiveness research have converged in an increasingly patient-centered
and digital space, providing the opportunity for the routine implementation of clinical
systems to collection patient-reported information.
PRO data collection is increasingly being integrated into health care. In the US, the
National Institutes of Health, Agency for Healthcare Research and Quality, and Patient
Centered Outcomes Research Institute (PCORI) have supported the development of
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
8
PRO methods for use in research and clinical practice (Lauer 2010, Wu 2010, Selby
2012). The Centers for Medicare and Medicaid Services and other payers, as well as
the Food and Drug Administration, use PROs to evaluate interventions and
programs(FDA 2009). The Office of the National Coordinator for Health Information
Technology (ONC) has supported the use of PROs by allowing their use as evidence
that providers are making “meaningful use” of EHRs to improve quality of care or patient
centeredness, and are therefore eligible for incentive payments (What is Meaningful
Use 2013). In 2012, the National Quality Forum, the American Medical Association-
convened Physician Consortium for Performance Improvement (PCPI) and the
American Society for Clinical Oncology all began initiatives to support the use of PRO
measures for quality measurement and improvement (NQF 2012a,b; personal
communication, email from Kristen McNiff, October 24, 2013). In September 2013, the
Institute of Medicine formed a Committee for Social & Behavioral Domains in Electronic
Health Records (The National Academies 2013) chaired by the Director of the NIH
Office of Behavioral and Social Science Research and PRO expert, Robert Kaplan.
PCORI is organizing a National Workshop to Advance the Use of PRO measures in
Electronic Health Records to be held on November 19-20, 2013 in Atlanta, Georgia. The
workshop aims to review the current state of use of PROs in EHRs, identify barriers and
facilitators to incorporating PRO measures in EHRs and identify specific actions PCORI
and other organizations can take to support and promote the expanded use of PROs in
EHRs.
In support of the meeting, this paper provides a landscape review (i.e., non-systematic
review) of the current state of use of PRO measures in EHRs. The pragmatic, rather
than comprehensive, nature of the review is necessary, as surprisingly little has been
published to date on most of the leading systems to measure PROs in EHRs. The
review includes descriptions of a broad range of initiatives currently underway across
the health care system to integrate PROs with the EHR. The 11 case studies illustrate
the feasibility of integrating PRO measurement systems in various clinical, health plan,
and population-based settings, and the utility of using PROs across clinical care, quality
improvement, and research settings. This review also discusses features of system
design and highlights key elements central to integrating PROs in EHRs on a larger,
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
9
broader scale – including barriers, enabling factors and current knowledge gaps.
What is a Patient Reported Outcome (PRO)?
With the recent emphasis on patient-centered care and research (Selby 2012; IOM
2001, 2012), there is also increasing awareness of the importance of incorporating the
patient’s perspective in quality measurement and improvement. One approach for
systematically capturing the patient’s perspective is the routine collection of patient-
reported outcomes (PROs). PROs are defined as any report coming directly from
patients about their health condition and treatment (FDA 2009) and include a range of
outcomes such as symptoms, functional status, and health-related quality-of-life
(Acquadro 2003). Some PRO measures are “generic” and appropriate for use in a wide
range of diseases, as well as healthy populations; other PRO measures focus on the
symptoms and side effects of a given disease, condition or treatment (Patrick 1989).
There is a long history of using PROs as outcome measures in research studies dating
back to the 1980s (Tarlov 1989; Katz 1987; Lohr 1987; Lohr 1989; Lohr 1992; Lipscomb
2005; Brundage 2007; Bottomley 2007) and somewhat more recently, examples and
investigations of using PROs for individual patient care (Nelson 1990; Wasson 1999;
Meyer 1994; Greenhalgh 2009; Snyder 2009a; Greenhalgh 1999; Valderas 2008;
Marshall 2006; Greenhalgh 2005; Aaronson 2011). A real advantage of PRO
assessment is that the data collected for one purpose can be used in multiple different
ways including clinical care, quality assessment, quality improvement, research and
public reporting (Wu 2013).
Taxonomy of PROs
Greenhalgh (2009) proposed a taxonomy for the different applications of PROs in
clinical practice. This taxonomy classifies whether the PRO data are used at the
individual or aggregated-level and whether the PRO data are used directly or indirectly
to inform patient care. For example, when an individual patient completes a PRO
questionnaire and that patient’s data are provided to his/her provider(s), the data can be
used at the individual level to screen for clinical problems, monitor progress over time,
or promote patient-clinician communication. If this information is aggregated across a
group of patients (e.g., at the provider or clinic level, or for a subgroup of patients) this
can be used to inform quality improvement or conduct population monitoring.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
10
Model for Use of PROs for Clinical Care, Quality Improvement and Research
Complementing the Greenhalgh taxonomy, Snyder and Wu (Snyder 2013a) have
proposed a model that describes the cycle of the use of PROs for quality assessment
and improvement (Figure 2). This model demonstrates how the different aspects of the
Greenhalgh taxonomy relate to each other, and can be used in a streamlined approach.
For both the Greenhalgh taxonomy and the Snyder & Wu model, it is possible to
implement either the full spectrum of applications, or one or more selected applications.
Thus, there are a wide range of opportunities for using PROs in quality measurement
and improvement.
For all applications, the cycle begins with assessing the PROs (Box 1). The
assessments may come from a number of sources, including clinical practice
applications, research studies and population surveys, as described in detail elsewhere
(Snyder 2013b). When PROs are used for clinical practice, they are collected from a
patient with the intention of informing his/her care and management. To date, the large
majority of integration with EHRs has been done for this purpose. When captured in
research settings, the most common use of PROs, it is usually as outcome measures in
clinical trials and observational studies and often occurs outside of EHR systems. In this
application, the primary purpose of the PROs is to describe the impact of various
diseases and/or treatments on measures of health extending beyond clinical endpoints.
Finally, PROs can be collected as part of population-based surveys that provide a
patient-centered perspective to complement other statistics (e.g., mortality rates)(Barr
2003).
In many cases, there is the potential to use these PRO data, regardless of the original
purpose for their collection, to evaluate the quality of care (Box 3). An example of this is
the United Kingdom’s National Health Service (NHS) Patient-Reported Outcome
Measures (PROMS) initiative (http://www.hscic.gov.uk/catalogue/PUB11360). (NHS
2013). Specifically, the NHS is evaluating the quality of care for select surgical
interventions, including hip replacement, knee replacement, varicose vein procedures,
and groin hernias. Patients complete pre-procedure and post-procedure PROs, and
these data provide insight into the patient-centered value of the procedures overall. For
the period from April 2012 to March 2013, there were nearly 240,000 procedures for
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
11
which PRO surveys were to be collected. For the approximately 163,000 pre-
procedures surveys returned, there were 88,000 post-operative questionnaires returned
(NHS 2013). These surveys provide valuable insights into the impact of these surgical
procedures on patient’s functioning and well-being. For example, the proportion of
patients reporting improvements on a general health status measure (the EQ-5D Index)
ranged from 50% for groin hernia respondents to 87% for hip replacement respondents.
The gains on the disease-specific measures were even greater, with 83% of varicose
vein, 92% of knee replacement, and 95% of hip replacement patients reporting
improvement. In addition to providing overall perspectives on the impact of these
surgical procedures, there is also the opportunity to compare different hospital providers,
though this may require case-mix adjustment (Devlin 2010). While the NHS example is
one of the largest applications of PROs for quality measurement, other groups are also
using, or planning to use, PROs in this way. For example, the American Society of
Clinical Oncology is exploring the incorporation of PRO measures as quality indicators
in its Quality Oncology Practice Initiative, focusing first on some common symptoms
such as pain and nausea (personal communication, email from Kristen McNiff, October
24, 2013).
PROs may also be used for population health screening. In this application, PRO
measures may be administered for disease screening, or for health risk assessment
Nelson 2012). Disease screening programs can be used to identify untreated disease,
such as depression in clinical care (American College of Surgeons 2011) or more
broadly in the general population. Screening for health risks, for example, can be used
to engage and motivate individuals to pursue changes in health behaviors, and the self-
management of chronic conditions (Shekelle 2003). Health risk assessment can also be
effective at inspiring the uptake of prevention and health promotion activities in
employee health programs. Linking health promotion to health care visits and physician
advice can potentiate the benefits of PRO screening.
While it is possible to go directly from PRO assessment (Box 1) to quality measurement
(Box 3), an alternative approach would also use the data to improve the quality of
individual patient management, as well (Box 2). The use of PROs in clinical practice
involves having patients not only complete the questionnaires – but making an
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
12
individual patient’s assessment available to the patient’s provider(s) to inform that
patient’s care. Multiple systems have been developed for collecting PROs and using
them for clinical practice (Jensen In Press, Bennett 2012, Rose 2009, Basch 2009). For
example, at Johns Hopkins, we have developed the PatientViewpoint webtool that was
linked to the institution’s home-grown electronic medical record
(www.PatientViewpoint.org) (Snyder 2009, Snyder 2012, Hughes 2012,
http://www.youtube.com/watch?v=S-r4ykaUhfU). This tool enables clinicians to order
PRO questionnaires much in the same way that they order lab tests or imaging studies.
Patients receive an email when it is time to complete a questionnaire and the results are
provided to both the patient and clinician. The use of PROs in clinical practice can
improve patient-clinician communication, and can also have an effect on patient care
and outcomes (Hayward 2006, Valderas 2008, Marshall 2006, Greenhalgh 1999,
Greenhalgh 2005, Velikova 2004, Velikova 2010, Berry 2011, Santana 2010, Detmar
2002, Bliven 2001, Boyce 2013, Espallargues 2000, Gutteling 2008, Lyndon 2011,
Taenzer 2000, Takeuchi 2011) Thus, the PRO collection in itself can be an intervention
with the intention of improving individual-level patient care.
As noted above, using PRO data for individual patient care in no way precludes using
the data for quality measurement. In fact, it facilitates the process if collected in a
systematic way. It is feasible to take some or all of the individual patient’s PRO
assessments and aggregate them to summarize the patterns and quality of care
received at the clinic or health plan level (Box 3). Examples of this might be for pay-for-
performance, or for pubic report cards.
The next step in the process is to use the PRO data to inform quality improvement (Box
4). Dr. John Browne from University College – Cork has described how this process
works using the example of breast reconstruction following mastectomy (Browne 2009).
In Browne’s example, individual surgeons are presented with the average PRO scores
of their reconstruction patients. The surgeons’ performance falls into a distribution, with
some surgeons’ patients reporting lower PRO scores and other surgeons’ patients
reporting higher PRO scores, on average. While PRO measures provide some
indication of relative performance, scores alone are not particularly illuminating in terms
of how to improve care. The key is to be able to translate the scores into descriptive
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
13
labels that can inform patient management. In Browne’s example, lower scores reflect
women who find their breasts’ shape to be acceptable when clothed; the average
scores reflect women reporting that their breasts ‘line up’ when unclothed; and the
highest scores represent women who report that their breasts are equal in size and
shape when unclothed. These descriptive labels can inform a lower-performing surgeon
regarding what areas require increased attention, making the PROs a powerful tool for
quality improvement.
The cycle then begins again with PRO assessment (Box 1) – ideally, with the PRO
scores demonstrating the improvements based on the applications of the PRO data for
quality assessment and improvement (Boxes 2-4). In addition, having accumulated a
number of PRO reports creates another opportunity for using the PROs in clinical
practice (Box 2). That is, group-level PRO data can be assembled in the form of
decision aids that can be used to explain the implications of treatment alternatives and
help patients and clinicians decide on the appropriate strategy for a given patient. In
contrast to the use of an individual’s PROs informing his or her care, in this application,
PRO data from other patients are summarized and presented to a patient to help
him/her understand other patients’ experiences with the various treatment options. For
example, Brundage et al. have shown that lung cancer patients presented with the
hypothetical option of chemotherapy used information regarding the impact of
chemotherapy on survival, toxicity as well as on health-related quality of life to inform
their decision (Brundage 2005). Thus, as more data are collected about patient
experiences using PROs, more patients can benefit from a clearer understanding of the
quality-of-life implications of different treatment options. This cycle is exemplary of the
functioning of a learning health care system – one in which best practices are
embedded in the delivery process and new knowledge is captured as an integral
byproduct of the delivery experience (Olsen 2007).
While the above discussion of the applications for PROs in quality measurement and
improvement around the cycle focuses primarily on the PROs themselves, the power of
PROs increases substantially when the PRO data are linked with other clinical
information (Wu 1997, Snyder 2013b). Linkage to treatments and clinical events can
provide information about treatment effectiveness for individual patients, while linkage to
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
14
patient and disease and provider characteristics can help to generate evidence about
the effectiveness of the care delivered by providers. The integration of PROs in EHRs
offers great potential in terms of applying PROs during each phase of the cycle. Below,
we provide additional background on EHRs and then describe how PROs, combined
with the EHR data, create valuable opportunities for patient-centered care and research.
Electronic Health Records (EHRs)
A more detailed review of EHRs is beyond the scope of this paper. The following is a
brief history of the development of the electronic health record and a description of its
current state. An EHR can be defined as a systematic collection of electronic health
information in digital format about individual patients or populations. Data may be
captured in many ways, including as structured data that can be subjected to immediate
analysis and other ways such as free text or images. Information is stored so that it can
be accessed across different health care settings including hospitals, clinics and other
care facilities, and even by individual patients. EHRs can collect a broad range of data,
including patient demographics, medical and social history, medication and allergies,
diagnoses and problems, immunization status, laboratory and other test results, vital
signs, physical examination findings, billing information and various documents.
Additional functions include the ability to execute orders for tests and medications,
schedule future appointments, generate referrals to other providers, track care and
outcomes, trigger warnings and file public health reports. Information within an EHR
can be used for secondary analyses for research, quality assessment, quality
improvement and reporting (Weiner 2012).
The acronyms EHR and EMR (electronic medical record) are often used
interchangeably. An EMR denotes an electronic version of the patient record created
for hospitals and clinics whereas an EHR has a broader purpose of giving access to a
patient’s medical information to multiple stakeholders across different facilities within
and institution or network, including patients, health care providers, managers, payers,
insurers, and employers.
The modern medical record dates back to the 1970s when records were still maintained
exclusively on paper (Weed 1972). There were a few initial attempts to digitize these
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
15
records and provide computerized decision support (NIH 2006). Electronic medical
records began to be developed in the 1980s for administrative purposes. A notable
effort was the Veterans Administration’s VisTA which was adopted universally across
VA Medical Centers (Brown 2003). In the 1990s the first Windows-based medical
records were released (NIH 2006). The scope of EMRs broadened in the early 2000s to
include a range of non-clinical health information, leading to the term "electronic health
record". Some of these EHRs have integrated patient portals and allowed patients to
communicate securely with health providers and to enter additional information. The
process was greatly accelerated by passage of the Health Information Technology for
Economic and Clinical Health Act and availability of stimulus funds to reward the
“meaningful use” of data by adopters of EHR (ONC 2012). A move to develop
standalone personal health records was also initiated (Tang 2006), which is proceeding
with increasing success but will not be reviewed here.
As these systems have developed, EHR-based patient portals (referred to as tethered
portals) have emerged as the most prevalent structure. These portals permit patients to
retrieve their records and to enter additional information (Tang 2006). It is uncertain
what proportion of EHRs have the capacity to collect PROs. A number of standalone
web tools have been designed specifically to capture and report PRO measures, in part
in response to the lack of availability of this capability within existing EHRs. Most of
these do not interface directly with EHRs. Other example as mentioned above is
PatientViewpoint, developed by our group (which does link to the EHR), with other
examples of integrated systems identified in the case studies section (Snyder 2009,
Snyder 2012).
EHRs systems have the potential to enhance patient-clinician interactions through the
incorporation of patient-level PRO scores. In some EHRs, clinicians can access
reference materials to support decision-making. Other systems have automated alerts
and decision support that is built-in to help guide practice precisely when it is needed.
Today, over 57% of office-based physicians use EHRs (Hsiao 2011). Concurrently,
despite worries about a potential “digital divide” exacerbating existing disparities in care,
internet use is increasingly prevalent for both genders and all age groups,
races/ethnicities and income levels. This advance is due in part to internet access from
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
16
laptops, tablets, telephones and other handheld devices (Zickuhr 2012).
We are now in the midst of a period of rapid development of EHRs capabilities
alongside an even more rapid proliferation of web-based options for PRO collection. A
natural place for these data to go is into EHRs, but there is relatively little published on
the best practices and new developments in linking PRO data collection to the EHR. For
that reason, we undertook a search of the published and grey literature on the current
state of use of PRO measures in electronic health records. We were interested in
identifying highly developed and innovative systems for PRO data collection that feature
EHR integration. We were also interested in common themes in the implementation and
integration of PROs within the EHR, including barriers and facilitating factors, and in
identifying key unanswered questions important to the future on PRO measure
integration in the EHR.
METHODS
We conducted a landscape (non-systematic) review to identify existing and leading
systems that collect PROs and link to EHRs. We were interested in systems that were
used in clinical practice. Systems were eligible if they are used in clinical care settings,
assess PROs electronically and provide summaries of the patients’ response to
providers.
Systems were identified through publications, as well as conference abstracts, white
papers, reports published online or in print and other grey literature. The latter included
information from unpublished presentations, publications and news reports. We
obtained additional detail through interviews with key informants identified by the
Planning Committee and our own network of colleagues and collaborators. PubMed,
MEDLINE and Embase searches used the following terms: [patient-reported outcomes
(outcome assessment, quality of life, health status indicators, patient-reported), and
clinical care (patient care, clinical care, delivery of health care)]. We excluded PRO
data collection systems intended exclusively for clinical trials. We also excluded
electronic PRO data collection systems that were not linked to EHRs. This exclusion
category included some major PRO reporting efforts (Meyer 1994, Gustafson 2001,
Wasson 1999,http://www.howsyourhealth.org/, Cohen 2013), but allowed this report to
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
17
specifically focus on how PRO data is integrated with EHR information. In addition, we
did not include the large number of efforts to measure patient satisfaction independent
of PRO measures of health status. We did not include the many other different kinds of
health related information, including the review of systems, family history, health
behaviors such as tobacco or alcohol use, and health care utilization.
We limited our selection of case studies to US systems (Table 1) although there are
important international efforts of note (Gilbert 2012, Dudgeon 2012, Black 2013,
Varagunam 2013; Bainbridge 2011, Engelen 2010). The cases we present do not
represent the full range of systems. While PRO collection and integration are most
common in specific clinical populations such as cancer, rheumatology and orthopedics,
we selected systems that represent a range of applications from the general population
to specific disease conditions. We included cases used in health systems with different
payment and organizational models and different scale from small to large. For each
case study, system characteristics and clinical implementation were identified and
abstracted using a structured review form created by the authors. We supplemented
the reviews with interviews with the principal system developers and users.The
summary extract of information was verified with the system developers from whom we
also obtained follow-up information when necessary. Table 1 summarizes the 11 case
studies, including their system affiliation and name, the initial clinical population, and
whether they are used at multiple practice locations and for multiple patient populations.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
18
TABLE 1: SUMMARY OF CASE STUDIES
# System Affiliation (Name) Initial
Population
Multiple
Sites/Clinics
Multiple
Populations
1 Epic Systems Corporation
(MyChart, EpicCare)
Epic Users Y Y
3 Cleveland Clinic (Knowledge
Program)
Neurological
Disorders
Y Y
2 Dartmouth Spine Center Spine Y Y
4 Group Health Cooperative
(Health Profile e-HRA)
General Y N
5 Cincinnati Children’s Hospital Rheumatology Y Y
6 Kaiser Permanante Colorado
(PATHWAAY)
Older Adults Y N
7 Essentia Health (MN
Community Measurement)
Depression Y N
8 University of Pittsburgh Medical
Center
Primary Care Y Y
9 Duke University (Patient Care
Monitor)
Cancer Y Y
10 UCLA/Michigan (My GI-Health) GI Disorders Y N
11 University of Washington/
Centers for AIDS Research
Networks of Clinical Systems
HIV Y N
For each case, the first page provides an overview of the system and its development,
including conditions and PROs included; the nature of integration of PRO measures
within the EHR, applications in clinical practice, research and quality improvement, and
future plans for the system. The facing page shows a graphic example of one aspect of
the system, a walk-through of the process of patient assessment and data flow, key
themes highlighted by the case, and data sources.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
19
Epic Systems Corporation MyChart
Basic System Summary:
MyChart (Epic Systems Corporation, Verona, WI) is a secure member website through
which registered patients can view portions of their medical record and exchange secure
messages with physicians. Although collection of PROs within Epic had been
implemented prior to the 2012 release of MyChart, the release of the “series” feature for
PRO ordering and “definition” tool added features that added value for researchers. Epic
has been granted permission to provide the following PRO measures as part of their
Foundation System. They are:
Measure Dimensions
Medical Outcomes Study SF-20 (20 item Short Form health survey), RAND-36
PHQ2/PHQ9 Depression
PROMIS
Adult (18 years and
greater) static short forms
Physical Functioning (10 items); Pain Interference (8 items);
Global Rating of Pain (1 item); Sleep Disturbance (8 items);
Fatigue (8 items); Depression (8 items); Anxiety (8 items);
Satisfaction with Participation in Social Roles (8 items)
Pediatric Self Report (8-
17 years) static short
forms
Physical Functioning--Mobility (8 items); Pain Interference (8
items); Global Rating of Pain (1 item); Fatigue (10 items);
Depressive Symptoms (8 items); Anxiety (8 items); Peer
Relationships (8 items)
Proxy (5-17 years) static
short forms
Physical Functioning--Mobility (8 items); Pain Interference (8
items); Global Rating of Pain (1 item); Fatigue (10 items);
Depressive Symptoms (8 items); Anxiety (8 items); Peer
Relationships (8 items)
EHR Integration:
Complete integration with the Epic EHR. PRO scores can be viewed and manipulated
alongside other clinical data elements such as laboratory test results.
Clinical Practice:
Epic ‘series’ definition functionality makes it possible on an individual or sub-population
basis (e.g., all patients age >
65) to specify the timings and intervals of automated
releases of one or more PRO measures. Organizations can build additional
EpicCare/MyChart questionnaires
Research-Related:
It is possible to specify the timing and intervals of automated releases of PRO
assessments. PRO data are aggregated across patients in Epic’s analytic environments
(Cogito Clarity and Cogito Data Warehouse) with common data structures across
organizations.
Quality Improvement:
Organizations that have implemented Clarity and/or the data warehouse are able to
develop reports that include PRO data.
Future Plans:
Long-term plans are to expand to additional PROs in their Foundation system.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
20
Epic Systems Corporation: EpicCare longitudinal integration of PRO and clinical
data
Walk-through of the Patient Assessment Process
A provider working within the EpicCare EHR orders one or more PROs to be completed
by the patient. The patient receives secure email notification via the MyChart patient portal
and completes the PRO assessment online. The results are released immediately to the
EpicCare EHR where they can be viewed by the provider alongside other test results.
Key Themes
• Completed via tethered patient portal MyChart
• Fully integrated within Epic ecosystem including EpicCare EHR. PRO data are
aggregated across patients in Epic’s analytic environments (Cogito Clarity and
Cogito Data Warehouse)
• Foundation system includes pre-specified PRO measures
• New PRO measures require Epic trained programmers to implement within
MyChart
Sources: Unpublished presentations, discussion and email communication (Nancy
Smider)
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
21
Cleveland Clinic Knowledge Program
Basic System Summary
The Knowledge Program (KP) is a data collection system and repository designed to
enhance electronic clinical data, including the addition of PROs. Goals were to help guide
care, to link to CarePaths (CP) to standardize management of specific conditions and to
increase the ability to extract data from the EHR for secondary use. KP was developed in
2007 by the Neurological Institute (NI) of the Cleveland Clinic in collaboration with the
Imaging Institute and IT Division. All departments in the institute started simultaneously in
a “Big Bang.” KP uses both standard PRO measures and custom questionnaires, which
can be built quickly using a content manager tool. Conditional flows of questions can be
built in. There are provider questions to create metadata, e.g., why the patient did not
complete a PRO or date of a clinical event. A KP query tool can extract different kinds of
data for secondary use. Most patients surveyed thought the data were useful (especially if
the provider reviewed results with the patient) and questionnaires were not too long.
Completion rates are >80%. The majority of providers surveyed also reported that PRO
data were useful. KP now includes all of orthopedic spine surgery, cardiology,
gastroenterology, head & neck, and plastic surgery. In addition to the standard EQ-5D and
PHQ-9, KP also collects the PDQ, a Pain scale, Central Sensitization Inventory, Modified
JOA (neck or cervical myelopathy) and questions on days missed and work status, for a
total of 164 PROs or ClinROs.
EHR Integration
: Survey build and data collection are done outside of the Epic
environment, but reports are viewable in EpicCare. Office-based collection of data via
tablets goes to a custom “NI health status” tab in the Epic navigation panel. The provider
display comprises screen shots, flow sheets and text and cannot be manipulated within
Epic. NI has encouraged use of MyChart for PRO data collection prior to an ambulatory
visit – MyChart now comprises up to 23% of PROs completed.
Clinical Practice
: A total of over 961,000 patient visits to 1,062 providers; There are more
than 330,000 individual patients with PRO data since 2007. Scores can be tracked over
time within KP. Neurologists have found PHQ9 helpful. A best practice alert (BPA) was set
for a score greater than 15; an order set pops up with recommendations for
antidepressants, referral to behavioral health specialists, and patient education.
Research
: In neurology, PROs are used in clinical research to track response by disease
and type of patient, and how depression affects management and outcomes for specific
conditions. Ortho/spine finds KP to be very useful, with 3-4 of their faculty doing
exclusively outcomes research. There are some frustrations with missing data and follow-
up. Many current studies are ongoing including: CER, e.g., in degenerative
spondylolithesis, radiculopathy treatment; Methodologic: e.g., responsiveness of mJOA
score to improved QOL; Clinical, e.g., the effect of obesity on patient outcomes; Health
services research: cost of surgery vs outcome; outcomes for 8 different surgeons (Gross
2013; Gogawala 2011, Conway 2012; Su 2012, Jehi 2011).
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
22
Cleveland Clinic Knowledge Program: PRO measure sets available in EpicCare
Walk-through of the Patient Assessment Process
Data collection is currently mostly on tablets which are given to patients in the office
waiting room when they arrive to register. Kiosks and increasingly MyChart (over 20% of
all PROs completed in 2012) are also used for data capture. The data go into the KP
databases and can also be seen by providers on a “KP Patient Data” tab in the Epic
navigation panel. A KP tool allows data extraction for research and quality improvement.
Key themes
• Initially built around neurological disease. Goals included individual patient care,
quality of care, research and policy/reimbursement (demonstrating outcomes
needed to show value)
• Large number of standard and custom PROs (and some ClinROs: a total of 164
patient or provider questionnaires)
• Initially tablet computer entry, now increased proportion via MyChart with
integration with EpicCare EHR
• Important enablers: leadership support, effort was clinically driven, allowing
individualized questionnaires, availability of metrics on completion and use,
integration of technical and clinical teams
• Barriers: dedicated resources are needed to analyze and report PRO results
Sources: KP webinar August 7, 2013; discussion (Irene Katzan, Ajit Krishnaney, Eric
Mayer); Katzan 2011 a,b; Katzan 2012; Atreja 2012; Gurland 2010).
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
23
Dartmouth Spine Center
Basic System Summary:
The Dartmouth system is a PRO data collection system integrated within the Epic EHR.
The Center began PRO data collection efforts in 1998, based at the Dartmouth Hitchcock
Medical Center in Lebanon, NH, led by Dr. James Weinstein (Weinstein 2000). The
collection of structured PRO data before each visit was a fundamental process for the
achievement of care planning, shared decision making, monitoring individual impact of
treatment and quality improvement, research, and public reporting. In 2005 Dartmouth
partnered with Dynamic Clinical Systems to collect PRO measures, interfacing with their
home grown EMR. At the time, reports were visible in the EMR as blocks of text. The initial
driving force came in part from early adopters who already believed in the value of PROs.
Every condition had a local physician champion and selection of PROs came about from
extended conversations among a multidisciplinary team. In April 2011, Dartmouth
transitioned to Epic and the decision was made to do as much as possible within Epic to
minimize interfaces. Dartmouth is currently collecting PROs within 12+ clinical programs
(Ortho, Plastics, Spine, Pain, Heme/Onc, Psychiatry, OB/GYN, Rehab, Neurology, Primary
Care, Surgery/Anesthesia, Vascular) and 25+ different health conditions
(Hip/Knee/Shoulder; Hand/Breast; Breast, Head & Neck, Neuro and Prostate Cancer;
Sleep, Depression, Anxiety; Epilepsy/MS). PRO measures include PROMIS, VR 12, PHQ-
9, condition specific instruments, a risk assessment tool, alcohol screening, and many
custom questionnaires (e.g. Review of Systems, ADLs). Baseline collection rate is >80%
but is more limited for follow-up data.
EHR Integration:
Data collection is either via MyChart or Welcome tablets/kiosks in the office and is fully
integrated in EpicCare EMR. Foundation PRO measures are entered as discrete data and
can be manipulated in Epic. The care team can review results immediately in chart review
or visit navigator.
Clinical Practice:
All electronic communication is via a secure patient portal. Summary reports are generated
from PRO data for making and monitoring the care plan. A survey of spine patients
(Hvitfeldt 2009) found that 80% rated the system excellent to good and 1/3 reported that
PROs had led to positive changes in their visit (pre EPIC implementation). Clinicians
reported that PRO measurement systems were important for follow up and feedback but
expressed mixed views on whether it saved or added time to the visit. Alerts are set up for
patient reports of suicidal ideation.
Research-Related:
PROs are used in practice-based research. Research consent forms are programmed like
other questions into the system. The Spine Outcomes Research Trial (SPORT) has
followed thousands of patients in 13 centers for >5 years, with most of the primary results
based on PRO data (Weinstein 2006, 2007).
Quality Improvement:
Data are used for program performance and improvement, and are reported on the
Dartmouth public website.
Future Plans:
Long-term goals are to improve the ability of clinicians to order PROs, to increase the
customization of clinical reports, and to improve score interpretation for patients and
clinicians. Additional goals are to improve compliance with follow-up PRO data collection
and system capabilities to extract population data for quality improvement and research.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
24
Dartmouth Spine Center: PRO domains flowsheet data in EpicCare
©2013EpicSystemsCorporation.
Usedwithpermission
Walk-through of the Patient Assessment Process
When a new patient or an annual visit is scheduled the scheduling system follows rules to
order appropriate PRO measures that are tied to the upcoming appointment. If the patient
is active on MyChart, s/he gets appointment and questionnaire reminders. If not, or if the
patient does not fill out the questionnaire through MyChart, the patient receives a tablet at
check in and completes PROs in the waiting or exam room. When the patient sees the
clinician they review current and trended data. The clinician charts core/meta data on the
patient. The warehouse receives, stores, manages, and analyzes data from multiple
sources including diagnostic tests and claims. The warehouse distributes reports for
individual patients, clinical populations, quality measures, and research. The next release
of Epic will allow series of PROs to be ordered.
Key themes
• Initially built for spine surgery, vision was importance of the patient perspective,
collecting information at point of care, simultaneously built into research.
• PROs for 12 different clinical programs and 25 different health conditions.
• Initially based on Dynamic Clinical Systems for PRO data collection and home
grown EMR, switched in 2011 to Epic and MyChart with some losses and gains in
functionality.
• Important enablers: support from top clinicians and leaders and enthusiasts; effort
was clinically driven
• Barriers: Epic lacks easily accessible features to customize clinical reports. It has
taken significant resources and expertise to get a standard report on questionnaire
completion rates.
Sources: KP webinar October 30
,
2013
;
discussion
(
Carol
y
n Kerri
g
an
)
;
Nelson 2012
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
25
Group Health Cooperative Health Profile (Health Risk Assessment)
Basic System Summary
The Health Profile is an electronic Health Risk Assessment (e-HRA) that targets adults and
is integrated with the EpicCare electronic record at Group Health Cooperative, a large
health plan in Washington State. The overall goal is to provide advice to patients and their
providers based on information entered by patients via the patient portal (MyGroupHealth)
that is integrate in the EHR. In 2006, Group Health developed the interactive online e-HRA
that collects information from members, integrates it with other EHR data and produces
individualized health improvement recommendations. The recommendations align with
Group Health’s clinical practice guidelines. The goal is to promote prevention and health
promotion behaviors through patient interaction with their Health Profile. Preventive health
behaviors are reinforced by clinicians since the recommendations align with Group Health
clinical practice guidelines. Data are also used for quality improvement. PRO measures
are part the e-HRA, including functional health status, health-related quality of life, and the
PHQ-2. The questionnaire takes approximately 20 minutes to complete and incorporates
branching logic and algorithms. Health Profile data appear in EpicCare as structured data,
free text and reports. Past reports are archived and can be reviewed. Messages are
triggered if an urgent need is identified. There is a paper-based alternative that is not
linked to the EHR.
EHR Integration:
Data collected via MyGroupHealth flow as reports into EpicCare. There is 70% uptake of
MyGroupHealth. These are structured data elements that cannot be manipulated within
Epic. Data are summarized in recommendations to be reviewed by the patient and
clinicians.
Clinical Practice:
Design of the system was driven largely by clinicians and executed by an integrated
multidisciplinary team. Selection of instruments, risk calculation and recommendations are
evidence based so they are more likely to be taken up.
Research- Related:
There has been relatively little use for research thus far.
Quality Improvement:
Population-based estimates of disease risk, health status, and gaps in care are delivered
to health care purchasers and to Group Health managers to direct resource allocation,
care management programs, and quality improvement activities.
Future Plans:
Long-term plans are to improve integration of reports and data within EHR. No specific
plans for expansion.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
26
Group Health: Health Risk Assessment viewed in EpicCare
Walk-through of the Patient Assessment Process
A new member is made aware of the HRA when s/he logs on to MyGroupHealth or
alternatively, from a prompt from an employer that provides an incentive for completing it. If
members do not complete it within 1 year, they receive a prompt. When the member
completes the HRA (takes approximately 20 minutes) s/he is provided with immediate
personalized health recommendations. A similar report goes immediately into EpicCare
and is available for clinicians to review. Specific patient risk factors trigger
recommendations based on Group Health clinical practice guidelines. Data are stored in
Epic and in a database for analysis and aggregation.
Key themes
• Part of a comprehensive e-Health Risk Assessment
• There is 70% uptake of MyGroupHealth patient portal
• PRO data and other patient output from the HP are viewable as PDFs in the EHR
• Important enablers: Employer incentives to complete the e-HRA
• Barriers: Full integration with the EHR is costly and complex and not yet
accomplished.
Sources: GHC internal documents; discussion (Paul Lozano, Rob Reid) (Reid 2010,
Nelson 2012)
©2013EpicSystemsCorporation.
Usedwithpermission
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
27
Cincinnati Children’s Hospital Medical Center
Basic System Summary
The Cincinnati Children’s system uses a tablet- and kiosk-based PRO data collection
system that integrates via Welcome with the Epic EHR. Cincinnati Children’s started
collecting PROs for clinical use five years ago under a broad strategic initiative to improve
patient outcomes with a focus on quality improvement methods and tracking of
performance and outcomes metrics. Individual clinics were invited to identify 3 medical
conditions on which to focus for their panel of patients. Efforts to collect PROs at the
hospital level have ramped up over the past two years, with a focus on using the best
possible methods and evidence possible with a trained psychometrician hired full-time to
consult with clinics on measure selection and data analysis. Currently, a number of clinics
are collecting a widely used PRO, the PedsQL™. The completion rate of patients eligible
to receive the measure via tablet is 79%. Many of the disease-specific and functional
measures developed and used at the clinic level are administered to targeted patient
groups.
EHR Integration:
Patients are administered questionnaires at intake based on provider-specified variables
pulled automatically from their EHR and date of previous survey (if previously
administered).
Clinical Practice
PRO scores generated using an algorithm within Welcome are used at the point of care to
inform the visit. In the rheumatology clinic physical function and pain interference are
important patient outcomes that are tracked and reviewed at each visit. Changes in these
scores are used to help identify and select the appropriate patient intervention (e.g.,
medication adjustment, physical therapy). In the Heart Institute, children with
cardiomyopathy are provided a depression PRO. Depression has been found to impact
clinical results in children with chronic diseases therefore the goal is prompt identification
and intervention. The Food Allergy clinic is working to identify anxiety children experience
from their allergic state. The goal is again prompt identification and intervention.
Research- Related
All administered measures must have a clinical rationale; nothing is collected purely for
research purposes. Examples of research are: Evaluation of psychometric properties of a
short version of the Children’s Depression Inventory in a clinical setting, and for Type 1
Diabetes care.
Quality Improvement
Key goals of quality improvement efforts are to reduce staff involvement in PRO
administration and minimize patient burden. Many different quality improvement methods
(e.g., swim-lane diagrams, process flow maps, PDSA cycles) have been applied at the
clinic-level to streamline the integration of PROs into the clinic work flow and capitalize on
electronic system automation to reduce staff burden. Results of these efforts include
identifying patients that receive a large number of pre-visit PRO questionnaires (e.g.,
psychological batteries) and scheduling those patients to come in 20 minutes earlier.
Future Plans
Long-term goals are to create a clinical repository for CER, and integrate dynamic
assessment (e.g., PROMIS CATs). Eventually, would also like to push all PRO and other
patient information capture outside of the patient visit.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
28
Cincinnati Children’s: Child PRO assessment (Used with permission)
Walk-through of the Patient Assessment Process
At the registration desk, the patient is given a personal numeric code and either a portable
electronic tablet or directed towards a computerized kiosk. The code is linked to their
electronic medical record ID. A pre-programmed algorithm (based on diagnosis, age, time
interval since previous assessment, and previous PRO scores) selects PRO
questionnaires personalized for the patient. If PRO questionnaires are not completed in
the waiting room, the tablet can be taken with the patient to the exam room. Clinicians
review scores with patients during the clinic visit.
Key Themes
• Automatic PRO selection is driven by patient characteristics
• Strong institutional support and infrastructure
• Focus on automation and workflow integration using quality improvement methods
Sources: Discussion (Esi Morgan-DeWitt, Ian Kudel, Evi Alessandrini)
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
29
Kaiser Permanante Colorado (KPCO) Proactive Assessment of Total Health &
Wellness to Add Active Years (PATHWAAY) for Seniors
Basic System Summary
The KPCO program is a comprehensive, population based method of screening to deliver
evidence-based interventions for common geriatric issues, developed in 2011. PROs are
seen as essential to handling geriatric problems. The goal was to develop a system of
member health assessments including PRO that links to the EMR (Epic) and integrates
with care to produce individualized action plans. Additional goals were to extract data from
the EHR for quality measurement and research. Interactive voice response (IVR) was
developed to allow data collection for patients not using KP.org (aka MyChart). IVR data
are also integrated into the KP Health Connect (aka EpicCare) EHR. Pilot work was
conducted in 2012 at 2 clinics, with implementation by July 2012 in all 24 KPCO primary
care clinics. PROs collected are part of the Total Health Assessment questionnaire,
including PHQ-2 and pain. Among seniors who rated the online process, 72% rated it “very
easy” to use. This rating increased to 75% for the IVR process. Patients are given
Personalized Prevention Plans (PPP) which are also integrated into Epic. People with
positive triggers get direct telephone outreach. Use of automation and centralized support
($700K in salary+benefits) is estimated to save 3.2 FTE of added PCP time resulting in
>$200K net savings.
EHR Integration: Both data collected via KP.org, and IVR data flow directly into KP Health
Connect EHR. Actual data elements can be manipulated within Epic. Data are summarized
in Health Trac, a PPP letter is created in Health Connect and is ready to be reviewed by
the clinician. SmartSets are available to order subsequent testing and treatment. Data can
then be extracted from the Epic Clarity database
Clinical Practice Since July 2012 over 41,000 of a total population of ~90,000
seniors have completed an annual wellness visit, of whom 78% completed a THA, 72% of
them before the primary care visit (40% via KP.org). Within Health Connect, screening is
followed by evidence-based interventions for the related geriatric issues. For example, a
positive screen for depression would be linked to a SmartSet for appropriate testing,
treatment and referral. For the 65% of patients with at least one positive trigger, the team
has contacted nearly half by phone for further assessments and to make algorithm driven
recommendations and referrals.
Research-Related: Relatively little use for research thus far, but collaborating with the KP
Institute for Health Research and others. There is a no-call list for patients who have opted
out of research participation. A waiver of consent for research is applied to the remainder.
Quality Improvement: Metrics are developed as required by Medicare, including for
depression and anxiety. There are links to other programs such as a depression care
management team, and a CMMI funded COMPASS project using team care. No PRO
related dashboards or pay-for-performance yet.
Future Plans: Expand to other KP regions including KPNorthwest, KPGeorgia, and
KPSouthernCalifornia. Increase branching logic for PRO measures. Complete an inter-
regional data repository. Increase capacity for use of tablet computers for in-clinic data
collection. Develop history tool to be collected on a one-time basis at enrollment. Increase
automation for generation and printing of Personalized Prevention Plans and data flow to
reduce staff burden. Results of these efforts include identifying patients that receive a large
number of pre-visit PRO questionnaires and scheduling them to come in 20 minutes
earlier.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
30
KPCO: Health Risk Assessment scores from Integrated Voice Response in EpicCare
Walk-through of the Patient Assessment Process
The process begins when a member calls for a non-urgent appointment or is scheduled for
an annual wellness visit (AWV). If the patient is on KP.org the call center assistant assigns
a THA to be completed 1-2 weeks before the visit; if not, it does warm transfer to IVR
system. The data goes into the KP Health Connect EHR, and are summarized in Health
Trac. Positive triggers go to Senior Assessment nurses who call patients to obtain more
information. Medical assistants use data to complete a PPP which is put into Health
Connect including instructions to print a copy for the provider. All information is available at
the time of the patient visit ready to be used by provider. SmartSets make evidence-based
recommendations and make documentation easier. If THA is not done before the visit, it is
assigned to be completed the day after. Data are stored in Clarity and can be extracted for
research and QI.
Key themes
50% of patients age > 65 are on KP.org
Warm transfer at scheduling of Annual Wellness Visit assures that patients not on
KP.org complete Total Health Assessment using IVR
PRO data from THA collected both via KP.org and IVR are fully integrated into EHR
Important enablers: CMS mandate to add a Total Health Assessment to the Medicare
AWV; Central Program Office support with investment in IVR; building at regional level;
cost effectiveness analysis
Barriers: Limited functionality of EHR, lack of universal internet connectivity in facilities
Sources: KP internal documents; discussion (Matt Stiefel, Wendolyn Gozansky, Eric Mayer)
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
31
Minnesota Community Measurement (MNCM) – Essentia Health System
Basic System Summary
The MNCM system is free-standing statewide data collection effort that includes PRO
measures. MNCM is a non-profit organization that emerged in 2002 out of a community-
wide need for a consistent way of measuring and reporting health care quality measures
to the community. Minnesota’s health plans were already working together to sponsor the
Institute for Clinical Systems Improvement (ICSI) for quality improvement. The medical
directors of these health plans wanted a single, combined report to compare patient care
and outcomes statewide, including public reporting. An explicit goal is to support the CMS
triple aim of improving Health, Experience, and Cost. Their first performance report was on
diabetes care; by 2004 results were published on the public website MNHealthscores.org.
Capturing patient experience of care has included the PHQ-9, the Asthma Control Test
(ACT), Asthma Therapy Assessment Questionnaire (ATAQ), Asthma Control
Questionnaire (ACQ), functional status tools for Knee Replacement and Spine Surgery.
EHR Integration:
Data are collected electronically at each site, but are not necessarily integrated into local
EHRs. At Essentia Health, a health plan that includes 18 hospitals and 68 clinics and that
utilizes MNCM, data are increasingly collected via the Epic “MyHealth” patient portal (25%
enrolled). The remainder are collected during patient visits, or mailed paper questionnaire,
or phoned to complete data collection. Quality metrics are built in to their Epic EHR. Data
are exported to an external Quality Data Warehouse.
Clinical Practice
MNCM works with many clinics, hospitals, and medical groups in Minnesota, Wisconsin,
and North and South Dakota. At Essentia, depression screening using PHQ-9 is linked to
a Depression Clinical Workflow Guide provided by ICSI, and a Help and Healing online
toolkit that gives providers evidence-based treatment guidance and easy-to-use resources
to assist in depression recovery. Some of the Essentia clinics have specially-trained RN
care managers to work one-on-one with patients who have depression.
Research-Related:
MNCM is a grantee of the Robert Wood Johnson Foundation Aligning Forces for Quality
(AF4Q) program.
Quality Improvement
:
The PHQ-9 based performance measure has been endorsed by NQF. Currently, MNCM
reports on over 76 measures at over 315 medical groups and 672 sites of care. Reporting
of PRO measures is linked to statewide quality improvement programs such as DIAMOND
(Depression Improvement Across Minnesota). At Essentia Health, quality metrics are built
into the EPIC EHR and are used by the performance improvement team.
Future Plans:
Long-term goals are to expand to include both cost and experience of care, and Additional
PRO measures for spine surgery, total knee replacement, and others.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
32
Essentia Health: After answering questions, patient can review and edit answers
Walk-through of the Patient Assessment Process
At Essentia Health, all patients with a recent visit diagnosis of depression complete the
PHQ-9 at a minimum of every 6 months. A total of 25% of Essentia patients are enrolled
on the Epic “My Health” patient portal. For patients with depression, if the patient does not
complete the PRO online, an aide calls or mails them to obtain the data. PHQ-9 data are
integrated into EpicCare EHR with standardized workflow for evaluation and treatment.
Quality metrics are calculated and addressed by the local quality improvement team and
are also reported to the State report card.
Key themes
Measurement of PROs supported by pay-for-performance at most health plans
Reporting is made mandatory by Minnesota Department of Health
Data collected at clinics and health plans; results are fed back and made publicly
available
Sources: Website, discussion (Tina Frontera MNCM COO; Patrick A. Twomey, MD CMO
Essentia); http://mncm.org/
; http://MNhealthscores.org/; http://mncm.org/submitting-
data/provider-tools/
©2013EpicSystemsCorporation.
Usedwithpermission
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
33
University of Pittsburgh Medical Center
Basic System Summary:
The UPMC system is a PRO data collection system built upon the Epic EHR. In 2004, this
group implemented a standardized electronic intake system in an academic general
internal medicine practice for collecting PROs and other patient data (exercise, smoking
status, review of systems). The results from this 10-year effort have laid the foundation for
a health system-wide initiative to integrate PROs into the health care system. In the next 5
years, electronic clinical intake, including PRO and patient history collection, will be
integrated into UPMCs EHR across 861 outpatient practices in 35 counties.
EHR Integration:
PRO data collection is integrated into Epic, through MyChart and Welcome. Patient
questionnaires are seen by the clinician within the visit navigator, allowing item level and
total scores to be displayed to the clinician, and are available in a flow sheet for review
outside of the clinical encounter.
Clinical Practice:
The system is designed primarily for clinical practice use. Standards to measure an
individual construct are set at the system level. For example, most clinics use the same
general quality of life measure (SF 12 or SF 36), while disease-specific questionnaires are
more variable (e.g., Seattle Angina Questionnaire, Pain Disability Index, Oswestry low
back pain questionnaire). In settings with a compelling alternative measure (e.g., the
FACT for cancer), those alternative measures are used. Many different longitudinal
reporting options are available with intervention overlays. For example, clinicians can
graphically overlay interventions or medication changes on PROs to help understand the
impact of intervention on patient-centered outcomes. Unanticipated benefits include the
planned use by the physical medicine and rehab group to provide additional objective
evidence of treatment benefit to argue for coverage of some clinical care delivered. In
addition to individual-level patient data, the “real-world” group level data can be used to
advocate for more global policy changes.
Research-Related
While the focus of this work is clinical care, as with all data collected for clinical use, it is
available for research and quality improvement purposes as well. These data can be
reported out with the same ease as other electronic data for use by investigators.
Quality Improvement
System specific: Every week managers of each group are sent usage statistics: number of
patients, percent of questionnaires completed and the change in percent completion from
the prior weeks. Departmental and practice managers can compare completion rates
across practice form the same service line. If a clinic reports consistent completion rate
under 80%, operation teams are sent in to review and identify the problems and work with
the practice manager to overcome these issues.
Future Plans
The program goal is to help meet meaningful use requirements (including communication
between the provider and patient outside of the care setting), decrease costs, provide
opportunities for new models of care, and leverage in new research opportunities.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
34
UPMC: PRO item in Epic Welcome for patient completion
Walk-through of the Patient Assessment Process
All patients scheduled for a visit automatically receive a questionnaire within a time
window deemed clinically appropriate according to the patient and the type of visit (e.g., 1
week for physical medicine and rehabilitation, 2 weeks for cardiology visits). Patients can
fill out the questionnaire at home or in the clinic. Our goal for completion time is under 10
minutes. Questionnaires vary based on the clinic type. Clinicians are required to
document in the EHR when they have reviewed the patient history information, PRO
questionnaires are filed automatically.
Key themes
• Scaling-up successfully to health system-wide integration
• Focus on the patient-provider experience
• Strong institutional support
Sources: Interview and email communication (Rachel Hess)
©2013EpicSystemsCorporation.Usedwithpermission
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
35
Duke University Patient Care Monitor
Basic System Summary:
Patient Care Monitor (PCM) is an electronic system to collect and store PRO measures
and provide the scores to cancer care providers. The vision for PCM was to fill the critical
need for continuously aggregating information at the point of care with reliable, high
quality, and clinically useful data. The foundation principle was around the question of
“how do we wire up healthcare to include the patient voice” and how PROs can capture
the patient voice in a systematic and clinically relevant way.
EHR Integration:
Integration of the PCM was accomplished at the site level. At Duke programming was
used to integrate PCM with the data warehouse and patient scheduling information.
Clinical Practice:
Summary reports are generated that can be viewed by the entire medical team, clinic
providers (nurse, social worker, PA/NP, physician), and staff. The reports are color coded
and designed to show doctors a pattern quickly. PRO scores can trigger immediate
patient education actions or flag patient eligibility for symptom-specific interventions (e.g.,
depression, sexual function, sleep disturbance). On the provider side, results are included
in the routing sheet.
Research-Related:
Data can be collected at point of care for research purposes. Consent forms are
appended as necessary and tailored to patients who are targeted for study enrollment.
Otherwise, PRO data are available from the data warehouse for retrospective studies.
Quality Improvement:
PRO data has been integrated into many quality improvement initiatives. For example,
using PRO pain scores, they have been able to predict how many pain physicians are
necessary to address patient pain concerns.
Future Plans:
Duke plans to roll out PCM to multiple sites across the country.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
36
Duke Patient Care Monitor: Physician screen showing embedded scores to aid
interpretation
©2013 Patient Care Monitor. Used with permission
Walk-through of the Patient Assessment Process:
The walk through is site-specific and variable. For instance, patients in one clinical
oncology clinic are given a handheld tablet computer at check-in to complete a
comprehensive, validated PRO questionnaire measuring physical symptoms, impaired
performance, impaired ambulation, treatment side effects, distress and despair. PRO
scores are summarized for the clinician’s review using a color-coded report that includes
current scores and change over time. Other clinical sites may use desktops, kiosks, or
mobile devices to have patients complete PROs prior to coming in for the visit. Providers
need to acknowledge to patients that they have reviewed the PRO data. This is key to get
patients to report prior to their next visit and, in turn, to have high quality and reliable data.
There is also a patient-version of the report that providers more detailed explanation than
the care team’s version.
Key Themes:
Built based on rapid learning health care infrastructure
Includes system review and visual aids
Facilitates patient education and enrollment into symptom management
interventions when they meet enrollment criteria
Source: Interview (Amy Abernethy); Abernethy 2009; 2010 a,b
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
37
University of California Los Angeles/University of Michigan My GI-Health
Basic System Summary:
The My GI Health Project (myGIhealth.org) is a web-based PRO collection and reporting
system developed by the University of California, Los Angeles and the University of
Michigan (Center for Health Communications Research) for use in GI clinics. Currently
this system is used in three clinics (UCLA, West Los Angeles VA, and U Michigan GI
clinics). This system was designed to collect PROs and provide interpretation for clinical
use. The focus is generating GI-symptom scores (e.g., reflux, gas/bloating, abdominal
pain, constipation) and building a standardized patient history of presenting illness (HPI)
that incorporates PRO information. It also provides patient education with
recommendations tailored to their PRO scores. My GI Health uses the PROMIS GI
Symptom Scales, enabling score comparisons to the general U.S. population.
EHR Integration:
This system is designed to work with EHR systems, but all integration is done manually by
the provider. PRO-based patient histories and scores can be added into the EHR through
copy and paste function commands. While a separate system, My GI Health has created
features to create the impression of integration between their tool and the EHR systems in
use at each site. Two methods have been used: (1) allow easy access to the My GI Health
system by placing a launch button in the EHR tool bar. (2) Providers use the same log-in
credentials for the My GI Health system as their EHR. Two data servers are used for this
system. Patient questionnaires are completed and scored through UCLA. All educational
content that forms the “personal educational prescription” occurs through a server at the
University of Michigan.
Clinical Practice:
This system was designed specifically to supplement clinical practice. Providers can
customize the experience for both themselves and their patients using various settings.
These options range from restricting when patients can review information to how much
information they view to restricting types of patients who enter the system. Providers can
also select what information is reported to them. For example, they can select the cut
points for symptom severity. Patients receive a personalized educational prescription
based on their scores that incorporates patient-centered animations, selected websites,
and other online resources based on their PROMIS “fingerprint” of scores.
Research-Related:
Future plans include a “big data perspective” and patient-reported informatics including
incorporating wireless biosensors to the collection of PRO scores. The system is currently
funded through a PROMIS supplemental grant to be tested in a randomized controlled
trial that will begin in Winter 2013 and end in Spring 2014. Another study is currently
testing the quality of the physician-generated HPI notes vs. the My GI Health-generated
automated HPI notes.
Quality Improvement:
None
Future Plans:
My GI Health is expanding in a number of ways: (1) Developing an iPhone “native app”
for patients to report their symptom information. (2) Considering the administration of
assessments beyond the clinical setting into the community to promote patient education
and self-management. (3) In discussions Athena health, to pilot an EHR-linkage.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
38
UCLA/University of Michigan: My GI-Health Report (Used with permission)
Walk-through of the Patient Assessment Process
All patients receive a letter prior to their visit with information about the system and how to
log on via the web. Patients can then register for the system and create log-in credentials.
They see a welcome video (that can be personalized by each provider), complete a
symptom assessment and provide a history of their symptoms. Once scored, patients may
be linked directly to educational resources and scores. When patients can view scores
and access educational content is based on provider settings. Providers may choose to
wait to release these features after the patient visit. Patients and providers then review
and discuss the symptom report at the upcoming visit.
Key themes
Built around a body system (GI) and related symptoms, not disease-specific
Providers are given a high degree of individual control on how and when PRO
information is reported to them and their patients
Considerable preliminary qualitative work with physicians was conducted to
develop and finalize symptom reports
Sources: Discussion (Brennan Spiegel)
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
39
University of Washington-Center for AIDS Research Networks of Clinical Systems
(CNICS)
Basic System Summary
The UW System is a web-based, self-administered patient-reported assessment tool
integrated into routine primary care for adult HIV-infected patients. The goal of this system
is to collect clinically-relevant patient-reported data and integrate it into clinical care. It was
first implemented as a research study modifying a PRO platform used in oncology settings
for HIV care and research. The ability to tailor to specific clinical sites was one of the keys
to succeeding in various settings. The developers worked with patients, and providers to
identify which patient domains were actionable, could inform clinical care and research,
and were relevant to the HIV/AIDS population. PROs integrated into the system includes
depression, medication adherence, and drug and alcohol use. Preliminary findings were
used to develop institutional buy-in and support, and have allowed for further development
and integration with broader quality improvement efforts. PRO data collected by this
system are integrated into a national disease specific research network.
EHR Integration:
Integration into the EHR is site-specific and variable. In some sites the system works side-
by-side with the EHR, enabling data elements to be pulled from the EHR system and put
back into the EHR. All assessments are done through a separate system. The benefit of
this type of integration is that it has allowed easier implementation at other sites (e.g.,
University of Alabama) with different EHR systems. Functionality and content is “adjusted”
according to site requirements for data to flow between the two systems. One of the
biggest challenges has been navigating the variety of EHRs across sites and cross-
platform adaptation of functionality.
Clinical Practice:
The system is currently used in clinical practice with s cores directing clinical decisions.
For example a positive response to the suicidal ideation item is flagged, and the mental
health team is notified immediately so that the patient is seen by a mental health worker
before the provider. An evaluation showed that using this system has lead to an increase
in depression screening, identification of medication adherence, and previously-
undocumented substance use.
Research-Related:
This assessment has been implemented into HIV clinics as part of the Centers for AIDS
Research Network of Integrated Clinical Systems (CNICS) cohort. PRO assessments
have been completed over 29,000 times.
Quality Improvement:
The UW clinic has implemented Plan-Do-Study-Act cycles to identify and address PRO
issues surrounding clinic flow, and how to integrate the scheduling and delivery of
assessment results.
Future Plans:
They are currently expanding the clinical assessment to a number of clinics within the
Community Health Applied Research Network (Likumahuwa 2013). Grants have been
submitted to examine how this information can be tailored for patients.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
40
University of Washington: Feedback to providers on patient PRO scores
Usedwith
permission
Walk-through of the Patient Assessment Process
Patients are scheduled to fill out a PRO survey 20 minutes prior to their provider visit
using a touchscreen tablet in the clinic. Patients fill out surveys every 4 to 6 months.
Results are delivered to providers and case managers at the time of the patient visit. Many
aspects of this process are tailored to each clinic.
Key themes
Integration into a large, national research network that integrates clinical and PRO
data
Developed quality improvement efforts around PRO collection to generate
institutional buy-in
Focused on elements of care that has a desirability bias in face-to-face
interactions: medication adherence, depression screening, drug use
Expanded beyond PROs to other patient-reported data (PRD)
Impressive expansion efforts into HIV clinics and community health centers
Sources: Interview and email communication (Heidi Crane); Crane 2011; Fredrickson
2012; Tufano 2010
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
41
SYNTHESIZING ACROSS THE CASE STUDIES
The systems profiled here represent a broad range of functionality, patient populations,
and applications involved in PRO collection and EHR integration. A synthesis of the
case studies aims to provide insight into the key components and considerations
associated with this intervention. Below, we summarize the systems’ features, outline
the key considerations related to the systems’ development and deployment, discuss
the themes that emerged from this review and identify barriers to and facilitators of PRO
integration in the EHR.
Summary of Systems
The case studies above cover a range of PRO applications, including health plans,
clinical efforts, and research applications. PRO collection efforts by three health care
plans — Kaiser Permanente Colorado (KPCO), Group Health, and Essentia Health —
illustrate a variety of applications of PRO collection at the plan level, including clinical
care, population-based screening, and quality-of-care evaluation efforts. Hospital-wide
PRO efforts at Cleveland Clinic and Cincinnati Children’s represent examples of the use
of PROs in specialty care and the expansion of PRO use in a hospital setting. Examples
of clinic-based, disease-specific PRO collection include UCLA/Michigan’s My GI Health
program (GI clinics), the University of Washington/CNICS (HIV), the Dartmouth Spine
Center (spine), the Patient Care Monitor at Duke University (cancer), and the University
of Pittsburgh Medical Center (primary care). Lastly, we have included Epic MyChart, a
patient portal that includes a PRO collection module, and that is an element in many of
the other systems described above. MyChart is similar with respect to the assessments
and design of the external collection tools described in this review.
Clinical care efforts primarily focus on providing information for a physician to use during
the patient visit, but some systems also include features that elicit information for follow-
up evaluation (e.g., Smart Set at KPCO, and Care Paths at Cleveland Clinic). Quality
improvement efforts were most commonly aimed at evaluating PRO collection, but
Essentia reports their PRO scores as part of a statewide (Minnesota) quality
improvement effort, including public reporting. Research efforts are generally limited but
are part of future expansion plans. A few sites (UCLA/Michigan, Cleveland Clinic,
Cincinnati Children’s, Duke) have integrated these systems into methodological and
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
42
clinical research. UW/CNICS have made integrated PRO and clinical (EHR) data
available to a nationwide HIV research network. And Cleveland Clinic includes data
from its PRO systems in its ongoing reports on cost-effectiveness research in multiple
disease areas.
Considerations of the Cases by System Features
This section presents a general summary of the system features across the different
systems profiled, supplementing the content included in the case studies. Topics
covered include (1) System Design/Implementation, (2) Measure Selection, (3)
Administration/Data Collection, (4) Reporting and Interpretation and (5) Analysis.
(1) System Design and Implementation
Concept and Purpose. For most of the systems examined, PRO data collection and
integration were intended to meet a specific clinically-oriented need by incorporating the
patient perspective into care. The origins of each system vary considerably; each
confronted different barriers and enabling factors in accomplishing PRO data collection
and integration. Some systems were developed based on other research-related
electronic data collection efforts, such as patient registries (UCLA) and clinical research
studies (UW). Others were developed explicitly for use in clinical practice (UPMC, Duke).
A third type of system design, most commonly at the health plan level, focused on
screening patients to target interventions (KPCO), engage patients (Group Health) or
monitor quality of care (MN). One barrier to integration identified was a lack of
knowledge by some developers about precisely how their system is designed to capture,
store and report PRO data, as well as the points at which it is integrated into the EHR.
This suggests a knowledge gap with respect to system design.
Expansion
. While all of the systems described here implement PRO data collection in
multiple settings (clinics or hospital-based practice sites), some started small (e.g., a
single clinic) and expanded over time. These systems often identified clinical
“champions” – early adopters already knowledgeable and interested in implementing
electronic PRO assessment in their clinic. Systems with wide-spread adoption reported
the development of best-practices and utilized multidisciplinary design teams to
streamline implementation. These teams evaluate current workflow, design the PRO
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
43
collection to conform to current clinic workflow and generate recommendations that can
range from PRO content to specific hardware needs. An unexpected, yet common,
barrier was implementation issues due to limited wireless internet access in a clinical
location. Most of the data collection systems collect PROs through a web-based
platform. Limited or intermittent connectivity presents a barrier to collecting and storing
the PRO data before linking them with the EHR.
(2) Measure Selection
Content. Although measure selection varies by system, all organizations focus on
collecting clinically-relevant PRO information. The range of content and measures
available was handled differently by each system. Some organizations allow clinicians
to select virtually any measure or individual question they want to include, others
establish a limited set of measures to be used across sites. Between these two options,
some systems present a compromise by letting providers select among a limited
number of “approved measures” within specific domains, or by allowing specific add-ons
to a core measure set. These different approaches to content selection provide different
trade-offs with respect to provider engagement, the ability to standardize measures
across clinics/sites, and the depth and specificity of PRO information available to
clinicians.
Modification & Implementation. Some systems focused on removing redundancy and
limiting PRO questionnaire length using skip patterns, screening based on disease
characteristics and automating measure selection based on patient characteristics. One
issue identified by several systems was the difficulty encountered in adapting legacy,
paper-based PRO measures for electronic use. Limitations in the visual design of
systems present barriers to maintaining fidelity when adapting PRO measures to an
electronic format. It was uncommon for electronic surveys to be skeuomorphic (to
resemble the paper forms). Another barrier was the limited evidence for the reliability
and validity of these new formats to ensure that they yield useful data for research.
Design Considerations
. The selection of measures and how they are modified and
administered require decisions that may limit the usefulness of information for both
research and practice applications. For example, a clinically-focused single item
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
44
measure may not provide sufficient measurement reliability to serve as an appropriate
outcome variable for a research study. Conversely, a lengthy, research-focused multi-
item scale may not be feasible for clinical use.
(3) Administration/Data Collection
EHR Integration. Two types of EHR integration were identified: (1) systems that fully
integrate PRO data into an EHR system (including collection within the EHR, such as
MyChart) and (2) systems that collect, store, and report PRO data independently, with
only the final scores integrated into the EHR. While fully integrating PRO data in the
EHR has definite advantages, benefits of the latter approach are that it allows PRO
collection across different EHR system software. However, this may limit integration to a
small number of data elements that can be easily integrated (e.g., summary PRO
scores), thus limiting functionality in the EHR itself.
De-linking PRO Assessment from the Patient Visit. Most of the case studies that use
PRO data in the clinical encounter allowed patents to fill out surveys online prior to a
visit, or at set intervals after the visit at a time most useful for evaluating recovery.
However, patient participation rates for long-term monitoring are currently low, ranging
between 20-30%. Some systems were more optimistic than others about how much that
number will increase as web-based collection becomes further integrated into care.
Greater integration of assessment opportunities outside of the patient visit offers
significant benefits to clinic workflow by reducing time spent during a clinical encounter.
Health plans with active patient portal systems suggested that patients, even those who
are older and may have more limited online experience, can complete assessments
prior to visits. For example, Group Health’s Health Risk Assessment has shown high
portal-based completion rates prior to an annual visit. KPCO seamlessly integrates
interactive voice response (IVR) and electronic capture, allowing for impressive
coverage of 78% (40% via the patient portal) for pre-visit screening of elderly patients.
(4) Reporting and Interpretation
Audience
. The audience for PRO information varied across different systems and
applications, including the patients themselves, clinicians, health care managers,
insurers and the public. Patient-level scores and change in scores are presented
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
45
primarily to the provider, whereas aggregated reporting is used for clinic-level evaluation
and quality improvement efforts.
Format Variations. The format, content, and mode of data collection (paper, electronic,
vs. both) and display can differ both between and within sites. For example, Dartmouth
employs multiple modes of administration to maximize completion rates. One repeated
theme was the current reporting limitations of PRO data displays (especially patient
displays) within the EHR. Even the data collected directly within an EHR may be stored
as an image file rather than discrete data within the record, which limits both provider
access and availability for secondary use.
Automation. The PRO assessment results may trigger a wide range of actions. The
most common alert identified in this review was the identification of a positive indicator
for suicidal ideation. When this occurs, clinicians are notified immediately. Other
examples of automation include the “SmartSet” developed at KPCO whereby a positive
depression screen triggers a series of evidence-based orders for testing, treatment, and
referral. PRO scores can also be used to tailor personalized educational interventions.
However, this type of integration was only identified in specialized care settings where
disease-specific PRO information could be used to direct patients to appropriate advice
or other information (e.g., GI symptom self-management). At UPMC, PRO measures
are being used by rehabilitation clinicians to provide information that argues against
payment denial for aspects of the clinical care that they deliver.
(5) Analysis
Clinical Value & Efficiency. Analyses conducted at several of the sites highlighted in this
review aimed to generate evidence of high-quality care (indicators of patient
management) and the benefits of PRO use in the clinical encounter. This includes
measures of patient and provider satisfaction (Cleveland Clinic), improved clinician
notes (My GI-Health), and patient self-reported medication adherence (UW). Quality
improvement efforts tended to center around improving the integration of PROs into the
workflow. In one case (KPCO), cost savings were calculated, illustrating some of the
benefits of PRO integration and automation.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
46
Major Themes
From the article review, qualitative interviews, and case-study synthesis, four major
themes emerge regarding the use of PROs in the EHR. These themes are (1) System
Customization, (2) Balancing Research and Practice Goals, (3) Demonstrating Value,
and (4) EHR Integration and Limitations.
Theme 1: System Customization
Clinical practice encompasses a broad range of settings, conditions, and patient
populations. Therefore, it is difficult to develop a single standard EHR that fits all health
care environments. Studies have shown that when non-customized, “off-the-shelf” EHR
systems have been adopted, the result has been problems with communication, missing
records, and decreased productivity (Maekawa 2006). A major theme that was
identified in the case studies was the need to customize systems to the settings in
which they were being used. We found that a great deal of customization appeared to
be necessary across all systems. Of particular importance is customizing the basic
software so that it is tailored to the workflows of the specific institution.
For maximum interoperability, an ideal EHR system would have a standardized
underlying record, plus interfaces that can be customized to each specific environment.
Having a standardized underlying record increases the ability to abstract and combine
data across systems whereas customization enables PRO collection to fit within the
clinical workflow, thereby facilitating clinician, staff, and patient buy-in. However, the
number of changes necessary even for “off the shelf” products suggests that
implementing PRO collection requires a high degree of expertise to be successful and
sustainable.
There is limited guidance and few examples to inform the implementation of electronic
PRO collection. The degree to which a system is customized has implications for the
accuracy and comparability of information when used for broader clinical, quality, or
research goals. Whether electronic PRO collection represents the expansion of a
current successful effort, a new user of an existing PRO collection system (e.g., Epic),
or a new PRO system developed de novo, systems need to consider the opportunity for
large scale PRO-EHR integration. Currently there is limited information on how
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
47
differences across systems can impact the quality and type of data reported. However,
it is likely that large differences, within or between systems, create barriers to scaling up
collection, producing PRO data reports and integrating PRO data for large-scale
evaluations.
There are other potential disadvantages to customization. There are up-front costs
needed to customize a standard system to a specific implementation. Health care
providers and the implementation team must invest a significant amount of time if
customization is to be responsive to local workflows and needs. Costs increase if
additional interfaces are needed.
Theme 2: Balancing Research and Practice Goals
Electronic PRO assessment facilitates opportunities to use data collected for a single
purpose in multiple different ways. While there were variations among systems in the
focus on research versus practice, all of the systems that we reviewed have considered
how collected PRO data can be extracted for secondary use for research, practice, and
quality improvement.
For nearly all of the systems described in the case studies, real-time use in clinical
practice was the guiding force to integrate PRO data into their EHR systems. (The
exceptions were Group Health, for which the delivery of personalized health-promoting
recommendations to ambulatory patients was of equal importance, and Minnesota
Community Measurement, for which quality measurement and quality improvement
were also central goals). For some of these systems, there is limited use of PRO
information for research efforts. One system (UPMC) explicitly stated that they did not
collect any un-actionable data, preferentially retaining only PRO measures that have
direct clinical relevance. A few systems focused on broad quality of care evaluation
efforts. MN Community Measurement and Essentia are one such example, in which
PRO scores were used to measure and report quality of depression care and guide pay-
for-performance. Additionally, a minority of systems, with UW/CNICS being one
example, indicated the use of PRO data in broader, population-based research efforts.
Balancing goals for research and practice requires numerous considerations all within
the limited timeframe of a clinical visit. Considerations include system design, patient
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
48
burden, staff buy-in and data standardization, all within the limited timeframe of a clinical
visit. Each of these areas requires attention for expansion to broader research use
beyond the clinical setting. Systems where population health was a high priority, such
as KPCO and Group Health, emphasized data collection prior to clinical visits. Both rely
on strong patient participation in their patient portals.
Theme 3: Demonstrating Value
Most systems identified stakeholder buy-in as both a barrier and a facilitator of PRO use
and integration. Key stakeholders identified in these interviews and the literature were
patients, clinicians and other staff and institutions (including employers and payers,
depending on the system). Ultimately, patient participation drives the collection of PRO
information and ability to use it in care. Methods for increasing patient involvement
include allowing patients access to their scores and what they mean (My GI Health,
Group Health) and linking to tailored patient education and interventions driven by PRO
scores (My GI Health, PCM, Group Health, KPCO). Including patients at every point
from questionnaire completion to reviewing results can provide a compelling opportunity
for patient engagement. However, systems collecting PROs within the EHR were
generally unable to generate patient-focused content and interpretation, instead relying
on clinicians to present and interpret PRO information for their patients.
Clinician engagement drives the incorporation by practitioners of PRO information into
patient visits. For systems that are free-standing and link into EHRs (e.g., My GI Health),
use of PRO information requires clinicians to take extra steps to access a separate,
non-integrated system. As this requires effort beyond accessing a patient’s EHR,
additional motivation is necessary. Systems take a number of different approaches to
facilitate clinician buy-in, from preliminary measure selection (Cincinnati Children’s), a
streamlined focus on clinically-relevant content (UPMC), and allowing clinicians to
control the reporting features (e.g., alerts) of the system (My GI-Health). Each strategy
is targeted to engage the clinician in this process. Several of the organizations (e.g.,
Dartmouth, UPMC) secured the initial involvement of clinical champions in each
program area. These early adopters were involved in many different aspects of system
implementation, including: identifying the PRO measures and/or domains of interest;
qualitative interviews to tailor PRO report content; playing a visible role in the initial roll-
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
49
out of the measures in practice; and undergoing the training necessary to understand
what scores mean and how they can be used in the clinical care setting.
Finally, most systems identified a need to demonstrate value to their own institution. All
systems require considerable staffing (e.g., IT professionals, clinic staff, providers) that
can be difficult to fund and sustain solely from research grants. A key element to
developing PRO assessment and ultimately gaining sustainability for many of these
projects is a focus on identifying and “proving” value. Value can be procured through a
reduction in staff hours, higher quality of care scores or receipt of incentive payments
and increased billing (e.g., automated referral, or other methods).
Theme 4: EHR Integration and Limitations
The case studies show the wide applicability and usefulness of PRO data. They include
a range of medical disciplines, age groups and locations (hospital, health plan, state,
and region). For each of these groups, specific considerations were identified to ensure
PRO collection was useful and relevant to specific clinical situations. However, scaling-
up PRO use raises new considerations related to action-item “ownership”. In other
words, how PRO information collected for a patient across different clinical encounters
is presented as actionable items to various providers matters. For example, ownership
would increase the likelihood that issues that are not commonly addressed in specialty
care (e.g., a patient’s mental health and social functioning) are in the patient’s record
and flagged for specialists. Conversely, primary care clinicians will be presented with
PRO information for a wide range of conditions and situations pulled from specialist
visits. PRO information may be available to the care provider. Further consideration is
needed on how the same scores can be used across providers with different training
and disciplines to inform the treatment of medically complex patients.
Most systems consider these issues based on their own needs and perspectives. Group
Health provides only “basic” PRO information about physical and mental health,
focusing on gathering other patient-reported information (smoking, diet, exercise) that
can inform preventive care and educational recommendations. Some systems
(Cincinnati Children’s and UPMC) consider this issue by evaluating and reducing patient
burden where possible. Methods to accomplish this include ensuring that patients with
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
50
multiple conditions are given a limited number of surveys: utilizing “aggressive” skip
patterns in an effort to tailor the experience to relevant content, or programming survey
assignment indexed against the last completed assessment, rather than last visit date.
This ensures surveys are not re-administered if a patient has multiple appointments
within a short window of time. More research is needed to consider conceptually how to
identify and assess complex patients and how this information can be used to inform
their care with the least amount of burden to the patient, provider, and system.
MOVING AHEAD
The individual case studies presented above combined with the synthesis of themes
across the case studies are instructive in considering how to promote the integration of
PRO data in EHRs more widely. One advantage of PRO methods is their ability to
capture the patient experience in a standardized, systematic way with established
reliability and validity (Acquadro 2003; NIH 2009). This is important for PRO measures
to be useful for clinical practice, for quality assessment and improvement and for
research. However, as described above, there is a diversity of approaches to integrating
PROs in the EHR, including measure selection, mode of administration, analysis,
results reporting and interpretation, data access and models of governance.
Standardization of EHR-based PRO measurement could provide benefits for both
research and quality assessment, in addition to use in the care of individual patients.
Scores on standard PRO measures could be pooled across different institutions for
various uses including comparative effectiveness research, or to compare the outcomes
obtained by different providers both within and across institutions. Comparing outcomes
across institutions could be particularly useful in calculating information about the value
of care.
To accomplish this kind of standardization, harmonizing efforts are needed on multiple
levels. Certainly, there should be coordination within individual organizations, which
should avoid using different PRO measures to capture the same concepts when
possible. This will need to be done thoughtfully given the trend for large health care
organizations to acquire other hospitals and practices. At the next level, standardization
efforts could focus on provider groups, integrated delivery systems, health plans, city-
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
51
wide or state-wide health information exchanges and regional health information
organizations. At each level, various degrees of standardization may be required to
ensure that PRO measures are collected, shared, and retrieved in a meaningful way.
The standardization efforts should ensure that similar PRO measures are available
across organizations, and the results are interpreted similarly. Professional societies
and organizations focused on specific disease conditions should also consider
standardizing methods. This has already been done with success by several national
and international organizations such as the American Urological Association which has
embraced the IPSS (International Prostate Symptom Score)(Cockett 1991; USLI 2012)
and the IIEF (International Index of Erectile Function)(Rosen 1997), the National Eye
Institute which developed and promoted the use of the National Eye Institute Visual
Functioning Questionnaire (NEI-VFQ) (Mangione 1998), and OMERACT (Outcome
Measures in Rheumatology) which strives to identify, endorse, and improve health
outcome measures (http://www.omeract.org/).
Market forces may result in a certain degree of standardization as well. The
consolidation of the market for large EHR systems is playing a role in consolidating
PRO measures. For example, the Epic MyChart includes several built-in PRO measures,
which are administered to patients online in the same manner for all installations of Epic.
However, even in this case, there is a proliferation of measures that are developed and
deployed using other third-party systems and integrated to different degrees within the
Epic EHR.
Conceptual System Architecture
A conceptual system architecture diagram may be useful to help understand and guide
standardization that will enable high-quality data and the ability to "measure" once
within EHRs to collect and store PRO data, and "cut" multiple times for different uses
across clinical, quality improvement and research applications (Figure 4).
Electronic data repositories of PRO measures are collected and stored in different
systems such as: tethered EHR-based patient portals (e.g., MyChart), standalone
patient portals (e.g., Microsoft HealthVault), standalone PRO web tools (e.g., Cleveland
Clinic Knowledge Program, PatientViewpoint, commercial data collection systems),
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
52
digitized data from local paper-based PRO data collection, Computerized Adaptive
Tests (CATs) (e.g., local Patient-Reported Outcomes Measurement Information System
(PROMIS)(Cella 2010)), and multi-center provider/payer PRO research data collection
systems (e.g., REDcap).
These various dispersed PRO repositories are currently not connected through health
information exchange initiatives. Health information exchanges collect, aggregate and
share specific clinical data across multiple different electronic health records/providers;
however, they generally do not include the data captured in EHR-based patient portals
(i.e., where PROs are collected) due to the lack of standardization and low prioritization
by providers. As a practical matter, health information exchanges do not collect data
from non-EHR repositories such as standalone web tools, research databases, and
population health datasets.
In an idealized system architecture, health information exchanges would develop
separate or embedded technical layers that collect, map, aggregate, store, retrieve and
analyze different sources of scattered PRO data for specific goals. In this framework,
data elements that are captured in patient portals can also be shared for other
research/clinical purposes. This general storage/analytic layer could be called the
“population health layer” which includes the capacity to handle PRO data in a
standardized way.
Unanswered Questions
Achieving the ideal system described above for PRO integration both within and across
EHRs will require both broad efforts for implementation and standardization, as well as
investigation of a number of unanswered questions (Table 2). These questions can
again be organized based on the different aspects of integration of PROs in EHRs.
Many of these gaps in knowledge could be addressed by targeted requests-for-
proposals from PCORI, ONC, or other funding agencies.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
53
Table 2: Knowledge Gaps in Integrating PROs in EHRs
Integration Aspect Question
Selection Can the same PRO measures be used for clinical care, research
and quality improvement?
How should organizations prioritize among PRO measures?
What is the place of proprietary PRO measures in EHRs?
Administration What standards should be recommended for PRO collection?
What metadata should be collected with PRO measures?
How can missing data be reduced?
Reporting What is the impact of standardizing interpretation?
Is there equivalence among different PRO language versions?
Analysis What risk adjustment is needed for PRO outcome measures?
What are the best strategies for handling missing PRO data?
Can a standard set of items be used to calibrate a series of
PROs?
Can automation accurately auto-populate clinical fields?
How much IT professional time is needed to generate reports?
How much IT professional time is needed to extract datasets ?
Access/Security When should PRO data be available in the clinical record?
(1) Selection
For all purposes, a PRO measure should meet basic standards of reliability, validity,
responsiveness, and interpretability of the measure in the intended population. A
number of guidelines have been published regarding these standards in the last few
years (Brundage 2012, Mokkink 2010, FDA 2009; Scientific Advisory Board MOT 2002;
EMA 2012; Snyder 2011). There are also catalogs of PRO measures, such as the
Patient-Reported Outcome & Quality of Life Instruments Database (PROQOLID), a
searchable, curated library of measures, with references and contacts (MAPI 2013).
Should the same measure be used for multiple purposes? It would be most efficient to
measure once and apply the same score for multiple purposes (Wu 2013). However,
there are differences between the ideal PRO measure for practice and for research. The
former might be parsimonious and directed towards actionable domains, while the latter
might maximize measurement precision to reduce needed sample sizes. Additional
questions include: Are existing PRO measures designed for research suitable for
clinical practice? How useful are generic measures in disease-specific studies?
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
54
How should health care organizations prioritize among PRO measures for
implementation? For clinical practice, a PRO measure should be acceptable to
clinicians, and its results should be interpretable and actionable (Snyder 2011). At the
start, selection in some clinical specialties may be easy, since there are a few measures
that are already widely used or endorsed by professional societies, such as the urology
examples mentioned previously. Aside from these, there is a lack of PRO measures
that provide good interpretation for clinical use. PRO measures that can be easily
misinterpreted (e.g., due to measurement error or poor face validity) could result in
confusion, and limited clinician buy-in. Therefore, these measures should be avoided.
Perhaps developing a standard group of PRO measures with add-ons for research and
quality improvement could address these concerns. This kind of rationalization would
also shorten the work queues of over-burdened IT professionals.
For research use, there is a greater diversity of PRO measures. Investigators may want
to use measures that are highly specific to a particular research question. It may be
most efficient for these data to be collected using an organization’s patient portal and
EHR. In multicenter clinical trials and observational studies, there may be a centralized
selection of PRO measures to be used. However, as some of these measures may
never have been used in clinical practice, to maximize safety, it would be desirable if the
results were only visible to investigators. For use as a quality improvement tool, a PRO
measure needs to be situated within a specific context (NQF 2012b) to create what the
National Quality Forum refers to as a PRO-PM (PRO Performance Measure).
What is the place of proprietary PRO measures in EHRs? A practical consideration is
whether permission and payment are required for the use of a selected PRO measure.
There is a charge for use of some of the most widely used generic PRO measures,
such as the SF-36 (Ware 1992), the Health Utilities Index (HUI-3) (Feeny 2002) and the
EQ-5D (EuroQol 1990). Health care organizations and EHR vendors are concerned
about delays to the implementation of their systems and several have elected to use
PROs that are available in the public domain, such as the PHQ-9 (Kroenke 2001) for
depression screening. Some publicly available versions are nearly identical to
proprietary tools, such as the RAND-36 (Hays 2001) or the VF-12 rather than the SF-
36v2 or the SF-12v2 (QualityMetric 2013 a,b). There is also a movement to practice
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
55
“copyleft”, in which a PRO measure is copyrighted and then made available for public
use free of charge (Heffan 1997).
(2) Administration
A major distinction can be drawn between PRO data collection using a tethered portal
(e.g., Epic MyChart) versus using an internet connected or freestanding device that
delivers data to the EHR (Kharrazi 2012).
What standards should be recommended for the collection of PRO measures?
To achieve acceptable data quality, it may be desirable to recommend standards for the
collection of PRO measures. This is analogous to instructions for obtaining a blood
sample for a lipid profile accompanied by the instruction or query if the patient was
fasting at the time of collection. In the case of PRO measures, instructions may be
general, such as “complete before the next clinic visit", or "complete without assistance".
What metadata should be collected in association with PRO measures? If it is likely that
the instruction for administration may not be followed (e.g., complete questionnaire
without assistance), an item can be added to capture whether the instructions were
followed, as is done at the Cleveland Clinic. This kind of metadata can aid with
interpretation (Pearcy 2008). For quality improvement, it can help to determine if the
data are eligible to be included in a performance measure. This information is
particularly helpful if data are to be aggregated and compared across different treatment
settings or organizations.
How important are missing data and what should be done about it? Missing data and
lack of follow-up are particularly problematic if the PRO data are to be used on a
population level to answer research questions. They are also important if the data are to
be used to determine the effectiveness of treatment for individual patients. To address
this problem, provision can be made to send patients reminders or to add supplemental
modes of data collection (e.g., Interactive Voice Response) of data collection. A related
problem is the phenomenon of “hidden” data, which represent events that occur but are
not recorded. An example of this is a dip in a patient’s mood that occurs between clinic
visits and is not captured. More work is needed to determine how modes of
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
56
administration affect data completion – e.g., the effect of different strategies allowing
missing responses, defaulting a response that patients can change if desired: including
fewer or more questions per screen, or allowing patients multiple log-ins per session? If
data are truly missing, analytic solutions are needed to handle those missing data.
(3) Reporting and Interpretation
Clinicians often complain that it is difficult for them to understand the meaning of PRO
scores. The display of PRO score results can be annotated to identify levels of
dysfunction or impairment that may signify the need for clinical intervention (Snyder
2010). This is analogous to the common practice within most EHRs to highlight
laboratory values that signify out of the normal range (e.g., a serum potassium of 6
would be highlighted in yellow). For a given population, it may be desirable to
standardize the cutoff score, such as which score on a pain scale represents a level of
distress that warrants action. For clinical purposes, it is important to designate what
cutoff indicates a “panic value” that would require immediate action. An example would
be a positive response to a PHQ-9 item indicating suicidal ideation, which would require
a timely response.
What is the impact of standardizing interpretation, e.g., cutoff scores? Although there
would be obvious benefits for researchers, some systems (eg., My GI health) offer a
very different perspective. They let providers set their own cutoffs with the idea that they
will adapt it to their needs over time. There are likely to be benefits to this approach,
such as clinician buy-in. However, research will be needed to see how it compares to
more standard methods of establishing statistical cutoffs.
Is there equivalence between different language versions of the same PRO measure?
US society is rapidly becoming more multicultural, and research studies have evolved to
include larger networks of sites, including international observational studies and clinical
trials. Therefore it is increasingly important to determine if the translated and culturally
adapted versions of a given PRO measure are equivalent to one another, and if data
from the two versions can be pooled (Herdman 1997, Wagner 1998). Research is
needed both to test this premise and to determine optimal ways to harmonize the
development of PRO measures in multiple languages.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
57
(4) Analysis
The analytic methods applied to PRO data will vary based on the purpose for which the
data are collected, as well as the specific questions being posed. For all purposes,
standardized scoring algorithms are available for most PRO measures. However,
beyond this, there are many practical and methodological questions to be answered
regarding the analysis of PRO data. For clinical purposes, estimation methods, such as
in the use of a PRO measure as a screening test, may be required. For research and
quality improvement, there is the following question:
What risk adjustment is needed for PROs used as measures of process or outcome?
For research and quality improvement, risk adjustment methods may be needed. As
noted above, it may be useful to have metadata elements such as location where
completed (at home vs. in-clinic), mode of administration (telephonic vs. electronic) or
completion with assistance (completed by surrogate vs. translated by interpreter vs.
completed by a family member) to incorporate into the analysis of data. These data
would help researchers generate evidence of the validity and responsiveness of PRO
measures under different conditions. Metadata can also help identify the unit of analysis
and contribute to risk adjustment.
What are the best strategies for handling missing data? Missing data on patients’ health
related quality of life are likely to be informative in studies of treatment effectiveness.
However, although there are a number of approaches in use, there is no consensus on
the best strategies for imputing or otherwise handling missing PRO data (Fairclough
2008, 2010). It is possible that the collection of metadata could help to estimate scores.
There are also novel modeling methods based on data mining and Bayesian network
modeling that might be considered. Research is needed in this area.
Can a standard set of items or measures be used to calibrate a series of PRO
measures? If this can be accomplished, then it would be possible for a PRO
questionnaire to be customized to specific uses, but for the results still to be pooled with
the results from other questionnaires to which scores have been calibrated in a known
manner. One NIH-funded effort, dubbed the “PROsetta Stone,” uses the PROMIS T-
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
58
score metric (a score of 50=the US mean, 10=1 SD) to compare PRO scores from
different questionnaires (Cella 2011, Choi 2012, Choi in review, Schalet in review).
Previous smaller efforts have identified how a similar method can convert scores
collected using one questionnaire into comparable scores for a different related
measure, to be compared or combined (Wu 2005; Chan 2011). If this is possible, it
might support the universal use of a set of common core items and scales with known
relationships to larger families of scales.
In addition, there are a number of practical questions to be addressed, such as to what
extent can automation be used to accurately auto-populate fields and generate reports?
How much IT professional and data manager/analyst time are needed to generate
reports and extract datasets?
(5) Access, Confidentiality, and Security
Data availability, confidentiality and security are growing concerns in society at large as
well as in health care. For clinical purposes, the data in an EHR are generally
accessible to all health care workers with an individual logon to the system, but only
permitted on a need-to-know basis for members of a patient’s care team. These rules
are enforced by the built-in capacity to monitor who views any data element. One
question to be addressed by health care organizations is:
When should PRO data be made available in the clinical record? The desired
accessibility to PRO data may vary depending on whether the data are used for clinical
care, research, or quality improvement. For both clinical care and research, there is an
option within some PRO systems to control whether to “release” a specific piece of
information from the patient portal to the EHR. For research, should a separate consent
procedure be required? For quality measurement, some PRO data (e.g., patient
satisfaction surveys) should only be available to individual providers on an aggregate
basis? Research is needed on the ethical and practical aspects of these issues.
Key Barriers and Enabling Factors
It is possible to summarize a number of modifiable as well as less-modifiable barriers to
incorporating PRO measures into EHRs from the perspective of the patient, the clinician,
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
59
and the health system. They include a wide range of concerns: PRO collection and
reporting; the clinical environment and engagement. Some of these concerns echo
themes that have been noted for decades (Deyo 1989). It is also possible to identify
potential enablers that could be implemented or developed to help overcome the more
modifiable barriers (Table 3).
From the patient perspective, there may be a lack of awareness or understanding of the
importance of the formal measurement of PROs. However, it is not difficult to convince
patients that it is important to communicate their experience and point of view to their
physician(s). The best ways to increase awareness have not been identified. However,
the concept of systematically asking patients to report their outcomes can be
empowering.
System promotion and branding was an unexpected enabler found in our review.
Systems that invested in a branding, through establishing a system name, easily
accessible online materials and even the occasional YouTube video, supported other
enabling factors and addressed engagement barriers. A memorable name helps
patients to remember and log-on, increases awareness for interested clinicians, and
provides advertising and promotional opportunities.
Health literacy can be a barrier to understanding and responding to questions in a PRO
measure. A partial solution is to assure that PROs are available at low reading levels.
A second approach is to make PRO data collection available through several different
channels, including by telephone. In the US, people whose first language is not English
may need PROs in their native language. This requires that translated and culturally
adapted versions of a PRO be available and tested for a wide range of languages.
Despite the increased availability of internet service not all individuals have access to
the internet and internet access may be most limited among disadvantaged populations.
To achieve greater participation in PRO data collection, multiple options for data
collection are needed including both electronic (e.g., web and cell-phone based) and
others. Studies are needed to demonstrate the equivalence of different modes of
administration (Kongsved 2007; Coons 2009).
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
60
Table 3: Barriers to Incorporating PROs in EHRs for care, research and quality
BARRIER MODIFABLE ENABLERS
Patient
Lack of awareness + Explanation, empowerment
Effective branding and marketing
Health literacy +/- Tailor PROs to lower reading level
Language barrier + PRO Translation/cultural adaptation
Lack of internet access +/- Multiple options for collection
Lack of engagement +/- Feedback results & recommendations
Communication training for clinicians
Active follow up
Time constraints +/- Computer adaptive tests
Improve capacity for branching logic
Clinician
Time constraints + Redesign work flow, use extenders
Design-in time savers
Financial incentives
Desire for single login + Full integration of PROs into EHR
Lack of familiarity + Just-in-time information support
Clinician education/training
Negative attitudes + Local clinical champion
Promote clinician innovation
Provide disease specific PROs
Institutional leadership support
Professional society support
Lack of self efficacy and
outcome expectancy
+ Recommend actions to take
Improve PRO integration in reports
Identify best practices in data display
System
Lack of IT workers +/- Training clinicians to program
Support IT workers from research
Select standard measures
Proprietary PRO tools + Negotiate large scale contracts
+ Use public domain PROs
Difficulty pooling data + Standardize cross site data collection
Promote calibration
Missing research data +/- Collect metadata
Develop data imputation strategies
Wi-Fi not universal + Support Wi-Fi infrastructure
Limited data exporting + Improve integrated software tools
Limited designer
understanding of IT
+ Work in multidisciplinary teams
Integration is > Epic + Development of additional systems
Informed consent Develop acceptable opt in/out strategy
Consent outside the clinical workflow
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
61
Many patients are not sufficiently engaged in their own health care, including signing up
for patient portals and completing PRO measures for clinical use. Research is needed
to demonstrate effective methods for engaging patients with PRO data collection.
Currently, promising methods include feeding back to patients their individual results
and providing actionable recommendations. Active follow-up methods may be needed
to achieve satisfactory rates of completion, especially over periods of time. Training
clinicians to discuss PRO results with patients is also associated with increase patient
enthusiasm for filling out PRO questionnaires. Although completing PRO
questionnaires takes time, patients who are convinced of their value will be willing to
invest greater amounts of time. PRO data may be completed more efficiently using
CAT (Reeve, 2007) and if systems incorporate branching logic to help individualize the
selection of instruments.
From the clinician perspective, any task that requires additional time will be met with
resistance. A key research question is how PRO data collection can be incorporated
into the clinical workflow of a busy practice. Potential solutions include delegating some
of the work involved to other providers and building in time savers such as the capacity
to auto-populate clinical notes with patient responses to PRO questions. Clinicians
have an aversion to logging in to multiple systems, a problem that can be solved by full
integration of PRO measures into EHRs. A different type of solution is to provide
incentives for using PRO data, such as pay-for-performance.
Lack of familiarity with PRO measures and what to do with them can be remedied by
providing education and training of practitioners, including during pre-clinical education
such as medical and nursing school. Especially in the short-term, just-in-time support,
analogous to other types of clinical decision support, may be the most efficient methods
of explaining the meaning of PRO scores and relevant interpretations in clinical care
settings. Clinician skepticism and other negative attitudes to PROs may be overcome
by the presence of a clinical champion in the clinician’s own discipline. These
individuals can demonstrate and promote the use PROs and facilitate active discussion
of their utility in clinical practice. The use of disease-specific PROs and reporting of
item-level patient responses may increase the salience of measurement to skeptical
specialists. Professional societies and institutional leaders can also help to establish
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
62
the importance of PRO data by encouraging and promoting their use. Lack of self-
efficacy and outcome expectancy can be addressed by recommendations of action to
take in response to the question “what can I do?” (Hughes 2012) or prescribed sets of
actions (MNCM). Improved PRO integration in EHR data displays, whereby PRO data
can be charted against other clinical data, can also increase understanding of patients’
progress over time and the relationship to treatment. Improved data report features can
also help clinicians to appreciate the PROs experienced by individuals and groups of
patients. Research is needed to design and test different approaches to gaining the
acceptance by clinician of PRO measures, as well as the necessary supporting
resources.
From the system perspective, perhaps the most obvious barrier to PRO integration is
that it competes for the time of IT professionals who are already busy with other
responsibilities. This represents a bottleneck in organizational workflow, particularly for
those that are in the process of adopting new EHR systems. One solution is to
decrease the demand by prioritizing a few standard PRO measures. An alternative
solution is to increase the supply of staff available to help with PRO integration. This
could include researchers and even clinician enthusiasts who can assume some of the
work of designing PRO workflows. Since many PROs will be used primarily for
research, it is also reasonable to train and support the time of additional IT
professionals using designated research funds. Larger organizations may choose to do
this centrally.
In the interests of avoiding hassles and expense, some systems and organizations have
chosen to avoid the selection of proprietary, albeit well-tested, PRO measures in favor
of those available for free in the public domain. An alternative strategy would be to
select a few proprietary measures and negotiate agreements for a large number of
users. Large health systems may be in the best position to do this.
Researchers, particularly those conducting multi-site studies, face the challenge of
pooling data collected at multiple sites using different collection methods. Ideally they
would like to have standardized methods of data collection across all of their different
sites. There is a compelling argument for standardization across the multiple phases of
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
63
PRO data use. An alternative solution is the calibration of different PRO measures that
share these core items, as noted in the previous section. Additional work is needed in
this promising area.
There are additional technical barriers related to the collection and handling of PRO
data. An unexpected barrier identified was problematic Wi-Fi coverage. Hospitals, for
example, may have limited network capabilities, reception, or only offer “guest” network
access that may have limited security protocols. Rooms themselves can limit reception
(e.g., basements or shielding). All these considerations can limit capturing PRO data,
especially when using hand-held, web-based collection. An obvious but costly solution
is to of increase availability of Wi-Fi. Another is the as provision of multiple modes of
data collection. Limits to the easy exporting of data to create analytic datasets is a
problem best addressed by EHR vendors. The finding of limited understanding of
health IT by PRO designers should not be so surprising given the complexities and
rapid developments in EHR design. An enabler noted in our cases is the presence of a
multidisciplinary team to help address local measurement, system and integration
knowledge gaps. A current issue is the observation that virtually all of the systems that
we found used or interfaced with the Epic system. Although other EHR vendors may
incorporate PRO integration to some degree, these efforts are relatively limited. The
current state of the market could potentially reduce innovation (Shaywitz 2012). There is
certainly the opportunity for other EHR vendors to develop competing systems.
A final category of barriers faced by organizations and systems is those related to
requirements for informed consent for research participation, and the practical aspects
of incorporating these requirements into the clinical work flow. We will not discuss
current discussions for need for consent for data that are collected for quality
improvement and those that are collected for research (Kass 2013; Faden 2013).
Potential solution include designing improved methods to identify patients willing and
unwilling to participate in research and moving consenting procedures outside of the
workflow entirely.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
64
CONCLUSION
The fields of PRO measurement, EHRs, and comparative effectiveness research have
converged to a space that is more patient centered and more electronically based than
ever before. There is the potential for patient portals connected to the EHRs to provide
a useful platform for computerized PRO measures. PROs are inherently different than
laboratory values and the other structured data elements already captured in the EHR.
The EHR is pivotal point at which all of these are data are integrated to be used for
different purposes. If system designer fail to plan PRO collection so that it
accommodates the needs of patients, if they employ PRO measures with inadequate
reliability and validity, if they are not mindful of clinical workflows, or the PRO data are
not presented clearly, then usefulness will be compromised.
This report suggests that current electronic PRO systems within EHRs vary significantly
in their focus and features. All of the systems consider key factors for sustainability and
expansion, as well as how system features shape current EHR integration capabilities.
The diversity in goals and design provides the opportunity for best practices to emerge
in the early stages of PRO implementation within the EHR. It is also evident that lack of
standards and standardization in itself poses a practical and financial barriers to large-
scale PRO collection and reporting. Recent development, from low cost computer
hardware to the expansion of PRO collection capabilities to tethered patient portals,
provide an opportune time to intervene and align various stakeholders to harmonize
PRO measures in the EHRs for clinical use and, more broadly, for comparative
effectiveness research. Coordinated research and educational efforts are needed to
ensure PRO integration is done in a way to ensure the accuracy and accessibility of
PRO data available in the EHR for all interested clinicians, researchers, and patients.
In the future, it will also be important to understand the features and mechanisms that
promote the systematic and useful capture of the patient perspective in care, quality
improvement and research. There is currently a lack of evidence regarding many
scientific and practical aspects of implementing PRO measures in EHRs. Many of these
gaps in knowledge could be narrowed by targeted requests-for-proposals from PCORI
and other funding agencies.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
65
REFERENCES
Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, Hess R, Miller DM,
Reeve BB, Santana MJ, Snyder CF. User’s guide to implementing patient-reported
outcomes assessment in clinical practice. 2011. Found at: www.isoqol.org accessed
October 10, 2013.
Abernethy AP, Herndon JE, Wheeler JL, Day JM, Hood L, Patwardhan M, Shaw H,
Lyerly HK. Feasibility and acceptability to patients of a longitudinal system for evaluating
cancer- related symptoms and quality of life: pilot study of an e/Tablet data-collection
system in academic oncology. J Pain Symptom Manage 2009 Jun;37(6):1027-38
PMID:19394793
Abernethy AP, Ahmad A, Zafar SY, Wheeler JL, Reese JB, Lyerly HK. Electronic
patient-reported data capture as a foundation of rapid learning cancer care. Med Care
2010 Jun;48(6 Suppl):S32-8.
Abernethy AP, Zafar SY, Uronis H, Wheeler JL, Coan A, Rowe K, Shelby RA, Fowler R,
Herndon JE 2nd. Validation of the Patient Care Monitor (Version 2.0): a review of
system assessment instrument for cancer patients. J Pain Symptom Manage 2010
Oct;40(4):545-58. Epub 2010 Jun 25. PubMed PMID: 20579839.
Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient's perspective into drug
development and communication: an ad hoc task force report of the patient-reported
outcomes (PRO) harmonization group meeting at the Food and Drug Administration.
Value Health 2001 Feb;6(5):522-31.
American College of Surgeons. Best Practice Guidelines: Optimal Preoperative
Assessment of the Geriatric Surgical Patient. 2011. Found at:
http://site.acsnsqip.org/wp-content/uploads/2011/12/ACS-NSQIP-AGS-Geriatric-2012-
Guidelines.pdf
Atreja A, Rizk M. Capturing patient reported outcomes and quality of life in routine
clinical practice: ready for prime time? Minerva Gastroenterol Dietol 2012 Mar;58(1):19-
24. PMID: 22419001
Bainbridge D, Seow H, Sussman J, Pond G, Martelli-Reid L, Herbert C, Evans W.
Multidisciplinary health care professionals' perceptions of the use and utility of a
symptom assessment system for oncology patients. J Oncol Pract 2011 Jan;7(1):19-23.
PubMed PMID: 21532805
Barr VJ, Robinson S, Marin-link B, Underhill L, Dotts A, Ravensdale D, Salivaras S.
The expanded chronic care model: an integration of concepts and strategies from
population health promotion and the chronic care model. Hosp Q 2010;7(1):73-82.
Basch E, Jia X, Heller G, Barz A, Sit L, Fruscione M, Appawu M, Iasonos A, Atkinson T,
Goldfarb S, et al. Adverse symptom event reporting by patients vs clinicians:
relationships with clinical outcomes. J Natl Cancer Inst 2009 Dec 2;101(23):1624-32.
PMCID:PMC2786917 PM:19920223
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
66
Basch E, Abernethy AP. Supporting clinical practice decisions with real-time patient-
reported outcomes. J.Clin.Oncol 2011 Mar 10;29(8):954-616.
Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in
oncology clinical practice. CA Cancer J Clin 2012;62(5):337-47.
Black N. Patient reported outcome measures could help transform healthcare. BMJ
2013 Jan 28;346:f167. doi: 10.1136/bmj.f167. PubMed PMID: 23358487.
Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider
communication with the electronic self-report assessment for cancer: A randomized trial.
J Clin Oncol 2011;29, 1029-35.
Bliven BD, Kaufman SE, Spertus JA. Electronic collection of health-related quality of life
data: validity, time benefits, and patient preference. Qual Life Res 2001;10(1):15-22
PM:11508472 .
Bottomley A, Aaronson NK; European Organisation for Research and Treatment of
Cancer. International perspective on health-related quality-of-life research in cancer
clinical trials: the European Organisation for Research and Treatment of Cancer
experience. J Clin Oncol 2007;25:5082-6.
Boyce MB, Browne JP. Does providing feedback on patient-reported outcomes to
healthcare professionals result in better outcomes for patients? A systematic review.
Qual Life Res 2013 Mar 17 [ePub ahead of print].
Brown SH, Lincoln MJ, Groen PJ, Kolodner RM. VistA--U.S. Department of
Veterans Affairs national-scale HIS. Int J Med Inform. 2003 Mar;69(2-3):135-56.
PubMed PMID: 12810119.
Browne J. Can we (should we!) use patient-reported outcome measures to compare the
quality of healthcare providers? Presentation at the 2009 Annual Meeting of the
International Society for Quality of Life Research, New Orleans, Louisiana, October 31,
2009.
Brundage M, Feldman-Stewart D, Bezjak A, Leis A, Degner, L, Fleming S, Tu D, Velji K,
Pater J. The value of quality of life information in a cancer treatment decision. ISOQOL
11th annual conference, San Francisco, 2005. Qual Life Res P350/1700, p2152.
Brundage M, Osoba D, Bezjak A, et al. Lessons learned in the assessment of health-
related quality of life: selected examples from the National Cancer Institute of Canada
Clinical Trials Group. J Clin Oncol 2007;25:5078-5081.
Brundage M, Blazeby J, Revicki D, et al. Patient-reported outcomes in randomized
clinical trials: development of ISOQOL reporting standards. Qual Life Res 2012 Sep 18;
[Epub ahead of print].
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
67
Cella D, Riley W, Stone A, et al. On behalf of the PROMIS Cooperative Group. Initial
item banks and first wave testing of the Patient–Reported Outcomes Measurement
Information System (PROMIS) network: 2005–2008. J Clin Epidemiol 2010;63:1179-94.
Cella D, Choi SW. PROsetta Stone: Project aims and results of preliminary analyses.
Presentation at PROMIS Psychometric Summit, July 2011, Washington, DC.
Cella D, Hahn EA, Jensen SE, Butt Z, Nowinski CJ, Rothrock N. Methodological Issues
In The Selection, Administration And Use Of Patient-Reported Outcomes In
Performance Measurement In Health Care Settings. Sept.28, 2012
Chan KS, Kasper JD, Brandt J, Pezzin LE. Measurement equivalence in ADL and IADL
difficulty across international surveys of aging: findings from the HRS, SHARE, and
ELSA. J Gerontol B Psychol Sci Soc Sci. 2012 Jan;67(1):121-32. PubMed PMID:
22156662
Choi SW, Podrabsky T, McKinney N, Schalet BD, Cook KF, Cella D. PROSetta Stone®
Analysis Report: a Rosetta Stone for Patient Reported Outcomes. Volume 1. Chicago,
IL: Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern
University 2012.
Choi SW, Schalet BD Cook, KF, Cella D. Establishing a common metric for depressive
symptoms: Linking BDI-II, CES-D, and PHQ-9 to PROMIS depression (in review).
Chow A, Mayer EK, Darzi AW, Athanasiou T. Patient-reported outcome measures: The
importance of patient satisfaction in surgery. J Surg 2009;146(3):435-443.
Cockett AT, Aso Y, Denis L, Khoury S, Barry M, Carlton CE, Coffey D, Fitzpatrick J,
Griffiths K, Hald T, et al. World Health Organization Consensus Committee
recommendations concerning the diagnosis of BPH. Prog Urol 1991 Dec;1(6):957-72.
Review. English, French. PubMed PMID: 1726946.
Cohen AN, Chinman MJ, Hamilton AB, Whelan F, Young AS. Using patient-facing
kiosks to support quality improvement at mental health clinics. Med Care. 2013
Mar;51(3 Suppl 1):S13-20. PubMed PMID:23407006
Conway DS, Miller DM, O'Brien RG, Cohen JA. Long term benefit of multiple sclerosis
treatment: an investigation using a novel data collection technique. Mult Scler 2012
Nov;18(11):1617-24 Epub 2012 May 31.PMID: 22653659
Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, Lenderking WR,
Cella D, Basch E. Recommendations on evidence needed to support measurement
equivalence between electronic and paper-based patient-reported outcome (PRO)
measures: ISPOR ePRO Good Research Practices Task Force report. Value Health
2009 Jun;12(4):419-29.
Crane H, Fredericksen R, Feldman B, et al. The effect of pre-visit reports to providers of
patient reported outcomes (PROs) in routine clinical care. 6th International Conference
on HIV Treatment and Prevention Adherence. Miami, Florida 2011.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
68
Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health- related
quality-of-life assessments and patient-physician communication: a randomized
controlled trial. JAMA 2002;288(23):3027-34.
Devlin N, Appleby J. Getting the most out of PROMs: Putting health outcomes at the
heart of NHS decision making. London: The King’sFund; 2010. Available at:
http://www.kingsfund.org.uk/sites/files/kf/Getting-the-most-out-of-PROMs-Nancy-Devlin-
John-Appleby-Kings-Fund-March-2010.pdf. Accessed Oct 10 2012.
Deyo RA, Patrick DL. Barriers to the use of health status measures in clinical
investigation, patient care, and policy research. Med Care 1989;27(3)Supp:S254- S268.
Dudgeon D, King S, Howell D, Green E, Gilbert J, Hughes E, Lalonde B, Angus H,
Sawka C. Cancer Care Ontario's experience with implementation of routine physical
and psychological symptom distress screening. Psychooncology 2012 Apr;21(4):357-64.
Engelen V, Haverman L, Koopman H, Schouten-van Meeteren N, Meijer-van den Bergh
E, Vrijmoet-Wiersma J, van Dijk EM, Last B, Detmar S, Grootenhuis M. Development
and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in
clinical paediatric oncology practice. Patient Educ Couns. 2010 Nov;81(2):235-44.
PubMed PMID: 20189747.
Espallargues M, Valderas JM, Alonso J. Provision of feedback on perceived health
status to health care professionals: a systematic review of its impact. Med Care
2000;38:175-186.
European Medicines Agency (EMA). Reflection paper on the regulatory guidance for
use of health-related quality of life (HRQOL) measures in the evaluation of medicinal
products; July 25, 2005. Available at:
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/
WC500003637.pdf. Accessed Oct 10 2012.
EuroQol Group. EuroQol--a new facility for the measurement of health-related quality of
life. Health Policy 1990 Dec;16(3):199-208. PubMed PMID: 10109801.
Faden R, Kass N, Whicher D, Stewart W, Tunis S. Ethics and informed consent for
comparative effectiveness research with prospective electronic clinical data. Med Care
2013 Aug;51(8 Suppl 3):S53-7. PubMed PMID: 23793051.
Fairclough DL, Thijs H, Huang IC, Finnern HW, Wu AW. Handling missing quality of life
data in HIV clinical trials: what is practical? Qual Life Res 2008 Feb;17(1):61-73. Epub
2007 Dec 11. PubMed PMID: 18071926.
Fairclough DL. Design and Analysis of Quality of Life Studies in Clinical Trials, Second
Edition. Chapman & Hall/CRC Interdisciplinary Statistics 2010.
Feeny D, Furlong W, Torrance GW, et al. Multiattribute and single-attribute utility
functions for the Health Utilities Index mark 3 system. Med Care 2002;40(2):113-28.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
69
Food and Drug Administration (US). Guidance for Industry. Patient Reported Outcome
Measures: Use in Medical Product Development to Support Labeling Claims. Fed
Regist 2009;74(35):65132-65133.
Food and Drug Administration (US). Guidance for Industry: Qualification process for
drug development tools; October 2010. Available at:
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guida
nces/UCM230597.pdf. Accessed Oct 10 2013.
Fredericksen R, Crane P, Tufano J, et al. Integrating a web-based patient assessment
into primary care for HIV-infected adults. J AIDS HIV Res In press. 2012.
Ghogawala Z, Martin B, Benzel EC, Dziura J, Magge SN, Abbed KM, Bisson EF,
Shahid J, Coumans JV, Choudhri TF, Steinmetz MP, Krishnaney AA, King JT Jr,
Butler WE, Barker FG 2nd, Heary RF. Comparative effectiveness of ventral vs
dorsal surgery for cervical spondylotic myelopathy. Neurosurgery 2011 Mar;68(3):622-
30; PubMed PMID: 21164373.
Gilbert JE, Howell D, King S, Sawka C, Hughes E, Angus H, Dudgeon D. Quality
improvement in cancer symptom assessment and control: the Provincial Palliative Care
Integration Project (PPCIP). J Pain Symptom Manage 2012 Apr;43(4):663-78.
Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work
and why? Qual Life Res 2009;18:115-23.
Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of
health in routine practice in improving the process and outcomes of patient care: A
literature review. J Eval Clin Pract 1999;5:401-406.
Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in
routine clinical practice: Lack of impact or lack of theory? Soc Sci Med 2005;60:833-843.
Grosso MJ, Hwang R, Krishnaney AA, Mroz TE, Benzel EC, Steinmetz MP.
Complications and Outcomes for Surgical Approaches to Cervical Kyphosis. J
SpinalDisord Tech. 2013 May 31 [Epub ahead of print] PubMed PMID: 23732179.
Gurland BP. Alves-Ferreira. Using Technology to Improve Data Capture and Integration
of Patient-Reported Outcomes Into Clinical Care: Pilot Results in a Busy Colorectal Unit.
Dis Colon Rectum 2010 Aug;53(8):1168-75. PMID: 20628281
Gustafson DH, Hawkins RP, Boberg EW, McTavish F, Owens B, Wise M, Berhe H,
Pingree S. CHESS: ten years of research and development in consumer health
informatics for broad populations, including the underserved. Stud Health Technol
Inform 2001;84(Pt 2):1459-563. PubMed PMID: 11604968.
Gustafson DH, Hawkins RP, Boberg EW, McTavish F, Owens B, Wise M, Berhe H,
Pingree S. CHESS: ten years of research and development in consumer health
informatics for broad populations, including the underserved. Stud Health Technol
Inform 2001;84(Pt 2):1459-563. PubMed PMID: 11604968.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
70
Gutteling JJ, Darlington AS, Janssen HL, Duivenvoorden HJ, Busschbach JJ, de Man
RA. Effectiveness of health-related quality of life measurement in clinical practice: a
prospective, randomized controlled trial in patients with chronic liver disease and their
physicians. Qual Life Res 2008;17:195-205.
Hays RD, Morales LS. The RAND-36 measure of health-related quality of life. Ann Med
2001;33:350-7.
Haywood K, Marshall S, Fitzpatrick R: Patient participation in the consultation process:
A structured review of intervention strategies. Patient Educ Couns 2006;63:12-23.
Heffan IV. Copyleft: Licensing collaborative works in the digital age. Stanford Law Rev
1997;49(6):1487-521.
Herdman M, Fox-Rushby J, Badia X. 'Equivalence' and the translation and adaptation of
health-related quality of life questionnaires. Qual Life Res 1997 Apr;6(3):237-47.
PubMed PMID: 9226981.
How’s Your Health? http://www.howsyourhealth.org/
Hsiao CJ, Hing E, Socey TC, Cai B. Electronic health record systems and intent to
apply for meaningful use incentives among office-based physician practices: United
States, 2001- 2011. NCHS.Data Brief 2011 Nov;(79):1-8.
Hughes EF, Wu AW, Carducci MA, Snyder CF. What Can I Do? Recommendations for
Responding to Issues Identified by Patient-Reported Outcomes Assessments Used in
Clinical Practice. J Support Oncol 2012;10:143-148. PMC3384764.
Hvitfeldt H, Carli C, Nelson EC, Mortenson D, Ruppert B, Lindblad S. Feed forward
systems for patient participation and provider support. Qual Manag Health Care
2009;18(4):247-256.
Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st
Century. Washington, DC: The National Academies Press, 2001.
Institute of Medicine. Committee on Comparative Effectiveness Research Prioritization.
Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The
National Academies Press, 2009.
Institute of Medicine. Learning Healthcare Best Care at Lower Cost: The Path to
Continuously Learning Health Care in America. Washington, DC: The National
Academies Press, 2012
Jensen RE, Snyder CF, Abernethy AP, Basch E, Reeve BB, Roberts A, Loeffler D,
Potosky AL. A Review of Electronic Patient Reported Outcomes Systems used in
Cancer Clinical Care. J Oncol Pract, In press.
Jehi L, Tesar G, Obuchowski N, Novak E, Najm I. Quality of life in 1931 adult patients
with epilepsy: seizures do not tell the whole story. Epilepsy Behav 2011 Dec;22(4):723-
7 PMID: 22019018.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
71
Kass NE, Faden RR, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. The
research-treatment distinction: a problematic approach for determining which activities
should have ethical oversight. Hastings Cent Rep 2013 Jan-Feb;Spec No:S4-S15.
PubMed PMID: 23315895.
Katz S, Downs TD, Cash HR, et al. Progress in development of the index of ADL.
Gerontologist 1970;10(1):20-30.
Katz S ed. The Portugal conference: measuring quality of life and functional status in
clinical and epidemiological research. J Chron Dis 1987;40:459.
Katzan IL, Rudick RA. Time to integrate clinical and research informatics. Sci Transl
Med 2012 Nov 28;4(162):162fs41. PubMed PMID: 23197569.
Katzan IL. Improvement in stroke performance measures: are we moving forward or in
circles? Circ Cardiovasc Qual Outcomes 2011 Sep;4(5):493-5.. PubMed PMID:
21934079.
Katzan I, M. Speck, C. Dopler, J. Urchek, K. Bielawski, C. Dunphy, L. Jehi, C. Bae, A.
Parchman. The Knowledge Program: an Innovative, Unique Comprehensive Electronic
Data Capture System and Warehouse. AMIA Annu Symp Proc 2011:683–692. PMID:
22195124
Kharrazi H, Chisholm R, VanNasdale D, et. al. Mobile personal health records: an
evaluation of features and functionality. Int J Med Inform 2012;81(9):579-93.
Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and
completeness of questionnaires: a randomized study of Internet versus paper-and-
pencil versions. J Med Internet.Res 2007;9(3):e25. PMID:17942387
Kroenke K, Spitzer RL.The PHQ-9: Validity of a brief depression severity measure. J
Gen Intern Med 2001;16(9):606–13.
Lauer MS and Collins F. Using science to improve the nation’s health system NIH’s
commitment to comparative effectiveness research. JAMA 2010 June 2; 303(21).
Likumahuwa S, Song H, Singal R, Weir RC, Crane H, Muench J, Sim SC, DeVoe JE.
Building research infrastructure in community health centers: a Community Health
Applied Research Network (CHARN) report. J Am Board Fam Med 2013 Sep-
Oct;26(5):579-87. PubMed PMID: 24004710.
Lipscomb J, Gotay CC, Snyder C (eds). Outcomes Assessment in Cancer: Measures,
Methods, and Applications Cambridge: Cambridge University Press, 2005.
Lohr KN, Ware JE Jr. eds. Proceedings of the advacnces in health assessment
conference. J Chronic Dise 1987;40(suppl 1):1.
Lohr KN. Advances in health status assessment. Overview of the conference. Med Care
1989 Mar;27(3 Suppl):S1-11. PubMed PMID: 2921881.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
72
Lohr KN. Applications of health status assessment measures in clinical practice.
Overview of the third conference on advances in health status assessment. Med Care
1992 May;30(5 Suppl):MS1-14. PubMed PMID: 1583924.
Lyndon GM, Dowrick CF, McBride A, Burgess HJ, Howe AC, Clarke PD, Maisey SP,
Kendrick T. Questionnaire severity measure of depression: a threat to the doctor-patient
relationship? Br J Gen Pract 2011;61(583):117-23.
Maekawa Y, Majima Y. Issues to be improved after introduction of a non-customized
Electronic Medical Record system (EMR) in a Private General Hospital and efforts
toward improvement. Stud Health Technol Inform 2006;122:919-20. PubMed PMID:
17102464
Mangione CM, Lee PP, Pitts J, Gutierrez P, Berry S, Hays RD. Psychometric properties
of the National Eye Institute Visual Function Questionnaire (NEI-VFQ). NEI-VFQ Field
Test Investigators. Arch Ophthalmol 1998 Nov;116(11):1496-504. PubMed PMID:
9823352.
MAPI Research Trust. Patient Reported Outcome and Quality of Life Instruments
Database (PROQOLID). http://www.proqolid.org. Accessed Oct 2012.
Marshall S, Haywood K, Fitzpatrick R: Impact of patient-reported outcome measures on
routine practice: A structured review. J Eval Clin Pract 2006;12:559-568.
Meyer KB, Espindle DM, DeGiacomo JM, Jenuleson CS, Kurtin PS, Davies AR.
Monitoring dialysis patients' health status. Am J Kidney Dis 1994 Aug;24(2):267-79.
PubMed PMID: 8048434.
Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international
consensus on taxonomy, terminology, and definitions of measurement properties for
health-related patient-reported outcomes. J Clin Epidemiol 2010 Jul; 63(7):737-45.
MN Community Measurement. Found at: http://mncm.org/
Minnesota HealthScores. Found at: http://MNhealthscores.org/
MN Community Measurement Provider resources and tools. Found at:
http://mncm.org/submitting-data/provider-tools/
The National Academies. Recommended Social and Behavioral Domains and
Measures for Electronic Health Records. September 24-25, 2013, Washington, DC.
Available at:
https://www8.nationalacademies.org/cp/meetingview.aspx?MeetingID=6856&MeetingN
o=1.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
73
National Institutes of Health. The Patient Reported Outcomes Measurement Information
System (PROMIS): A walk through the first four years. PROMIS History. January 12,
2009. Available at:
http://www.nihpromis.org/Documents/PROMIS_The_First_Four_Years.pdf?AspxAutoDe
tectCookieSupport=1. Accessed Oct 2012.
National Institute of Health (US), National Center for Research Resources. Electronic
Health Records overview. The MITRE Corporation. [NIH website] April 2006. Available
at:
http://www.himss.org/content/files/Code%20180%20MITRE%20Key%20Components%
20of%20an%20EHR.pdf. Accessed Oct 2012.
National Quality Forum - Patient-Reported Outcomes in Performance Measurement.
December 2012. Available at:
http://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=72537
National Health Service (NHS) Provisional Monthly Patient Reported Outcome
Measures (PROMs) in England - April 2012 to March 2013; August 2013 release
(http://www.hscic.gov.uk/catalogue/PUB11360).
Nelson EC, Landgraf JM, Hays RD, Wasson JH, Kirk JW. The functional status of
patients. How can it be measured in physicians' offices? Med Care 1990
Dec;28(12):1111-26. PubMed PMID: 2250496.
Nelson EC, Hvitfeldt H, Reid R, Grossman D, Lindblad S, Mastunduno MP, Weiss LT,
Fisher ES, Weinstein JN. Using patient reported information to improve health
outcomes and health care value. Technical Report. Dartmouth Institute for Health
Policy and Clinical Practice/ Lebanon, NH June 2012.
Office of National Coordinator (ONC) for Health IT. Meaningful Use. [Health IT website].
Available at: http://www.healthit.gov/policy-researchers-implementers/meaningful-use .
Access Oct 2012.
Olsen LA, Aisner D, and McGinnis MJ, editors, Roundtable on Evidence-Based
Medicine. The Learning Healthcare System: Workshop Summary. Washington, DC.
National Academies Press 2007.
OMERACT: Outcome Measures in Rheumatology. Found at: http://www.omeract.org/
Patrick DL, Deyo RA. Generic and disease-specific measures in assessing health status
and quality of life. Med Care 1989 Mar;27(3 Suppl):S217-32.
Pearcy R, Waldron D, O'Boyle C, MacDonagh R. Proxy assessment of quality of life in
patients with prostate cancer: how accurate are partners and urologists? J R Soc Med
2008 Mar;101(3):133-8.
QualityMetric SF36v2 Health Survey. Found at:
http://www.qualitymetric.com/WhatWeDo/SFHealthSurveys/SF36v2HealthSurvey/tabid/
185/Default.aspx.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
74
QualityMetric SF12v2 Health Survey. Found at:
http://www.qualitymetric.com/WhatWeDo/SFHealthSurveys/SF36v2HealthSurvey/tabid/
185/Default.aspx.
Reeve B, Hays R, Bjorner J, et al. Psychometric evaluation and calibration of health-
related quality of life items banks: Plans for the Patient-Reported Outcome
Measurement Information System (PROMIS). Med Care 2007;45(5):S22-S31.
Reid RJ, Coleman K, Johnson EA, Fishman PA, Hsu C, Soman MP, Trescott CE,
Erikson M, Larson EB. The Group Health medical home at year two: cost savings,
higher patient satisfaction, and less burnout for providers. Health Aff (Millwood) 2010
May;29(5):835-43. PubMed PMID: 20439869.
Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical
practice: an overview and practical examples. Qual Life Res 2009 Feb;18(1):125-36.
Rosen RC, Riley A, Wagner G, Osterloh IH, Kirkpatrick J, Mishra A. The international
index of erectile function (IIEF): a multidimensional scale for assessment of erectile
dysfunction. Urology 1997 Jun;49(6):822-30. PubMed PMID: 9187685.
Santana MJ, Feeny D, Weinkauf J, Nador R, Jackson K. et al. The use of patient-
reported outcomes becomes standard practice in the routine clinical care of lung-heart
transplant patients. Patient Relat Outcome Meas 2010;1:93-105.
Santana MJ, Feeny D, Johnson JA, et al. Assessing the use of health-related quality of
life measures in the routine clinical care of lung-transplant patients. Qual Life Res
2010;19:371-9.
Schalet BD, Cook KF, Choi SW, Cella D. Establishing a Common Metric for Self-
Reported Anxiety: Linking the MASQ, PANAS, and GAD-7 to PROMIS Anxiety (in
review).
Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status
and quality of life instruments: Attributes and review criteria. Qual Life Res
2002;11(3):193-205.
Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute
(PCORI): National priorities for research and initial research agenda. JAMA
2012;307(15):1583-4.
Shaywitz D. Epic challenge: What the emergence of an EMR giant means for the future
of healthcare innovation. [Forbes website] June 9, 2012. Available at:
http://www.forbes.com/sites/davidshaywitz/2012/06/09/epic-challenge-what-the-
emergence-of-an-emr-giant-means-for-the-future-of-healthcare-innovation/2/. Accessed
Oct 2013.
Shekelle PG, Tucker JS, Maglione M, Morton SC, Roth E, Chao B, Rhodes S, Wu Shin-
Yi, Newberry S. Health risk appraisals and Medicare. In Center for Medicare & Medicaid
Services (Ed.). Baltimore MD: Centers for Medicare & Medicaid Services, 2003.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
75
Snyder CF, Aaronson NK. Use of Patient-Reported Outcomes in Clinical Practice
[Comment]. The Lancet 2009;374:369-370.
Snyder CF, Jensen R, Courtin SO, Wu AW, Website for Outpatient QOL Assessment
Research Network. PatientViewpoint: a website for patient-reported outcomes
assessment. Qual Life Res 2009 Sep;18(7):793-800. PM:19544089
Snyder CF, Blackford AL, Brahmer JR, et al. Needs assessments can identify scores on
HRQOL questionnaires that represent problems for patients: An illustration with the
Supportive Care Needs Survey and the QLQ-C30. Qual Life Res 2010;19:837-45.
Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes
assessment in clinical practice: a review of the options and considerations. Qual Life
Res 2012 Oct;21(8):1305-14. Epub 2011 Nov 3. PubMed.
Snyder CF, Blackford AL, Wolff AC, et al. PatientViewpoint Scientific Advisory Board.
Feasibility and Value of PatientViewpoint: A Web System for Patient-Reported
Outcomes Assessment in Clinical Practice. Psycho-Oncology 2012 Apr 30 [Epub ahead
of print]. PMC3415606.
Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-Reported Outcomes: Putting the
Patient Perspective in Patient-Centered Outcomes Research. Med Care 2013;51(8
Suppl 3):S73-9. NIHMSID #491342.
Snyder C, Wu A. Patient-Reported Outcomes and Care Quality: Intervention?
Outcome? Adjuster? Presentation to Centers for Medicare and Medicaid Services
Measure Forum. January 17, 2013.
Su CS, Liu KT, Panjapornpon K, Andrews N, Foldvary-Schaefer N. Functional
outcomes in patients with rem-related obstructive sleep apnea treated with positive
airway pressure therapy. J Clin Sleep Med 2012;8(3):243-247. PMID: 22701379
Taenzer P, Buktz BD, Carkson LE, Speca M, DeGagne T, et al. Impact of computerized
quality of life screening on physician behavior and patient satisfaction in lung cancer
outpatients. Psycho-Oncology 2000;9(3):203-213.
Takeuchi EE, Keding A, Awad N, Hofmann U, Campbell LJ, Selby PJ, Brown JM,
Velikova G. Impact of patient-reported outcomes in oncology: a longitudinal analysis of
patient-physician communication. J Clin Oncol 2011;29(21):2910-7.
Tang PC, Ash JS, Bates DW, et al. Personal Health Records: Definitions, benefits, and
strategies for overcoming barriers to adoption. JAMIA 2006;13(2):121-6.
Tarlov AR, Ware JE Jr, Greenfield S, et al. The medical outcomes study. An application
of methods for monitoring the results of medical care. JAMA 1989 Aug 18; 262(7):925-
30. PubMed PMID: 2754793.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
76
Tufano JT, Fredericksen R, Schmidt S, et al. Evaluating integration of an HIV
medication adherence computer-assisted self-administered interview (CASI) with
routine patient care. Paper presented at: 5th International Conference on HIV Treatment
Adherence 2010; Miami, Florida.
Urological Surgeons of Long Island (USLI). International Prostate Symptom Score (I-
PSS) questionnaire. Available at: http://www.usli.net/uro/Forms/ipss.pdf. Accessed Oct
2013.
Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-
reported outcomes in clinical practice: a systematic review of the literature. Qual Life
Res 2008;17:179-193.
Varagunam M, Hutchings A, Neuburger J, Black N. Impact on hospital performance of
introducing routine patient reported outcome measures in surgery. J Health Serv Res
Policy 2013 Sep 26. [Epub ahead of print] PubMed PMID: 24072815.
Velikova G, Keding A, Harley C, Cocks K, Booth L, Smith AB, Wright P, Selby PJ,
Brown JM. Patients report improvements in continuity of care when quality of life
assessments are used routinely in oncology practice: secondary outcomes of a
randomised controlled trial. Eur J Cancer 2010 Sep;46(13):2381-8. PM:20570138.
Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, Selby PJ. Measuring
quality of life in routine oncology practice improves communication and patient well-
being: a randomized controlled trial. J Clin Oncol 2004 Feb 15;22(4):714-24.
PM:14966096.
Wagner AK, Gandek B, Aaronson NK, Acquadro C, Alonso J, Apolone G, Bullinger M,
Bjorner J, Fukuhara S, Kaasa S, Leplège A, Sullivan M, Wood-Dauphinee S, Ware JE
Jr. Cross-cultural comparisons of the content of SF-36 translations across 10 countries:
results from the IQOLA Project. International Quality of Life Assessment. J Clin
Epidemiol 1998 Nov;51(11):925-32. PubMed PMID: 9817109.
Ware JE, Sherbourne CD. The MOS 36-item Short-Form health survey (SF-36): I.
conceptual framework and item selection. Med Care 1992;30(6):473-83.
Wasson JH, Stukel TA, Weiss JE, Hays RD, Jette A. M., Nelson EC. A randomized trial
of the use of patient self-assessment data to improve community practices. Eff Clin
Pract 1999;2(1):1-10.
Weed LL. Medical records that guide and teach. N Engl J Med. 1968 Mar
14;278(11):593-600.
Weed LL. N Engl J Med. 1968 Mar 21;278(12):652-7 concl.
Weiner JP. Fowles JB. Chan KS. New paradigms for measuring clinical performance
using electronic health records. Int J Qual Health Care 2012;24(3):200-5.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
77
Weinstein J, Brown PW, Hanscom B, Walsh T, Nelson EC. Designing an ambulatory
clinical practice for outcomes improvement: From vision to reality - the spine center at
Dartmouth-Hitchcock, year one. Qual Manag Health Care 2000;8(2):1-20.
Weinstein J, Tosteson TD, Jon D, Lurie JD. Surgical vs. nonoperative treatment for
lumbar disk herniation: The spine patient outcomes research trial (SPORT): A
randomized trial. JAMA 2006;296:2441-2450.
Weinstein J, Lurie JD, Tosteson TD, Hans- com B, Tosteson AN, Blood EA, Hilibrand
AS , Herkowitz H, Cammisa FP, Albert TJ, Emery SE, Lenke LG, Abdu WA, Longley M,
Errico TJ, Hu SS. Surgical versus nonsurgical treatment for lumbar degenerative
spondylolisthesis. NEJM 2007;22(356):2257-2270.
"What is Meaningful Use? | Policy Researchers & Implementers | HealthIT.gov".
Healthit.hhs.gov. Found at: http://www.healthit.gov/policy-researchers-
implementers/meaningful-use .Retrieved 2013-10-23.
Wu AW, Cagney KA, St John PD. Health status assessment. Completing the clinical
database. J Gen Intern Med 1997 Apr;12(4):254-5. PubMed PMID: 9127232; PubMed
Central PMCID: PMC1497100.
Wu AW, Huang IC, Gifford AL, Spritzer KL, Bozzette SA, Hays RD. Creating a
crosswalk to estimate AIDS Clinical Trials Group quality of life scores in a nationally
representative sample of persons in care for HIV in the United States. HIV Clin Trials
2005 May-Jun;6(3):147-57. PubMed PMID: 16255084.
Wu AW, Snyder C. Getting ready for patient-reported outcomes measures (PROMs) in
clinical practice. Healthc Pap 2011;11(4):48-53.
Wu AW, Snyder C, Clancy CM, et al. Adding the patient perspective to comparative
effectiveness research. Health Aff (Millwood) 2010;29(10):1863-71.
Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice—adding
patient-reported outcome measures to the electronic health record for comparative
effectiveness research. J Clin Epidemiol 2013 Aug;66(8 Suppl):S12-20. PubMed PMID:
23849145
Zickuhr K, Smith A. Digital Differences. Pew Internet & American Life Project; 2012 Apr
13.
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
78
FIGURE 1: The convergence of patient-reported outcome (PRO) measures, electronic
health records (EHR) development, and comparative effectiveness research (CER)
(adapted from Wu AW, Kharrazi H, Boulware LE, Snyder CF. J Clin Epidemiol. 2013.)
HSM=Health Status Measurement
HRQOL=Health Related Quality of Life
PRO = Patient Reported Outcome
PROMIS = Patient-Reported Outcomes Measurement Information System
PCOR=Patient Centered Outcomes Research
PHR=Personal Health Record
CER=Comparative Effectiveness Research
EHR=Electronic Health Record
Circles indicate measurement of PROs
Triangles indicate the comparative effectiveness research field
Squares indicate the predominant forms of medical records
Electronic-ness
Patient-Centeredness
HSM
HRQOL
1980s
PRO, PROMIS
2001, 2012
CER
1990s
PCOR 2011
Medical
Record
1960s
PHR, Patient Portal 2012
EHR 2000
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
79
FIGURE 2: Model for Using Patient-Reported Outcomes in Quality Assessment and
Improvement (Snyder C, Wu A. Patient-Reported Outcomes and Care Quality:
Intervention? Outcome? Adjuster? Presentation to Centers for Medicare and Medicaid
Services Measure Forum. January 17, 2013.)
1. Assess PROs
3. Evaluate Care Quality
2. Use in Clinical
Practice
4. Inform Quality
Improvement
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
80
FIGURE 3: Idealized conceptual system architecture for an interoperable city/ state/
region/ national HIE-based PRO platform for multiple uses (adapted from Wu AW,
Kharrazi H, Boulware LE, Snyder CF. J Clin Epidemiol. 2013)
EHR=Electronic Health Record
PRO=Patient Reported Outcome
HIE=Health Information Exchange
CAT=Computerized Adaptive Test
PROMIS=PRO Measurement Information System
Orange: Typical HIE data sources
Dark green: Population health data sources
Light green: PRO data repositories
Light gray/dashed border: sample systems
Arrows to middle silo: PRO messaging standardization required.
PRO data PRO data PRO data
My Health-eVet
VistA
MS HealthVault
MyChart
Epic
PRO module
Population Health
Layer
PRO
data
EHR 1
EHR 2 EHR … EHR n
EHR-based
Patient Portal
PRO
data
EHR-based
Patient Portal
HIE
PRO
data
Standalone
Patient Portal
PatientViewpoint
Standalone
PRO Portal
PROMIS CATs
Local PRO
Research Repository
Extra Lab
Systems
Non-
customizable
Patient Portal
Extra Rx
Systems
Extra
Imaging
Systems
Public Health
Data
Claims
Data
Other Data
CER
Researcher
Advances in the Use of Patient Reported Outcome Measures in Electronic Health Records
81
ACKNOWLEDGMENTS
The authors would like to acknowledge the contributions and time from the following
individuals:
Amy Abernethy
Evi Alessandrini
Todd Anderson
Ethan Basch
Heidi Crane
Paul Crane
Lori Frank
Tina Frontera
Wendolyn Gozansky
Rachel Hess
Joyce Hines
Irene Katzan
Carolyn Kerrigan
Hadi Kharrazi
Ajit Krishnaney
Ian Kudel
Joy Lee
Paula Lozano
Eric Mayer
Leo Morales
Esi Morgan-DeWitt
Eugene Nelson
Elizabeth Pfoh
Rob Reid
Nancy Smider
Brennan Spiegel
Matt Stiefel
Mary Tinetti
Patrick Twomey
Alexander Young
Allen Zhang
