About Us

This document describes how PCORI created our Engagement in Health Research Literature Explorer (Literature Explorer). The Literature Explorer is an online repository for peer-reviewed literature that describes or evaluates stakeholder engagement in health research. It provides detailed information about our PubMed/MEDLINE search strategy, inclusion/exclusion criteria, article tagging definitions, and process for creating and maintaining this resource. We welcome your comments and inquiries about the Literature Explorer at eval@pcori.org.

Contents


PubMed/MEDLINE Search and Supplemental Sources

To keep this collection current, we regularly conduct a literature search that spans various sources. One component includes a recurring search of the PubMed/MEDLINE database, hosted by the National Library of Medicine.

Table 1 lists the terms we include in our search string. We created the initial search using previous systematic reviews,1,2,3 as well as a content analysis of search results. We vet new terms for inclusion in the search string annually.

Recently, we changed our search process in PubMed/MEDLINE and are currently reviewing a large set of publications for inclusion in the Explorer. We expect to have an up-to-date collection of articles available in the repository in Spring 2021. If you are using the Explorer to conduct research, please connect with us at eval@pcori.org so that we can discuss how our efforts to curate literature on engagement in health research can support your research needs.

Table 1: Terms Included in PubMed/MEDLINE Search String

  Concept: Stakeholders Concept: Engagement Concept: Publication Type Concept: Research Other Terms
Medical Subject Headings (MeSH)^
  • Community participation[MeSH]
  • Patient participation[MeSH]
  • Stakeholder participation[MeSH]
  • Clinical trial[ptyp]
  • Comment[ptyp]
  • Comparative study[ptyp]
  • Editorial[ptyp]
  • Evaluation studies[ptyp]
  • Letter[ptyp]
  • Meta-analysis[ptyp]
  • Observational study[ptyp]
  • Pragmatic clinical trial[ptyp]
  • Randomized controlled trial[ptyp]
  • Systematic[sb]
  • Citizen science[MeSH]
  • Comparative effectiveness research[MeSH]
 
Key Words^^
  • Caregiver(s)
  • Clinician(s)
  • Community(ies)
  • Consumer(s)
  • Employer(s)
  • Insurance
  • Lay
  • Partner(s)
  • Patient(s)
  • Payer
  • Pharmaceutical
  • Policy makers
  • Provider(s)
  • Public Stakeholder(s)
  • Collaboration
  • Collaborative
  • Engage
  • Engaged
  • Engagement
  • Engages
  • Engaging
  • Input
  • Involvement
  • Participation
  • Partner(s)
  • Partnered
  • Partnership
  • Clinical control(led) trial
  • Clinical trial
  • Comparative study
  • Observational study
  • Pragmatic clinical trial
  • Pragmatic trial
  • Randomized clinical trial
  • RCT
  • Randomized control(led) trial
  • CER
  • Collaborative research
  • Community-academic partnership
  • Community-engaged research
  • Comparative effectiveness research
  • Comparative effectiveness trial
  • Engagement method*
  • Participatory action
  • Patient and public involvement
  • Research
  • Conceptual framework
  • Conceptual model
  • NOT preference*
  • Practical framework
  • Practical model
  • Public Involvement Impact Assessment Framework

Criteria for Publication Inclusion and Exclusion

We screen articles for inclusion in the literature explorer in two phases, using the abstracts (Table 2) and full text (Table 3). Articles sourced from PubMed/MEDLINE and the Journal of Patient-Reported Outcomes are first screened at the abstract level followed by a full-text review. Articles from all other sources go directly to a full-text review.

We will provide additional information regarding the number of articles screened at each phase upon request.

Table 2: Inclusion and Exclusion Criteria for the Title and Abstract Screening Phase

Inclusion Criteria

  • In English

Exclusion Criteria

  • Not a peer-reviewed article
  • Not about health research (e.g., agricultural health, occupational health, environmental health)
  • Evaluates or describes engagement in health care rather than engagement in research
  • Evaluates or describes patient preference on health care

 

Table 3: Inclusion and Exclusion Criteria for the Full-Text Review Phase

Inclusion Criteria

  • In English
  • Describes engagement in health research (e.g., in planning, conducting, or disseminating research; NOT limited exclusively to comparative effectiveness research [CER])
  • Includes information relevant to at least one of the five engagement topics: (1) examples of engagement in health research studies; (2) detailed descriptions of engagement in health research studies; (3) evidence for engagement; (4) editorials or commentaries; and (5) models or frameworks

Exclusion Criteria

  • Not a peer-reviewed article
  • Not about health research (e.g., agricultural, occupational health, environmental health)
  • Evaluates or describes engagement in health care rather than engagement in research
  • Evaluates or describes patient preference on health care
  • For engagement topics 1 and 2 only: No description of the effects of engagement in research (i.e., only describes anticipated engagement or only mentions the engagement of partners without describing how learnings from partners were incorporated, changes were made as result of engagement, or some other perceived impact of engagement)
  • Patients/stakeholders involved only in delivering or administering the intervention; no other forms of patient/stakeholder engagement described

Article Tagging

For articles that meet the inclusion criteria, we review the full texts and apply applicable tags across three engagement-related categories: engagement topic, stakeholder involvement and phase of research engagement.

Engagement Topic (not mutually exclusive)

  • Examples of Engagement in Health Research
    • Manuscripts with a primary objective of reporting on a health research study that engaged partners in at least one phase of the research and describe at least one impact of engagement on the research study
  • Detailed Description of Engagement in Health Research
    • Manuscripts with a primary objective of reporting a detailed description of the planning, conduct, and/or impacts of engagement within the context of a health research study or research agenda
  • Evidence for Engagement
    • Manuscripts that include a formal evaluation of engagement within the context of a health research study, or a study with the primary objective to evaluate or synthesize engagement methods/impacts in health research
  • Editorial or Commentary
    • Manuscripts that express a theoretical view on engagement in health research, including scientific commentaries or opinion briefs
  • Model or Framework
    • Manuscripts that introduce a new or revised conceptual framework or model to organize or contextualize concepts related to engagement in health research

Stakeholder Involvement (not mutually exclusive)

Stakeholder groups actively engaged in the research as described in the article, if applicable.

Table 4: Stakeholder Descriptions and Examples

Stakeholder Group

Description

Example

Clinicians

Providers of health care in a clinical setting, including physicians, nurses, physician assistants, rehabilitative professionals, pharmacists, mental healthcare providers, complementary and alternative healthcare providers, and members or representatives of professional societies that serve clinicians

Family physician in direct patient care in a small group practice, hospital floor nurse, psychologist in private practice, dentist, member of the American Medical Association or the American Nurses Association

Patients

Persons with current or past experience of illness or injury, persons at risk for a condition or who consume healthcare services, or members or representatives of advocacy organizations that represent patients

Person with type 1 diabetes, cancer survivor, person whose leg was amputated after a traumatic injury, person with bipolar disorder, patient council leader for a national heart disease advocacy organization, patient engagement specialist for a state-based newborn health coalition, executive director of a stroke patient advocacy foundation who also has personal experience with strokes

Caregivers

Family members or other unpaid persons who provide care to patients, or members or representatives of advocacy organizations that represent caregivers

Spouse of a patient receiving home hospice care for advanced amyotrophic lateral sclerosis (ALS), representative from the National Alliance for Caregiving

Hospitals/health Systems

Organizations where care is delivered, including public and private hospitals and health systems, urgent care centers, retail health clinics, and community health centers, as well as organizations that represent these facilities

Clinic administrator of a multispecialty physician practice, chief operating officer of a health system

Industry

Companies that design, invest in, or manufacture diagnostics, devices, pharmaceuticals, electronic records systems, and mobile apps, as well as organizations that represent the life sciences industry

Chief medical officer of a device manufacturer, representative from a pharmaceutical company, The American Medical Informatics Association

Purchasers

Companies that purchase health benefits for employees and their dependents, including individual businesses as well as local, state, regional, and national business groups; coalitions that represent businesses; and health coalitions

Owner of a family-run business; staff of a state chamber of commerce; health benefits manager of a large, ERISA-covered employer

Payers

Companies that function as financial intermediaries in the health system, including private insurers, public insurers, and organizations that represent insurers, such as America’s Health Insurance Plans

Chief medical officer of a health insurance cooperative; claims adjuster at a large insurer; policy analyst at an insurance trade association; insurers BlueCross BlueShield, Medicaid, and Medicare

Training institutions

Institutions that deliver health professional education, including public and private universities and colleges, individuals affiliated with the delivery or administration of health professional educations, and trade or professional associations that represent these institutions, organizations, and individuals

Dean of a nursing school, director of a residency program, manager of a continuing medical education provider

Policy makers

Persons who help craft public policy at any level of government, including federal, state, and local government officials; representatives of federal, state, and local units of government; and representatives from organizations that represent policy makers

State Medicaid director, representative of the Office of the National Coordinator for Health IT, policy analyst at the National Association of County and City Health Officials, National Conference of State Legislatures, National Association of Counties

Other

Other groups not covered by the categories above, such as community organizers, faith-based leaders, subject matter experts, or researchers who are not part of the primary research team

Community organizer for Planned Parenthood, faith leader for a place of worship, subject matter expert on patient engagement who is not affiliated with the research team

Phases of Research Engagement (not mutually exclusive)

of research in which stakeholders were engaged as described in the article, if applicable (Table 5).

Table 5: Phases of Research Engagement Descriptions and Examples

Phase of Research in Which Engagement Occurs

Description of Engagement Activities

Example of Engagement Activities

Research topic and agenda

Identifying, refining, and/or prioritizing topics and populations to study

Providing insight about the concerns and priorities of the patient population or stakeholder groups

Research questions and aims

Developing and/or refining the research questions and aims of the study

Reframing aims of the research study to highlight health issues within certain communities 

Study comparators and content

Selecting, designing, and/or refining interventions to be tested in the study

Altering the delivery mode of the intervention to be more aceptable to patients

Study outcomes

Selecting and/or refining study outcomes and deciding how to measure them

Selecting endpoints that are important to patients and other stakeholders 

Study design

Contributing to a specific plan or protocol for conducting the study, including which inclusion criteria should be used to select participants or helping select which methods will be used to conduct the research

Suggesting to use a study design that takes patients’ preferences into account instead of using randomization, in order to better appeal to the study population

Participant recruitment and retention

Refining approaches and/or materials for finding patients who are eligible to participate in the study, enrolling them into the study, and/or making it easier for them to participate and remain in the study

Rewording the consent form to include clearer descriptions of study activities

Data collection

Developing and/or refining approaches and methods to best collect data from study subjects

Improving the timing of data collection to better align with clinic flow

Results review and analysis

Analyzing collected data or reviewing study results to help interpret and explain research findings

Assisting with the transcription and coding of qualitative data, and/or interpreting preliminary findings

Translation

Applying study results to real-world practice, health condition management, policy, or future research

Contributing to the development of clinical practice guidelines

Dissemination

Planning, creating, and/or presenting materials that share information about the study or the results of the study

Suggesting the best means of returning results and other study findings to study participants, and/or giving presentations at conferences to spread results to a wider audience

Additional tags and other bibliographic data may be available upon request. Please contact us at eval@pcori.org to learn more.

Download this Document


^ National Library of Medicine’s controlled vocabulary thesaurus used for indexing articles for PubMed
^^ Key words for stakeholder and engagement concepts are used in combination in our search string. Please contact us at eval@PCORI.org to learn more.

In addition to articles identified via PubMed/MEDLINE searches, we screen articles for inclusion by reviewing all PCORI awardee articles with descriptions of engagement, highly relevant journals not yet indexed in PubMed (e.g., Research Involvement and Engagement, The Journal of Patient-Reported Outcomes), and other articles recommended by PCORI staff.

1 Concannon TW, Fuster M, Saunders T, et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. Journal of General Internal Medicine. 2014;29(12):1692-1701. doi:10.1007/s11606-014-2878-x.
2 Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: A systematic review. BMC Health Services Research. 2014;14:89. doi:10.1186/1472-6963-14-89.
3 Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015:4(2):133-45. doi: 10.2217/cer.14.79.


Posted: June 8, 2018; Updated: September 8, 2020

What's Happening at PCORI?

The Patient-Centered Outcomes Research Institute sends weekly emails about opportunities to apply for funding, newly funded research studies and engagement projects, results of our funded research, webinars, and other new information posted on our site.

Subscribe to PCORI Emails

Image

Hand pointing to email icon