Longitudinal Implementation Strategy Tracking System

A man and woman look at a computer screen with a diagram on a white board behind them.
A man and woman look at a computer screen with a diagram on a white board behind them.

Longitudinal Implementation Strategy Tracking System

What Is LISTS?

The Longitudinal Implementation Strategy Tracking System (LISTS) offers a systematic method for documenting, specifying, and tracking the use of implementation strategies in a given project or study. It was developed collaboratively between researchers and practitioners to fill a research gap in systematically tracking implementation strategy use and modifications in research studies and other implementation efforts.

By accurately characterizing the dynamic use of implementation strategies and how they change over time, LISTS increases the precision and reliability of changes in implementation strategy use, and encourages the use of common data elements to support data analysis across sites and synthesis across studies. It includes three components: 1) a strategy assessment, 2) a data capture platform, and 3) a user guide.

LISTS enables users to bundle strategies together and visualize strategies through an interactive dashboard, and incorporates timeline follow-back procedures to support accurate, ongoing tracking of implementation strategies.

LISTS project dashboard.
Launch the LISTS online tool

Check out the instructions and resources below to learn what data elements you need to use the tool.

Why Use LISTS?

    • Data Collection: LISTS allows for the systematic collection of data at multiple points throughout the implementation process. This helps capture progress and changes over time.
    • Analysis Integration: By providing a structured framework, LISTS enables the integration of various data points into the user’s analysis. This deepens understanding of the effectiveness of different strategies and interventions.
    • Outcome Assessment: With the data collected, LISTS facilitates the assessment of implementation outcomes. It helps by identifying patterns, trends, and any deviations from expected results.
    • Informed Decision-Making: The insights gained from LISTS can guide decision-making. It allows users to determine whether to continue, modify, or halt a particular strategy by tracking the strategy’s performance.
    • Alignment with Goals: LISTS ensures that the strategies being implemented are aligned with the overall goals and objectives. It provides a measurable way to track progress and make necessary adjustments.

Tools and Procedures to Get Started

Users are encouraged to use the companion spreadsheet below to specify the data elements of each strategy before using the LISTS interactive online tool. Users will also need to sign up and create an account to use the tool. Accounts are free and easy to create.

Download the step-by-step user guide
View the interactive user guide
Download the companion spreadsheet

Recommended Procedures

1. Identify your team and who will be responsible for LISTS tracking and updating. These individuals should have direct knowledge of implementation activities. Example team members include:

    • Implementation scientist(s)
    • Investigators and co-investigators
    • Embedded physician scientists
    • Project/implementation coordinator(s)
    • Practice champions
2. Meet with your team at routine intervals

    • Weekly/monthly/quarterly depending on the nature of the study, centrality of modification/adaption, study design, phase of the study
3. Initially populate LISTS

    • Review the full list of ERIC discrete strategies (prompt)
    • Enter strategies into the Excel spreadsheet
    • Confirm LISTS elements (other team members, calendars, meeting notes)
    • Establish a point person for compiling/entering strategies into the online tool
4. Update LISTS using the timeline follow-back procedure

    • Set up regular check-ins with implementers and LISTS team members re: changes/new strategies/discontinuations
    • Review entered strategies to assess for changes routinely

Necessary Data Elements

Data elements needed before using the LISTS online tool:

    • Key study dates (e.g., grant start, project start, implementation go-live, other)
    • Implementation strategy name(s) and definition(s)
    • Strategy specifications
        • Actors
        • Action
        • Action target(s)
        • Dose
    • Dates when the strategy starts/stops
    • Implementation outcomes each strategy is targeting
Data elements recommended before using the LISTS online tool:

    • Implementation determinants the strategy is addressing (optional)
        • The Consolidated Framework for Implementation Research (CFIR)
        • Exploration, Preparation, Implementation, Sustainment (EPIS) Framework
        • Practical, Robust Implementation and Sustainability Model (PRISM)
        • Socio-Ecological Model (SEM)

LISTS Presentations

The LISTS Method: A Method for Specifying and Tracking Strategies Over Time

Mountain States Community-Engaged Dissemination and Implementation Science webinar on August 11, 2024, by James Merle, PhD.

In this webinar, Merle discussed the importance of implementation strategies, the “black box problem” in implementation science, and the development of the LISTS method to track and evaluate implementation strategies. He also provided a demonstration of the LISTS tool.

Using LISTS to Improve the Rigor of Implementation Research

Seminar presented on June 3, 2021, by J.D. Smith, PhD.

Watch this seminar recording to learn more about LISTS as a feasible approach for longitudinal tracking, and how it can facilitate synthesis across units and within projects and cross-project initiatives of similar interventions.

More on the Development of LISTS

An iterative process between implementation researchers and practitioners was used to develop, pilot test, and refine LISTS prior to evaluating its use in three trials. The LISTS method was used with research and practice teams for approximately 12 months and subsequently evaluated for its feasibility, acceptability, and usability using established instruments and novel questions developed specifically for the study. Initial evaluation of LISTS indicates that it is a feasible and acceptable method, with content validity, for characterizing and tracking the use of implementation strategies over time. The software was developed by the Research Application Design and Development team at Northwestern University.

LISTS aligns with established taxonomies for specifying and justifying implementation strategies, while simultaneously offering flexibility for users by enabling them to link their strategies to one of several determinant frameworks as well as a wide variety of implementation and service outcomes. Informed by the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS), LISTS also gives users the ability to track adaptations and modifications made to strategies over time. Potential future developments for LISTS include additional features, fewer data elements, and interoperability with alternative data entry platforms.

The development of LISTS was made possible by funding from NIH. ISCI is funded by an administrative supplement to the Third Coast Center for AIDS Research, an NIH-funded center (P30 AI117943).

Suggested LISTS Citation

Smith JD, Norton WE, Mitchell SA, Cronin C, Hassett MJ, Ridgeway JL, Garcia SF, Osarogiagbon RU, Dizon DS, Austin JD, Battestilli W, Richardson JE, Tesch NK, Cella D, Cheville AL, DiMartino LD, the IMPACT Consortium. The Longitudinal Implementation Strategy Tracking System (LISTS): Feasibility, Usability, and Pilot Testing of a Novel Method. Implementation Science Communications, 4:153, November 2023.

Related Articles

Examples of Applying LISTS

Ridgeway JL, Cheville AL, Fischer KJ, Tesch NK, Austin JD, Minteer SA, Pachman DR, Chlan LL, Ruddy KJ, Griffin JM. Tracking Activities and Adaptations in a Multi-site Stepped Wedge Pragmatic Trial of a Cancer Symptom Management Intervention. Contemporary Clinical Trials Communications, 38:101269, April 2024.

Smith JD, Merle JL, Webster KA, Cahue S, Penedo FJ, Garcia SF. Tracking Dynamic Changes in Implementation Strategies over Time within a Hybrid Type 2 Trial of an Electronic Patient-Reported Oncology Symptom and Needs Monitoring Program. Frontiers in Health Services, 2:983217, November 2022.

Articles Informing the Development and Use of LISTS

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The Updated Consolidated Framework for Implementation Research Based on User Feedback. Implementation Science, 17:75, October 2022.

Miller C, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: A Framework for Documenting Modifications to Implementation Strategies in Healthcare. Implementation Science, 16:36, April 2021.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A Refined Compilation of Implementation Strategies: Results from the Expert Recommendations for Implementing Change (ERIC) Project. Implementation Science, 10:21, February 2015.

Proctor EK, Powell BJ, McMillen JC. Implementation Strategies: Recommendations for Specifying and Reporting. Implementation Science, 8:139, December 2013.

Sobell LC, Sobell MB. Timeline Follow-Back. In: Litten RZ, Allen JP, editors. Measuring Alcohol Consumption: Psychosocial and Biochemical Methods. Totowa, NJ: Humana Press; 1992. p. 41–72.