 |
This Web site is a component of the SAMHSA Health Information Network. |
|
Evidence-Based Practices:
Shaping Mental Health Services Toward Recovery
Co-Occurring Disorders:
Integrated Dual Disorders Treatment
Using Fidelity Scales for Evidence-Based Practices
What is fidelity?
Fidelity refers to the degree of implementation of an evidence-based practice (EBP). A fidelity scale measures fidelity. Such scales have been developed for each of the six EBPs included in the Implementing EBP Project (assertive community treatment, supported employment, integrated treatment for dual disorders, illness management, family psychoeducation, and medication guidelines). Each scale assesses approximately 15 to 30 critical ingredients of the EBP, based on its underlying principles and methods. The scale items provide concrete indications that the practice is being implemented as intended. For example, one item on the Supported Employment Fidelity Scale concerns "rapid job search." This item is rated as fully implemented if the consumers in a program average one month or less between admission to the supported employment program and their first job interview.
Why measure fidelity?
Several assumptions underlie the use of fidelity scales. First, a fidelity scale should adequately sample the critical ingredients of the EBP to differentiate between programs that follow the practice and those that do not. Research suggests that fidelity scales for supported employment and for assertive community treatment do accomplish this. Second, fidelity scales should be sensitive enough to detect progress in the development of a program from the start-up phase to its mature development. There is some evidence that fidelity scales achieve this goal as well. Third, high-fidelity programs are expected to have greater effectiveness than low-fidelity programs in achieving desired consumer outcomes. Several studies comparing fidelity ratings to outcomes also support this assumption.
One key use of fidelity scales is for monitoring programs over the course of their development (and even after they are fully established). Experience by implementers suggests that routine use of fidelity scales provides an objective, structured way to give feedback about program development. This is an excellent method to diagnose program weaknesses and clarify strengths for providing positive feedback on program development. Fidelity scales also provide a comparative framework for evaluating statewide trends and outliers. The strategic use of repeated evaluations of programs using fidelity scales, either on an individual program or statewide level, is based on the general principle that whatever is paid attention to is more likely to be improved.
How are fidelity scales used?
In the Implementing EBP Project we have developed fidelity scales that are simple to understand. EBP items are rated on a 5-point response format, ranging from 1 equaling no implementation-to 5 equaling full implementation, with intermediate numbers representing progressively greater degrees of implementation. The response alternatives are behaviorally anchored, identifying measurable elements of the practice. Independent evaluators using multiple sources of information make the most valid ratings. Sources of information include interviews with staff, observation of team meetings, review of charts, and intervention observations. A daylong site visit is the optimal method for acquiring this information. Interviewers should be familiar with the EBP being rated. Although we recommend outside raters, fidelity scales can also be used by program managers to conduct self-ratings. The validity of self-ratings (or any ratings, for that matter) depends on the knowledge of the person making the ratings, access to accurate information pertaining to the ratings, and the objectivity of the ratings. We encourage the use of self-ratings, with appropriate caveats regarding potential biases that could be introduced by raters invested in seeing a program "look good" or who do not fully understand the principles of the EBP. In addition to the scales developed for independent evaluators and program managers, companion fidelity measures intended for consumers and family members are under development for some EBPs.
Graphing fidelity ratings
We recommend that programs implementing an EBP graph their fidelity ratings over time, using their total fidelity score. By graphing this score, a program can see its change over time. When the program shows greater fidelity over time, this serves to reinforce their efforts. Another feature of graphing fidelity is to examine the cut-off score for "full implementation." A program can use this score as a target and measure accordingly.
This document is part of an evidence-based practice implementation resource kit developed through a contract (no. 280-00-8049) from the Substance Abuse and Mental Health Services Administration's (SAMHSA) Center for Mental Health Services (CMHS) and a grant from The Robert Wood Johnson Foundation (RWJF), and support from the West Family Foundation. These materials are in draft form for use in a pilot study. No one may reproduce, reprint, or distribute this publication for a fee without specific authorization from SAMHSA.
Back to Co-Occurring Disorders:
Integrated Dual Disorders Treatment
|
|
