This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services Program for Children and Their FamiliesOutcomes: Linking Clients, Services, and Improved Mental HealthThis section discusses several types of outcomes supported by data from the graduating sites. These include functional and behavioral outcomes as measured by standardized assessment instruments, service utilization data generated by billing for services, and some cost data. As described in chapter III, each set of outcome data has limitations that warrant caution in interpreting results too definitively. For example, functional and behavioral outcomes are based on data from children, adolescents, and families who agreed to participate in a longitudinal outcomes study. However, the criteria for recruiting participants into the outcomes study sample were not the same in each site, and participation rates differed markedly. In several sites, the sample sizes were quite small relative to the total number of children, adolescents, and families who participated in the system of care. In one case, data were not collected for the same three 6-month intervals as in the other four sites. Only children who remained in services continued to participate in the evaluation across time and data completion rates at the 6-month and 1-year evaluation intervals were less than optimal. When outcome analyses were conducted for each program, demographic factors were compared for the sample completing all data points (e.g., intake and 6 months; intake, 6 months and 12 months). Samples participating in the outcomes were similar to the characteristics for the overall groups of children and families receiving services at each of the sites. As discussed below, the services and cost data emanate from management information systems (MIS) that are generally designed for efficient billing and tracking of traditional services. As a result, these systems, while useful, tend not to capture the many wraparound services that families cite as such important factors in improving their quality of life. Thus, the service and cost data should be viewed as promising but incomplete benchmarks for the potential achievements of systems of care. Despite these caveats, the findings described below do suggest that systems of care in the five graduating sites have had a measurable and positive impact on the children, adolescents, and families they serve. As with the national data presented previously, the overall trends support the direction in which systems of care have moved, but understanding the specific nuances of how and why these changes occur will require the more detailed data and analyses that comprise the national evaluation effort in the future. Relationships Among Outcomes As the demographic data about clients of these systems of care suggest, the interplay among client characteristics, services, and outcomes is complex and dynamic. Figure 34 illustrates a conceptualization of how child characteristics (i.e., client case mix) and services received (i.e., client service mix) influence outcomes. Aggregating data across all children and families obscures the fact that some subgroups make more dramatic gains while others may experience less favorable outcomes. Subgroups of children also experience the service system differently as evidenced in their utilization patterns. By disaggregating the samples, we can investigate rigorously which groups of children experience the most favorable outcomes, and describe their utilization patterns. Conversely, we can also begin to understand which children are not responding to system-of-care services and which factors, either case mix or service mix, are most important to address to improve outcomes. Figure 35 illustrates the relationship between client case mix and service mix. In this model, client case mix is measured in terms of demographic characteristics, clinical status, and pretreatment history. Service mix is assessed in terms such as service duration, services received and service intensity. The following sections present data and selected analyses from the graduating sites in each of these categories of outcomes. Functional and Behavioral Outcomes Standardized instruments were used to assess client behavior and functioning. These included:
Decreasing scores across time on all three instruments indicate improvement. For some of the subsequent analyses, data were truncated at 6 months due to limited numbers of participants at the 1-year evaluation point. Figures 36, 37, and 38. show changes in each score for all five graduating sites. The general trend documented in Figures 36–38 is one of overall improvement, with some regression or slowing of progress during the second 6-month period. The magnitude of changes across time varied for the sites, with small to medium effect size changes for the CAFAS and CBCL and small to large effect size changes for the YSR. However, even when the rate of progress slowed or was reversed, the scores after 1 year were still lower than at intake. It should be noted that the sample size for this analysis is considerably smaller than the total number of children receiving services in each system. The number of children for this analysis was constrained to those who had complete data at intake, 6 months, and 1 year. Possible explanations for variations in the rate of change are discussed in greater detail below. Service Outcomes The goal of systems of care is to provide a full array of services, spanning a continuum that includes the services listed in Table 13. The system-of-care philosophy emphasizes individualized services, which means that different and unique combinations and patterns of services will be available to children and families. Choices regarding when services are provided and in what combinations are highly dependent upon the structure of the system of care. Even within individual systems of care that serve large populations and span several counties or a whole State, the relative emphasis on specific service delivery components can vary widely. For example, some areas may emphasize traditional outpatient mental health services, increasing the frequency of this specific form of service delivery. Other areas may rely more heavily on home-based services, depending on their need, resources, capacity and treatment philosophy.
Regardless of the service array available to children and families, a comprehensive computerized management information system (MIS) is critical to understanding the use of individual services, combinations of services, and their costs. Without an adequate MIS, it is difficult to understand which services are utilized by which children and families and how individual children with serious emotional disturbance participate in the system of care. Compiling accurate information regarding the costs of services also rests upon the existence of an accurate and up-to-date MIS. Unfortunately, the technology for the development of MIS’s has lagged behind the development of systems of care in individual communities, resulting in a situation where it is difficult to comprehensively capture the available service mix. Interagency MIS’s are typically not available and combining individual MIS’s often results in information that is not clearly linked to individual children and families and is therefore difficult to interpret. Interpretability of information contained in MIS’s can also be heavily influenced by the extent to which direct versus indirect service information is coded. In many systems, indirect service information is not directly coded due to the fact that MIS’s are primarily used for fiscal purposes (e.g., billing, tracking contractual arrangements). This may create incentives to bill for particular services or at least aggregate services into particular categories so they can be reimbursed. Thus, MIS data may underrepresent services and costs in systems of care that offer, for example, indirect services provided by parents or paraprofessionals. Thus, MIS information must be supplemented by other information sources in order to describe service delivery patterns or analyze outcomes. Categories of Service Utilization Tracking and understanding the use of services delivered within systems of care is a prerequisite to understanding their impact on outcomes. Service utilization can be broken down into the following five variables:
Using these five variables allows for the description of the service mix provided to children in an individual system of care. For example, for the five sites highlighted in this report, descriptive information is presented regarding the types and extent of services provided. Despite the limitations associated with individual MIS’s, this information provides an overall picture of which mental health services are emphasized in system-of-care sites. Beyond basic descriptive information about the array of services available in individual systems of care, it is important to understand the relationship of services to one another. One approach to addressing this issue is to focus on the following combinations of factors that influence the intensity and extent of services provided to children:
Patterns of Service Use, Indicators at Entry, and Outcomes at Entry, and Outcomes in the Graduating Sites Despite the limitations of MIS’s and other data sources, each of the graduating sites offered at least some data that provided insights into which factors influence the intensity and extent of services provided to families and children. Available data are summarized in Table 14. Service Intensity In East Baltimore, service intensity was compared with baseline clinical outcome scores of children and adolescents entering the system of care. (Service intensity is defined as the ratio of service hours to the time interval over which services are provided, usually months.) The comparison showed that the children and adolescents with the highest CAFAS scores at intake received services more intensively than those with scores in the mild or moderate ranges (see Figure 39). Similarly, in Stark County, children who reported one or more child or family risk factors (such as a history of family violence, substance abuse problems, or a history of mental illness) received significantly more hours of service than children from families that did not report these factors. Children and adolescents who had run away from home, experienced drug and/or alcohol problems, or reported a history of sexual abuse also received significantly more services than those who did not report these risk factors (see Figure 40). At The Village Project in Charleston, children receiving 10 or more hours of services had more functional problems at intake than children receiving fewer hours of services. The investment in more intensive services paid off quickly, since these children made more progress than their counterparts with fewer problems at intake. After 6 months, the CAFAS scores of children who had initially high scores at intake were not significantly different from children who had fewer functional problems at intake and received fewer services (see Figure 41). These results indicate that service intensity is appropriately matched to children’s and families’ levels of problems when they enter systems of care. On a conceptual level, more intense services should be directed toward higher risk situations where more functional impairment is apparent at intake. A higher intensity level of services should also be associated with changes across time for more functionally impaired children, as noted by the results obtained from the Village Project. These analyses support the basic tenet of systems of care that individualized services should be provided based on children’s and families’ needs. Community VS. Gaps in Services In addition to service intensity, the continuity of services appears to be important. (Continuity of care is defined as care with no intervals between services that last longer than 30 days.) Several sites documented the effects of continuity of services on improved scores, as well as its corollary: breaks in service associated with less improvement in scores. In East Baltimore, children who received continuous care displayed more positive behavioral changes than children receiving discontinuous care after 6 months in services (see Figure 42). The same pattern was observed in Vermont. Children with no service breaks displayed more positive outcomes at 1 year than children with multiple service breaks. Children who had breaks in service improved less dramatically over the first 6 months, and some of those gains were reversed during the second 6 months (see Figures 43 and 44). In Charleston, two groups of children entered the system of care with identical CBCL scores. Six months later, children who had experienced no service breaks had made more progress in their behavioral outcomes, moving below the clinical cutoff for CBCL scores. The group that had at least one service break improved also, but did not experience as dramatic an improvement as the children who received continuous care (see Figure 45). Continuity of care is a strong value of service provision within systems of care. The above analyses suggest that continuity of care is associated with more positive outcomes for those who remain in services for up to 1 year. Continuity of care may reflect a more global dimension of engagement in services or active involvement with systems that has importance for individual child and family change. Specific Services In addition to service intensity and continuity, some sites reported improvements associated with a particular type of service. For example, in Stark County, children who received intensive in-home services showed greater reductions in functional impairment after 6 months in services, as measured by CAFAS scores, than children who did not receive these services (see Figure 46). As this example illustrates, linking service use to outcomes requires a clear understanding of how different demographic and functional indicators at entry influence the services that are used. Children differing in their presentation at entry may require different combinations of services. For example, those with higher levels of functional impairment may participate in a wider range and higher intensity of services. Furthermore, children and adolescents at varying ages may also differ in service use, partially due to the selective availability of services at different ages and also related to accessibility for younger versus older participants. It is important to understand how different groups of children are being served before attempting to understand the general impact of services on outcomes. As a preliminary step, differences in service delivery patterns by such demographic factors as gender, ethnic/racial status, family income, and age is important. In addition, differences in services provided based on children’s clinical characteristics at entry into the system of care are crucial. The relationship between intake characteristics such as the type of services delivered, amount of service, duration, continuity of care, and service combinations should be addressed. Once the impact of intake characteristics on service delivery patterns is understood, then the relationship between service use patterns and outcomes can be explored. As the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program continues to mature across time, it will be possible to follow up on the general directions suggested by the above analyses and to more intensively investigate the relationships between intake characteristics and services delivered as a preliminary step to evaluating outcomes. Satisfaction With Services Overall, families and caregivers gave the graduating sites high marks for the quality of the services provided to children and adolescents. Across the sites, at least 75 percent and as many as 92 percent of families indicated that the services they had received were either "excellent" or "good." In Charleston, those receiving 10 or more types of services reported higher overall ratings of the quality of services and also indicated that their ideas and opinions were more frequently considered (see Table 15). A similar pattern was reported for East Baltimore for overall ratings of services, although satisfaction with services was not different for those receiving a greater number of services (see Table 16). Satisfaction across individual types of services was relatively high in Stark County (see Figure 47). In Vermont, over 80 percent of parents either agreed or strongly agreed with the statement "I feel involved in my child’s treatment program" (see Figure 48). Some evidence for relationships between satisfaction and specific services received as well as outcomes was found. Significantly higher percentages of families who participated in care coordination and interagency staffings in The Village Project in Charleston reported that they always felt that their concerns were being heard (see Figure 49 ). In the East Baltimore Mental Health Partnership, parents reporting higher levels of satisfaction with services also reported greater decreases in behavior problems for their children from intake to 6 months (see Figure 50). Costs of Services The main barriers to obtaining and interpreting cost data are the absence of linkages between costs and services in the sites’ MIS’s and the different definitions of what constitutes a "cost." For example, some sites may record the "billable" charge for a particular service, while other sites report the actual amount that they are reimbursed. Still others may report the actual cost to provide the service, including overhead and administration. Cost data should be interpreted with these caveats in mind. Other differences exist across sites in terms of sample size and completeness of follow-up outcome data. For example, 1-year data were included in analyses whenever possible. However, in some sites 1-year data were available for a relatively small number of children. Hence, some analyses only used intake and 6-month data. For the Southern Consortium for Children in Ohio costs were compared for children who spent time in residential treatment settings with those who did not. The average cost for children who received residential treatment services was approximately $16,500—four times greater than the average cost for children in the system of care who did not participate in residential treatment (see Figure 51). In Charleston, children received fewer but more targeted services on a more regular basis during the program’s third year. As Figure 52 indicates, the cost of providing mental health services was reduced significantly, yet children in the program continued to improve over time. |
|
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
