Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services Program for Children and Their Families

Longitudinal Comparison Study

Current Progress in Longitudinal Comparison Study Sites

As discussed previously, the comparison study component of the national evaluation was initiated to address the question: "Can improvements in children’s behavior and functioning be attributed to the system-of-care approach compared to a traditional service delivery approach?" This question can be addressed by comparing three CMHS-funded system-of-care sites with three non-CMHS-funded sites that have made less progress toward adopting a system-of-care approach.

Selection of comparison study sites and subsequent data collection were initiated in the fall of 1997. The comparison study had interviewed a total of 373 children and caregivers through July 1998 as displayed in Table 8. Most of the interview data has been collected from the Stark County and Youngstown pair, which started earlier than the other four sites. Six-month data collection has begun in most sites as well, with 80 children and caregivers interviewed.

The following preliminary data focus on the demographic and clinical characteristics of the children and families participating in the comparison study at intake. Not enough information is available at the 6-month follow-up to begin to compare changes across time for the sites.

Description of Children in the Longitudinal Comparison Study

The data set for this analysis includes all 373 children enrolled in the comparison study prior to July 1, 1998. The purpose of analyzing this data set and presenting results is to conduct a general assessment of the population participating in this arm of the evaluation and to make preliminary comparisons with children and families participating in the national evaluation’s descriptive and outcome study samples. The purpose of the comparisons is to determine the generalizability of the findings of the longitudinal comparison study to the rest of the grantee sites. Without demonstrating the similarity of the children and families participating in the comparison study component of the national evaluation to the descriptive and outcome study samples at the remaining system-of-care sites, any subsequent analyses of differential outcomes in the comparison pairs would be subject to speculation regarding the representativeness of the findings for system-of-care sites as a whole. Caution is recommended in the interpretation of these results as full projected enrollment for the comparison study is 1,100 participants, and intake characteristics of comparison study participants are likely to change across time.

Demographics of Population

Table 9 compares demographic information for the 373 children enrolled in the comparison study prior to July 1, 1998, with overall data available from the descriptive study sample for the national evaluation. Data are reported separately for the three funded grantee sites (N = 170) and the three control sites (N = 203) participating in the comparison study. A slightly higher percentage of children enrolled in the grantee and control sites for the comparison study were males, compared to all children entering grant-funded programs. Mean age, however, was slightly lower for comparison study participants. Across both the grantee and control sites of the comparison study, females tended to be older (12.7 years) compared to males (11.1 years). Racial/ethnic distribution for the children entering the comparison study through the three grantee sites was similar to the overall program, with the exception of a lower percentage of children from Hispanic/Latino backgrounds. At this point of recruitment into the control sites, there was a higher percentage of African American children and a corresponding lower percentage of white children when compared to the three grantee sites participating in the comparison study. Three-fourths of Hispanic children were from the Santa Cruz County and Travis County sites. Finally, the children being served in both the grantee and control sites for the comparison study primarily came from poor families. Almost two-thirds of the children were from families whose annual income was below the poverty level, and 20 percent of families earned $5,000 or less per year. In contrast to the national evaluation’s descriptive study sample, participants in the comparison study were more likely to be male, on average somewhat younger, with a higher rate of African American and a lower rate of Hispanic racial/ethnic status. Distribution of family income was essentially the same. The differences displayed between grantee and control sites in demographic characteristics are preliminary and should be interpreted cautiously as they will likely change as enrollment reaches the full 1,100 projected for the study.

Behavior and Functioning at Intake

The total CAFAS scores provide a broad assessment of how children are able to function in the different domains of their life. The baseline total CAFAS scores indicate that most children in the three grantee and three control sites included in the comparison study sample have serious functional problems as they begin receiving mental health services (see Figure 32). Ninety-one percent of all children entering the comparison study have total CAFAS scores in the moderate, marked, and severe ranges, and just 2 percent display minimal impairment as they begin services. The three control sites have a higher percentage of children rated in the severe range while the three grantee sites have a higher percentage of children rated within the moderate range. Baseline CAFAS scores in the comparison studies are somewhat different than baseline CAFAS scores for the outcome study sample. A higher rate of mild impairment and lower rate of moderate impairment characterized the outcomes study sample at intake.

Table 10 provides information about the problems children are having in school in the past 6 months as reported by the caregiver at entry into the comparison study. In general, about half of the children in the study had serious problems at school. Absenteeism was similar for children in the grantee and control sites, with 42 percent overall absent from school at least once every 2 weeks. Higher percentages of children were receiving special education services in the three grantee sites. Higher rates of suspension and failure were reported for children from the control sites. Academic performance appeared to suffer as a result of this behavior, as almost one-third of children were failing half of their classes or more in the control sites.

Similar to the outcome study, older children had a tendency toward more serious functional problems according to the CAFAS scores (see Figure 33). The average CAFAS score consistently rose by about six points for every age group, and there was a significant difference between the average CAFAS scores of 6- to 11-year-old children and 12- to 17-year-old children (t = -4.3, df = 360, p < .000). CAFAS scores at each age level, however, were lower for the outcomes study sample than for the other two samples. In addition, CAFAS scores at each age level except 15 to 17 were higher for the three control sites compared to the three grantee sites.

An in-depth analysis of CAFAS subscale scores revealed that the overall age-related differences were primarily due to higher scores for youth in functional impairment on the self-harm and substance use subscales. However, no difference existed in the current comparison study sample between males and females. The average total CAFAS score for males was 71.5, and the average for females was 72.0. Males (26.2) scored slightly higher than females (25.8) on the CAFAS Roles subscale, which measured more of the external aggressive behaviors, but the difference was not statistically significant. Females (17.5) also scored slightly higher than males (15.7) on the CAFAS Moods/Self-Harm subscale, which measured more of the internal emotional conditions, but the difference again was not statistically significant.

The CBCL scores also revealed that most children in the comparison study met the clinical criteria for serious mental health disorders. The clinical cutoff point for both the Internalizing and Externalizing CBCL scale scores was 63, and overall means were above that level for the three grantee and three control sites (Table 11). The differences in scores between the three grantee sites and three control sites was minimal and not clinically significant. Even though most children met the CBCL’s criteria for clinical problems, little difference was found between males and females or among different age groups at this point in the data collection process. Males and females had similar average scores on both the Externalizing and Internalizing scales. Older and younger children also tended to score at the same level according to caregiver reports. Every gender and age group of children scored above the clinical score of 63 except for the 15- to 17-year-old children on the Internalizing scale. On average, CBCL scores were lower at intake for the outcomes study sample than for the comparison study sample.

Summary

Preliminary analysis of intake data from the comparison study indicated that children participating in this "arm" of the evaluation, as compared to the national descriptive or outcomes study samples,

were more likely to be male and on average somewhat younger,
were more likely to represent African American and less likely to represent white or Hispanic racial-ethnic backgrounds,
displayed higher rates of functional impairment at intake, and
displayed higher rates of behavioral and emotional problems at intake.

The analysis also indicated some differences in demographics and clinical measures at intake between children in three funded systems of care and children in three control communities. Variations in how services were structured and accessed in each community may be related to the initial sample differences that were displayed. For example, variability in referral sources and types of services that were available may have influenced significantly the characteristics of children who entered care. Each community participating in the comparison study will be assessed with a system-of-care site visit in the upcoming year to evaluate infrastructure and service delivery processes that may be contributing to the characteristics of the populations that are being served. These preliminary differences will be monitored closely as more children and families are recruited into the study.

Table of Contents | Previous | Next