This Web site is a component of the SAMHSA Health Information Network

Enter Keywords Choose Website This Site Mental Health at HHS All Mental Health

Online Publications Order Publications National Library of Medicine National Academies Press Publications Homepage

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services Program for Children and Their Families

The National Evaluation

The Comprehensive Community Mental Health Services for Children and Their Families Program has two overarching goals. The main goal is to build innovative home and community systems of care for children with serious emotional disturbance and their families. A secondary goal is to generate new knowledge about the most effective ways to meet the needs of children with serious emotional disturbance and their families. In keeping with this second goal, section 565(c) of the Public Health Service Act that authorized the program mandated an annual evaluation of its effectiveness. A basic requirement of the mandate is to describe the children and families served by the system-of-care initiative. Other requirements include an assessment of how systems of care develop and what factors impede or enhance their development, whether the children served by the program experience improvement in clinical and functional outcomes and whether improvements endure over time, whether the program can be associated with a reduction in residential treatment, and whether consumers are satisfied with the services they receive.

Given this, CMHS funded a national, cross-site evaluation that began in 1994. The evaluation is conducted by Macro International Inc. and its partners, the Louis de la Parte Florida Mental Health Institute of the University of South Florida and the Federation of Families for Children’s Mental Health. The evaluation is designed to maintain accountability and address the pressing need for information upon which to base programmatic, funding, and policy decisions.

This evaluation is the most comprehensive study to date of a children’s mental health service delivery program and of a community-based approach to addressing highly prevalent mental health problems in children and adolescents. Hopefully it can help fill in the gaps in the knowledge base. Much is to be learned from the evaluation that can help shape the delivery of mental health services to the Nation’s children and adolescents. There will be lessons for those involved in children’s system reform as well as those undertaking managed care initiatives affecting children with serious emotional disturbance and their families. Specific lessons will be learned about the organization and delivery of children’s mental health services; the effectiveness of the system of care; the financing and cost of services; program development, implementation and sustainability; and the long-term effects of systems of care.

Findings from this evaluation will address the following questions:

• Who are the children and families served by the program and by the grantees? What are their demographic characteristics, and how do they compare to children in other similar studies?
  
• To what extent and how do systems of care develop over time? What are the variations in system-of-care features among grantees? How do these features manifest themselves? To what degree does the implementation of systems of care stay true to the philosophy and values?
  
• Are children’s and families’ experiences consistent with system-of-care philosophy? Are providers’ attitudes and practices consistent with system-of-care philosophy?
  
• What services do children and families receive? How do service use patterns relate to child behavioral and functional diagnoses and outcomes? How do service use patterns differ across grantees and across subgroups of children?
  
• To what extent do children’s clinical and functional outcomes improve over time? Does this vary by grantee? by subgroup? Do children experience improved school performance and fewer contacts with the law? Do families experience improvements in family life?
  
• Are greater improvements in children’s behavior and functioning associated with a system-of-care approach to service delivery? Are better outcomes associated with more fully developed systems?
  
• How satisfied are children and families with the services they receive? How does the level of family involvement differ across grantees?
  
• What are the costs of these services? Are systems able to redirect their investments? Are they able to contain costs? How has managed care reform affected systems of care?
  
• Are systems of care able to sustain themselves after the Federal funding has ended? What factors facilitate or impede sustainability?

Figure 2 graphically depicts the system-of-care program theory model (shaded boxes) and shows a relationship to continuous quality improvement and system reform (dotted lines). This figure illustrates the connection between resources allocated to the program, program activities, and intended outcomes. The national evaluation focuses on the relationship between program activities and system, practice, and child and family outcomes. However, the model depicts an intended relationship between funding systems of care, demonstrating their effectiveness, and improving and sustaining them so that system reform is the ultimate outcome.

The evaluation is designed to provide an understanding of the overall system-of-care approach as portrayed in the program theory model, whether the program has been implemented as intended, and whether the goals of the program are reached. As the model shows, it is also designed to provide feedback to continuously improve the quality of the program. Broadly speaking, evaluation findings will provide

• data on the mental health and other service needs of children and families,
  
• data on the effectiveness of the system of care in serving children with mental health needs,
  
• an increased understanding of the interactions between providers and families,
  
• an increased understanding of systemness,
  
• information to guide service delivery approaches at the national and local levels, and
  
• empirical information to help inform policy and funding decisions.

Data from the evaluation have already proven useful at the national, State, and local levels. For example, in a multi-county service system in Southeast Kansas, informational presentations with data from the evaluation were conducted before local communities, State agency personnel, and Kansas legislators. They helped generate funding increases of over $5 million per year for services for children with serious emotional disturbance and their families. The State of Rhode Island passed two important pieces of legislation supporting the development of systems of care statewide and relied heavily on data derived from the evaluation.

Phase I and Phase II of the National Evaluation

The evaluation design is complex and includes multiple components that involve a variety of methodologies, both quantitative and qualitative. The following sections describe the Phase I evaluation, including a description of the longitudinal comparison study and a special study on managed care. Changes that have been made to improve the Phase I design in Phase II are also described.

Phase I

Phase I of the evaluation includes the 22 initial grantees funded in FY 93 and FY 94 (Cycles I–III). This phase is focused on both client and system outcomes. It was designed to generate a national picture of how systems of care develop and change over time, to describe the children and families who use system-of-care services and the functional and behavioral changes that occur during the time the child receives system-of-care services, and to depict the relationship between the system-of-care approach and child and family outcomes.

To address the research questions that were discussed above, a longitudinal quasi-experimental design has been used. A longitudinal design was used to obtain child outcomes because of the importance of understanding the types of changes that children with serious emotional disturbance undergo while they receive services. Also, a longitudinal design helps illuminate whether effects are due to the intervention or to normal developmental maturation. Data collection is conducted in all grantee sites and in selected comparison sites, where services are delivered in a more traditional manner. (A more in-depth description of the comparison study component follows.) Four major components comprise the Phase I evaluation:

The system-level assessment examines whether systems of care have been implemented in accordance with the program theory, and documents how systems develop over time to meet the needs of the children and families they serve. Two types of data collection are utilized. Site visits are conducted annually to all grantees to assess system-level development and implementation. This data collection is supplemented by family-centered interview visits to selected grantees to explore the implementation of system-of-care service principles.

A child and family descriptive component describes the children enrolled in the funded systems of care in terms of their demographics, functional status, living arrangement, diagnosis, risk factors, and mental health service history. Family demographics, socioeconomic status, and composition are also described. Data are collected for all children and families upon entry into the system. Selected data are collected again at 6 months, at 1 year, and annually for as long as the child is in the system of care.

Changes in the functional and behavioral status of the child are examined in the child and family outcome component of the evaluation. Outcome data on child clinical and functional status are used to assess change over time in the following areas: symptomatology, diagnosis, social functioning, substance abuse, school attendance and performance, delinquency and juvenile justice involvement, and stability of living arrangements. These data are collected on a sample of children

at each system-of-care site, at intake, 6 months, at 1 year, and annually for as long as the child is in the system of care.

Service use and the associated costs, and the relationship between service use and outcomes are explored in the services and costs component. Service and cost data are maintained continually by grantees in their fiscal (e.g., charge, billing) management information systems (MIS’s) and transmitted to Macro International at regular intervals. Of interest are the types of services, the combination of services, continuity or gaps in care, and the length of treatment. Where possible, service data from the MIS’s of a variety of agencies from different service sectors are integrated (e.g., mental health, juvenile justice, education, child welfare).

The Longitudinal Comparison Study

A significant aspect of the national evaluation is that it includes comparisons between those children and families served by CMHS-funded systems of care and those served by traditional service systems. Few studies in the children’s mental health field have included comparison populations with long-term follow-up of multiple outcomes. Those that have been conducted have yielded results with limited applicability to the broader arena of service delivery systems for children with mental health needs. The longitudinal comparison study is embedded in both Phase I and Phase II of the evaluation.

The comparison study component of the national evaluation was initiated to address a central question: "Can greater improvements in children’s behavior and functioning be attributed to the system-of-care approach compared to a traditional service delivery approach?" In responding to this question, the study will address several other topics of interest. It will provide an understanding of what happens to children and families after services are terminated, which will increase the knowledge base about stability of treatment effects over time. Other areas for examination include educational outcomes; substance abuse; the increasing importance of juvenile justice issues; and family life, family involvement in service delivery, and satisfaction with services. Because the follow-up component of the study will track youth for approximately 2 years, the increase in the number of older youth under study (ages 16 to 18) will provide an opportunity to examine the effects of the system-of-care approach on vocational and independent living skills.

To answer these questions, three CMHS-funded system-of-care sites are matched in Phase I with non-CMHS funded sites that appear to have made less progress toward adopting a system-of-care approach. Two to four additional comparison pairs will be selected in Phase II of the evaluation. Children are followed for a period of up to 2 years, both during services and after services are completed. Data also are collected from families, providers, and service agencies. The Phase I comparison study sites and their unique characteristics are summarized in the chart below.

Additional questions that will be addressed in the Phase II comparison study will provide a better understanding of what happens at the intervention level: Are planned services actually received? What factors contribute to adherence to service plans? Are children’s and families’ experiences consistent with the system-of-care philosophy? Are providers’ attitudes and practices consistent with the system-of-care philosophy?

The comparison study will make a valuable contribution to the field of children’s mental health by providing descriptive and outcome data on a large number of children with mental health needs drawn from many sites across the country. A significant contribution will be to document the differences in child and family outcomes of those served by CMHS-funded systems of care and those served with traditional service approaches. If the assumptions underlying the system-of-care concept hold true, and if the CMHS-funded sites implement the concept, then it is anticipated that better outcomes in behavior and functioning will be realized among children who have been served by these systems of care. Specifically, children will have greater success in school, at home, and in the community; will have fewer encounters with the juvenile justice system; will report less substance abuse; and will have greater satisfaction with services and supports than children served by traditional service approaches. Further, the unique characteristics of the populations being studied will provide information on how systems of care impact special populations of children and their families.

Finally, decisions to implement and support a system-of-care approach will be based not only on the intuitive sense and expert opinion that the system-of-care approach is the "right thing to do for families and children," but also on positive outcomes attributable to this approach.

Special Study on Managed Care

The communities receiving grant support from CMHS are endeavoring to develop comprehensive, community-based systems of care for children and adolescents with serious emotional disturbance and their families. Their activities, however, are occurring in the context of other significant environmental forces and reforms that inevitably affect children and families as well as service systems. One such contextual influence is the rapid adoption and application of managed care technologies to the delivery of mental health and substance abuse (i.e., behavioral health) services, with goals that often include improving the efficiency of service delivery and containing costs. This special study is designed to track and analyze managed care reforms as they affect children and adolescents with serious emotional disorders and their families, and the systems of care that serve them. More specifically, the purposes of the special study include the following:

• To assess the impact of managed care initiatives on the development and operation of local systems of care.
  
• To assess the impact of managed care initiatives on children and adolescents with serious emotional disturbance and their families through the lens of various stakeholders.
  
• To explore how different groups of stakeholders view managed care reforms, the positive effects, and the problems that have been created.
  
• To begin to identify aspects of the structure, design, organization, and characteristics of managed care systems that have facilitated or created problems for mental health service delivery to children, adolescents, and their families.
  
• To identify practical technical assistance for States and communities related to managed care for this population of children and families.

The study involved reviewing information related to managed care from the system-level site visits conducted through the national evaluation. Based on this review, sites that reported activity related to managed care were identified and a sample of these sites were selected for inclusion in the study. Variation in the type of system-of-care project, geographic diversity, and variation in the characteristics of managed care reforms that were either planned or underway were among the criteria considered in the sample selection process. Three sites were selected for telephone interviews and three sites for in-depth site visits.

At the time of the telephone interviews and site visits, Las Cruces was planning for managed care, Lane County had recently implemented managed care, and all other sites had implemented managed care for a minimum of 1 year and had a substantial base of experience. Telephone interviews were typically conducted with the site director and one or two additional key informants from the site. The site visits involved approximately seven group interviews at each site with stakeholder groups, including the site director and key staff; representatives of the entity responsible for purchasing or overseeing behavioral health managed care; representatives of the behavioral health managed care organization; mental health service providers; care managers and clinicians; families; and representatives of child-serving systems, including child welfare, juvenile justice, and education.

The report synthesizes information across sites and raises critical issues related to the implementation of managed care and the development of community-based systems of care for children and adolescents with serious emotional disorders and their families. Findings indicate that

within the group of sites, the system-of-care philosophy and approaches can, in fact, be maintained in a managed care environment, and under the right circumstances, the system-of-care philosophy and approaches can even be used to shape managed care systems. However, there are clear indications that without specific attention to and planning for children and adolescents with emotional disorders, many system-of-care features can be threatened with managed care reforms. Despite these threats, there is evidence of efforts to stay true to the system-of-care philosophy even in the context of major system changes such as managed care reform.

It is the intent of the project that the findings will be used at Federal, State, and local levels to refine managed care reforms and guide future reforms so that they will be more responsive to the needs of children and adolescents with emotional and substance abuse disorders and their families.

Building on Phase I: Improvements in Phase II

Phase II of the evaluation includes the 9 grantees funded in FY 97 and the 14 grantees funded in FY 98 and FY 99 (Cycles IV and V). Phase II includes the four components described above, but to build on the knowledge gleaned from Phase I and from the field in general, Phase II has incorporated changes and improvements to the Phase I design and the data collection instruments. Some of the lessons learned from the Phase I evaluation and from the field, and how they are addressed in the Phase II evaluation, include the following:

• In the Phase I evaluation, outcome data are collected only from children who are actively receiving services. In recent years, however, the field has learned that children with serious emotional disturbance have pervasive problems that often require intermittent services and that many children cycle in and out of services. In addition, sites use a variety of definitions of "exit from services," making it difficult to equate lengths of stay across sites or to determine whether a child should be included in follow-up data collection. As a result, the clinical and functional outcomes of many children cannot be captured over time. Phase II has addressed this issue by following children for the duration of the evaluation regardless of whether they continue to receive services.
  
• The Phase I evaluation emphasizes child outcomes. However, information obtained during the site visits and input from family representatives in the Phase I evaluation suggests that family functioning influences the level of participation in treatment and is an important dimension to assess. A growing number of studies have also revealed the important role families play in their children’s use of services and continuation in treatment. The increasing emphasis on community-based alternatives has placed greater responsibility on families in the treatment and management of their children’s emotional and behavioral problems. Especially for children, the family is the primary context where emotional and behavioral problems emerge and are manifested. As children seldom seek mental health services for themselves, families play an important role in bringing children to treatment. Caregivers are subsequently important agents in making and following through on treatment decisions. Given the central role of families, the Phase II evaluation includes three family measures to assess family functioning, adequacy of material resources, and caregiver strain.
  
• The need to understand whether children and families actually receive services in accordance with the system-of-care program theory has become increasingly clear. For example, one gap of the Fort Bragg Evaluation Project was that too little was known about children’s and families’ experiences with services to explain the findings. It is important to know not only whether the system of care was developed in accordance with the program model, but also whether individual clients received those services as expected. Without understanding how well system-of-care principles are realized in actual service delivery, it is difficult to explain evaluation findings. Phase II has incorporated a practice-level assessment to address this gap. This component, to be implemented in the comparison study, will examine children’s and families’ experiences at the practice level, which includes recording multisector service contacts, client satisfaction questionnaires, family ratings of their experiences with the services and the system, and a questionnaire for providers to assess the extent to which they embrace the system-of-care approach. This assessment will also include a substudy on adherence to service plans.

Current Status of Phase II

The design and instrument package for Phase II were developed with diverse inputs from many sources, including Federal agencies outside SAMHSA; the Research Advisory Board to the national evaluation; Phase I grantees; various national-level partners, including the Federation of Families for Children’s Mental Health; and families of children with severe emotional disturbance. Final approval of the Phase II instrument package by the Federal Office of Management and Budget was received on September 29, 1998.

Through introductory site visits to discuss the local evaluation plan, Macro International has established relationships with each of the Phase II sites in Cycle IV. Additional visits to Cycle IV sites to conduct Phase II data collection and evaluation procedures training will be timed to coincide with the launching of local-level evaluations. Most sites devote the first year of the grant funding period to the development or refinement of system-of-care services, hiring, and other start-up activities. The first wave of baseline data collection will begin after this start-up period (fall and winter of 1998 in the case of Cycle IV). Initial contact with Cycle V sites will begin in the fall of 1998 and data collection for Cycle V sites is expected to begin in the fall of 1999. 

Table of Contents | Previous | Next