Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services Program for Children and Their Families

Assessment Methodology

System development site visits have been conducted annually to the 22 Phase I grantees for the past 3 years. Since some grantees operate multiple local systems of care, site visits are conducted in a subsample of these local systems, for a total of 31 site visits per year. Research staff from Macro International Inc. and the Louis de la Parte Florida Mental Health Institute, University of South Florida (USF), and a group of nationally known children’s mental health experts conducted these visits. The map in Figure 3 illustrates the system-of-care sites that have been visited as part of these assessments. The findings from these site visits are presented below.

Site visits were organized around a comprehensive and detailed interview protocol, which asked multiple respondents to address questions pertaining to system-of-care development. Each site visiting team was comprised of two site visitors familiar with system-of-care theory and trained to implement the protocol. Using semi-structured interviews tailored to specific respondent categories, site visitors interviewed staff and families representing diverse perspectives on the system of care. Interview respondents included system administrators, direct service providers, families served in the system, and community representatives.

The site visit protocol was designed to assess system development, which has been called "systemness" for purposes of the evaluation. Macro International, with input from the University of South Florida, the Federation of Families for Children’s Mental Health, and a panel of national experts including researchers, theorists, and practitioners from the field of children’s mental health, created this index of systemness. The purpose of the index is to capture the degree to which systems of care have been implemented and are evolving over time. This index was constructed in several stages, beginning with an extensive literature review and a review of grantee applications to distill the critical characteristics of a system of care. This distillation was reviewed by the panel and, with their feedback, two overarching domains were identified and a list of attributes comprising eachdomain was developed. A series of questions was developed that operationalized each of the attributes. These questions were reviewed by the panel to determine how well they operationalized the attributes and whether the domains had been sufficiently represented. The index was revised and sent again to the panel for review. The panel also ranked each question according to its importance in assessing system development. Responses were tallied and the index was again revised.

Table 2: displays the two domains and the attributes that comprise the system-of-care approach.

Systemness Domains and Attributes

Family-centered site visits complemented the system-level assessment by more specifically focusing on family perceptions of and experiences with the system of care. Conducted by staff from USF, these site visits followed a case study format and were conducted with 12 families in each of 8 sites. These interviews were designed to provide information about service delivery in the context of the communities in which the systems of care operate. Study protocols were developed to explore issues related to system-of-care principles. During these family-centered site visits, interviews were conducted with an individual child receiving services, his or her primary caregiver(s), persons who provide informal support to the family, and representatives of the different systems serving them. Findings from the family-centered site visits, which have been shared with each of the sites, indicate that

the service experiences of families are more consistent with system-of-care objectives at the CMHS-funded systems than at their matched comparison sites, and
systems of care may be having a significant impact on caregiver perceptions of child improvement when assessed at the time of service provision.

Table of Contents | Previous | Next