Appendix D

Survey Design and Methodology

School Mental Health Services in the United States, 2002-2003 is a survey of a nationally representative sample of public K—12 schools and their associated school districts.  It was conducted as a mail self-administered survey during the 2002-2003 school year. The survey consisted of two questionnaires. The school questionnaire collected data on the types of mental health problems encountered in schools, the mental health services provided, the types and qualifications of staff providing services, and the arrangements for delivering mental health services. The district questionnaire collected data on funding sources for mental health services and issues related to funding. This appendix describes the design and methodology of the survey, including the sampling design, the data collection methods, the data preparation, unit and item response rates, including estimation of bias due to non-response, and weighting and variance estimation.

I. Sampling Design

A. Objective of the Design and Target Population

The objective of the sampling design was to select a nationally representative sample of public K—12 schools in the U.S. The universe for this survey consists of all regular public elementary, middle, high and combined schools with any of grades 1—12. The selection of regular (that is, comprehensive schools) denotes the exclusion of special education, vocational technical, and alternative schools.  The selection of schools with any of grades 1 through 12 does not preclude the selection of schools with kindergartens, but prevents selection of schools that have kindergarten or lower as the only grade(s).

 The sampling unit for this survey is schools, as opposed to school districts.  Therefore, schools were sampled first, and then the school districts corresponding to the sampled schools were included in the survey to answer questions about the funding of mental health services. Sampling schools first and then including their districts is a more efficient for obtaining national estimates of characteristics of schools than is a two-stage method in which school districts are selected first and then schools are selected within school districts.

 The National Center for Education Statistics’ (NCES) Common Core of Data (CCD) was used as the sampling frame.  The CCD, which is updated annually, contains the universe of public K-12 schools in the nation.  The latest version available at the time the sample was drawn was the 2000-2001 CCD, two school years prior to the year of data collection.  

B. Stratification, Sample Size and Allocation

 The strata for the selection of the sample schools were created by the cross-classification of two variables that are important in the analysis and reporting of the data:  school level and school size. School level refers to the grade configuration of schools and, for the purposes of this survey, was categorized as elementary, middle, secondary, and combined.  School size is measured by student enrollment and, for the purpose of this survey, was categorized as small (from 1 to 250 students), medium (251 to 500 students), large (501 to 1000 students), and very large (1001 and more students). The following exhibit shows the distribution of the population of schools by these strata.

 Exhibit I.1. The universe of regular public K—12 schools, by school level and size. 

 

Source: U.S. Dept. of Education, National Center for Education Statistics,

Common Core of Data: 2000-2001. 

Sample Size: A sample of 2,125 schools was drawn. With a minimum 80 percent response rate, and assuming that some schools would be closed and some would be out of scope, 1,600 completed school surveys were anticipated. The size of the sample was designed to provide reliable estimates of the universe of regular public schools by three levels (elementary, middle, and secondary) and, separately, by three size categories (small, medium, and large/very large).  The sample was designed to detect differences of 8 percentage points, with 80 percent power, between the three school levels, the three size categories, as well as between four regions, three locales (urban, suburban, rural), three categories of free-lunch eligibility and three categories of minority student enrollment.

Sample Allocation: The overall sample was allocated to each size group in proportion to the square root of the total number of students in each size group.  Several sample allocations were considered.  If the overall number of schools in the sample was allocated to each size group in proportion to the number of schools, there would be a very large number of small schools in the sample and not enough large schools.  Similarly, if the sample of schools was allocated in proportion to the number of students in each size group, there would be a small number of schools in the first size group.  As a compromise, the square root allocation was adopted, which gives a moderate sample size from each size group.  The sample in each size group was then allocated to each school level in proportion to the number of schools in the population belonging to that level.  The distribution of the sample by strata is shown in the Exhibit I.2.

Exhibit I.2.  Number of sampled schools by level and size

	Level
Size	Elementary	Middle	High	Combined	Total
Missing	3	2	3	17	25
1--250	147	34	48	28	257
251--500	396	70	58	16	540
501+	531	285	455	32	1,303
Total	1,077	391	564	93	2,125

Within each stratum, schools were sorted by region (Northeast, Midwest, South, West), locale (urban/central city, suburban/large town, small town/rural), and then school size. This sampling strategy was designed to yield estimates to populate the standard table shell used in this report (The definitions of the standard reporting categories and the tables of estimates for school and district data are in Appendix A.).

Selection of School Districts Associated with Sampled Schools

 The sampling unit for this survey is schools, as opposed to school districts.  Therefore, schools were sampled first, and then the school districts corresponding to each sampled school were drawn into the sample to answer questions about the funding of mental health services and other administrative questions that could not easily be answered at the school level. There were 1,595 districts associated with the 2,125 sampled schools. Procedures were developed for weighting the district data to obtain national estimates of school districts (see section IV. School District Weights).

II.  Data Collection Methods and Procedures

Data collection began on November 4, 2002 and was completed on June 30, 2003.  The following sections describe the methods used to identify contacts at the districts and schools, to encourage participation, and to ensure that the data were representative of the schools and districts in the nation.  

A. Phase One: Pre-survey contact with school districts

Mailing to District Superintendents

On November 4, 2002, a study packet was mailed to the 1,595 school districts associated with the sampled schools. The objective of the mailing was to inform district superintendents about the study and its sponsor, to enlist their cooperation, and to obtain the names and contact information of designated respondents for the district and school surveys.  The study packet included a letter of explanation, contact form, school and district questionnaire content sheets, business reply envelope, and listing of Substance Abuse and Mental Health Services Administration (SAMHSA) publications, that districts could order free of charge in appreciation for their participation.

Superintendents who wished to participate in the study were asked to complete the contact form with the name of a designee who would complete the district questionnaire, and the names of the principals at each of the sampled schools.  Districts could return the form to Abt Associates by fax or mail.  A toll-free number was included on the forms so districts could submit the information by phone or call in with questions about the study.  Telephone interviewers at the Amherst phone center answered the phone and returned messages left on the voice mail.    

Follow-up Telephone Calls to District Superintendents

On November 15, 2002, telephone interviewers began placing calls to superintendents in the 1,536 districts that had not returned a contact form.  The goal of the call was to complete the contact form over the phone or confirm that the district would be returning the form to Abt Associates.  The calls were also intended to inform districts of the importance of their participation in the survey.  

All of the interviewers were experienced and trained on the project. Training included an overview of the project, the sample, and the structure of districts and schools. Trainers reviewed the script to be used when calling the district offices with the interviewers.  The script included text for the different scenarios interviewers might encounter during the calls, as well as refusal aversion strategies, designed to preempt district refusals. Interviewers practiced the script with partners and supervisors evaluated interviewers’ project knowledge before they began follow-up calls. 

Training emphasized the importance of establishing the legitimacy and importance of the study with gatekeepers such as administrative assistants, receptionists, and secretaries. Superintendents were often unavailable to take outside telephone calls, so gatekeepers played a critical role in ensuring that the superintendents were aware of the study and the forms were completed and returned.

During the telephone follow-up, interviewers often found districts that had not received or had lost the study packet. In these cases, clerical staff faxed or mailed a new form the same day.  The full study packet was requested by and re-mailed to 152 districts.   The contact form was faxed to 1,179 districts upon request and in some cases the form was faxed to the district more than once.  After the materials were re-mailed, interviewers would suspend calling for a week, after which, if the form had still not been received, another round of reminder calls was initiated. 

Research Applications

During the first phase of data collection, project staff discovered that 58 districts, 3.5 percent of the sampled districts, required that a research application be approved before research could be conducted in the district or their schools.  The process varied in each district and ranged from a simple form submitted to a district committee, to a lengthy application process requiring a phone-conference presentation of the project.  In several districts, it was necessary to obtain consent from each sampled school and submit signed consent forms to the district before approval was granted. Of the 58 research applications, 46 (75 percent) were approved.  Two hundred and fifteen schools (10 percent of the original school sample) were in these 46 districts.    

Redistricting

The original sample included 2,125 schools in 1,595 districts.  When calling the districts, telephone interviewers found some very large districts had decentralized into several smaller districts. As a result, several schools were in districts not originally identified as part of the study. A total of 26 districts were identified and added to the sample, for a revised total of 1,621 districts. 

Second Mailing to District

By mid-January, contact information had been received from 56 percent of the districts.   After excluding districts that had refused, were closed or were out of scope, and in which research applications were outstanding, study packets were re-mailed to 578 districts (36 percent of the district sample).  Project staff conducted an Internet search to identify the remaining superintendents’ names, so the second mailing could be personalized. 

Refusal Conversion

In January 2003, refusal conversion efforts began in the districts that had declined to submit contact information.  A refusal conversion letter was sent to the superintendents, outlining the importance of the study and of their participation.  A week after the mailing, telephone follow-up began. The most successful interviewers were trained in refusal conversion techniques and called districts, attempting to speak directly with superintendents or administrators whose responsibilities included management of mental health services.  Of the 261 districts that had initially refused to participate, telephone interviewers successfully converted and gathered contact information from 69, for a refusal conversion rate of 26 percent.

Phase One Response Rate Summary

By the end of Phase One, 12 percent of the districts had declined to participate, 6 districts were considered out of scope, and 9 districts were closed.  Exhibit II.1.shows the disposition of the district cases and of the schools associated with those districts. 

Exhibit II.1.  Disposition of districts and their associated schools at the end of Phase One 

	Total	Complete	Refused	Out of scope	Closed	Outstanding
District	1,621	1,313  81%	192  12%	6    0.3%	9  0.5%	99     6%
School	2,125	1,732  82%	242  11%	6    0.2%	9  0.4%	135    6%

B. Phase Two:  Mailing and Collection of Survey Questionnaires 

Advance letter and first mailing of questionnaires to districts and schools

The second phase of data collection – the mailing of the advance letters and the first mailing of survey questionnaires to districts and schools – began in January of 2003. As the district and school contact information with names of respondents became available, personalized advance letters were mailed.  The letter explained the study and alerted the designated person that he/she had been chosen by the district office to complete a questionnaire that would be arriving shortly by mail.

Approximately a week after the advance letter was sent, a questionnaire packet was mailed to the designee at each school and district.  Included in the packet was a letter explaining the importance of the study, the appropriate school or district questionnaire, a business reply envelope, and a list of SAMHSA publications that could be ordered free of charge in appreciation for their participation.  The toll-free number was included in the letter and on the questionnaire to answer respondent questions. The first batch of questionnaires was sent to districts on January 20, 2003 and to schools on January 29, 2003.   Exhibit II.2. shows the schedule of the initial mailing to the schools and districts.  In total, 1,419 districts (87 percent of the district sample) and 1,880 schools (88 percent of the school sample) were mailed a questionnaire. The remaining sample were refusals, closed districts, or out of scope.

Exhibit II.2. Schedule for first mailing of questionnaires to districts and schools

District (n=1,621)		School  (n=2,125)
*Date	Total Mailed	*Date	Total Mailed
1/17/2003	849     52%	1/29/2003	962     45%
2/20/2003	271     17%	2/20/2003	320     15%
3/18/2003	241     15%	3/18/2003	448     21%
4/14/2003	12         1%	4/14/2003	80         4%
4/25/2003	20         1%	4/25/2003	26         1%
5/8/2003	26         1%	5/18/2003	44         2%
Total	1,419   87%	Total	1,880  88%

*Advance letters were mailed one week before, and a reminder postcard was mailed two weeks after the questionnaires.

Telephone Follow-up

Approximately a week after the questionnaires were mailed, telephone interviewers began calling the designees at each school and district to confirm that the questionnaire had been received, answer any questions and determine if and when the questionnaire would be returned.  During the calls, telephone interviewers reminded respondents about the importance of the study, and the need for all sampled districts and schools to participate to ensure the accuracy of the data.

All of the interviewers trained to work on Phase Two had worked on the first phase of the project.  During the training, staff provided interviewers with an overview of the questionnaires to prepare them to answer respondent questions.  Interviewers were also trained in the script for calling the districts and schools and in refusal aversion techniques.  Interviewers practiced using the scripts with partners and supervisors evaluated interviewers’ project knowledge before they began calling respondents. 

After the first week of calling the schools, interviewers reported that some school administrators felt they should not complete the questionnaire because the school did not provide mental health services.  When a designee indicated this during the telephone call, interviewers were trained to explain that it was still important that data be collected from the school.  Telephone interviewers were instructed to ask designees to write on the questionnaire “school does not provide mental health services”, and answer as many questions as they thought applied to the school.       

The initial telephone calls to remind the schools and districts to complete the questionnaires were placed to the person designated during Phase One.  However, telephone interviewers discovered that questionnaires were often forwarded to another person in the school. As a result, many calls had to be made to the districts, and especially to schools, to track down the persons ultimately responsible for completing the questionnaire.  During this procedure, interviewers received numerous requests for re-mails.  Thirty one percent of the districts requested one re-mail and eight percent requested more than one. Thirty nine percent of the schools requested one re-mail and 12 percent requested more than one.

Reminder postcard

Two weeks after the initial mailing of the questionnaires, a postcard was mailed to the school and district designees, reminding them to complete and return the questionnaires.  The toll-free number was included on the postcard so designees could call with questions or to request another questionnaire. 

Mailing to schools and districts with outstanding contact information

The standard procedure for mailing surveys to districts and schools was to first obtain the contact information from the district. In Phase Two, there were a number of sample cases that had neither refused participation nor provided contact information. In an effort to maximize the response rates, questionnaires were mailed to these 119 districts and 205 schools. The names of most superintendents had been obtained during follow-up telephone calls and the names of most principals had been obtained through an Internet search, so the packets were personalized. We continued to collect and process contact forms until the end of data collection, obtaining forms from 81 percent of the districts.

Second Mailing to Schools

By the beginning of April, 47 percent of the districts and 34 percent of the schools that had been sent questionnaires had returned them. To improve the school response rate, questionnaires were re-mailed to all schools that had been mailed questionnaires in January and February, but had not returned them and had not refused to participate. If telephone interviewers obtained revised respondent names, the questionnaires were addressed to the new contact person.  On April 11, 2003, questionnaire packets were re-mailed to 326 schools.   

 

C. School Refusal Conversion and School Nonresponse Bias Survey

Phase Two refusal conversion efforts began in the schools on May 19, 2003.  The refusal conversion interviewers from Phase One conducted the Phase Two refusal conversion.   The primary goal was to contact the actual respondents and ask them to reconsider participation in the study and to return the questionnaires.  Two unanticipated issues arose during this phase, requiring adjustment of procedures. One was the identification of schools that did not provide any mental health services to their students and the second was a high rate of school non-response that threatened to bias the results of the survey.

Schools that do not provide mental health services: During refusal conversion, a number of schools indicated that they had refused because they did not provide mental health services and therefore thought they were not eligible for the survey. This was unanticipated in the design of the survey, so a procedure was designed to identify these schools. This information was needed to calculate the percentage of schools in the nation that do not provide mental health services for their students.  Project staff created a set of screener questions to administer during refusal conversion. Telephone interviewers either administered the screener questions on the phone or faxed the form to the school to be completed and faxed back.  

School non-response and potential non-response bias:  As the school year was nearing its end, the rate of returns on the survey forms that had been mailed was 69 percent for districts and 54 percent for schools. To target the refusal conversion efforts, the school response rates were examined by sampling strata, revealing that the response rates for large urban schools were lower than those for other schools. Urban schools serve larger student populations and therefore contribute more weight to the national estimates.  In addition, urban schools may be different from other schools in their mental health programs and circumstances. For these two reasons, higher rates of non-response from these schools could potentially result in biased estimates.  Project staff decided it was essential to supplement the school data by collecting a sub-sample of survey items from the non-responding schools.  A non-responder was defined as any school that was mailed a questionnaire but had not returned it and had not refused to participate.

Critical items survey: The critical items survey was designed to both measure the potential bias due to school non-response and to increase response rates among these schools. The survey consisted of the subset of survey items considered most “critical” to producing comprehensive national estimates on school mental health. In addition to being critical, items were selected that could be completed on the phone in about ten minutes. A random sub-sample of schools was selected in each stratum, with the sample size set to raise the stratum response rate to at least 60 percent.  The result was a sample of 402 non-responding schools. The stratum level response rates before and after the administration of the critical items survey are presented in section III.

The use of the screener questions and of the critical items survey is explained in the section that follows.

Schools that had refused to participate were separated into two groups for refusal conversion, non-urban and urban.

Non-urban schools – Telephone interviewers attempted to contact the 124 non-urban schools that declined to participate, and ask the contact person to reconsider completing the questionnaire.  If possible, while re-contacting the schools, telephone interviewers administered the screener questions to determine if the school provided mental health services.  Interviewers were unable convert the refusal or administer the screener questions in 52 of the 124 non-urban schools that refused to participate.  In the 72 schools successfully converted, 7 schools returned the questionnaire and 65 schools completed the screener questions.  Of the 65 schools completing the screener questions, 92 percent indicated they provided mental health services and 8 percent indicated they did not provide mental health services. The use of these cases in data analysis is explained in section III.  Exhibit II.3. summarizes the non-urban school refusal conversion efforts. 

Exhibit II.3. Non-urban School Refusal Conversion Summary

Total Sample	Outstanding   Sample	Questionnaire Returned	*Screener Questions Completed
124   100%	52      42%	7      6%	65    52%

*  Of the 65 screener questions completed, 60 schools indicated they provided mental health services and 5 schools indicated they did not provide mental health services.

Urban schools – Telephone interviewers attempted to contact the 34 urban schools that had refused to participate in the study, and to ask the contact person to reconsider completing the questionnaire.  If possible, while re-contacting the schools, telephone interviewers attempted to administer the screener questions and determine if the schools provided mental health services.  If telephone interviewers were unable to convert the refusal, but had determined the school provided mental health services, they attempted to collect the critical items.  Interviewers could administer the critical items by phone or fax.   Interviewers attempted to convert 34 urban refusals.  They were able to make contact with 13 schools, 2 of which returned a full questionnaire.    The remaining 11 schools  completed the screener questions and all of them indicated they provided mental health services.  Interviewers were able to collect the critical items from 9 of the schools indicating they provided mental health services.  Exhibit II.4. summarizes the outcome of the refusal conversion efforts with the urban schools.      

Exhibit II.4. Urban School Refusal Conversion Summary

Total Sample	Outstanding Sample	Questionnaire Returned	Screener and Critical Items Completed	Screener Questions Completed
34	21	2	9	2
100%	62%	6%	26%	6%

 

On May 30, 2003, telephone interviewers began contacting the remaining non-responders.  During the telephone calls, interviewers reminded contacts how important the study was and still attempted to obtain the full questionnaire from the school.   If the school provided mental health services, but declined to complete the full questionnaire, interviewers attempted to complete the critical items.  Telephone interviewers used a combination of collecting screener questions and the critical items data by phone and fax.  As a result, some schools completed the set of screener questions and not the critical items and some completed the critical items, but not the screener questions.  Below is a table that summarizes the outcomes of the non-responder data collection. 

Exhibit II.5.  Non-respondent Sample Summary

Total Sample	Outstanding Sample	Questionnaire Returned	Screener and Critical Items Completed	Screener Questions Completed	Critical Items Completed
402	190	63	136	8	5

Data Collection Summary

When data collection ended on July 30, 2003, questionnaires had been received from 1,064 districts and 1,147 schools.  The screener questions were administered to 220 schools; 213 schools indicated they did provide mental health services, while 7 of these schools indicated they did not provide mental health services.  The critical items data were collected from 150 schools.  

        III.      Data Editing and Calculation of Final Response Rates

 

A. Editing School Data and Calculating Final Response Rates

The process of data entry, cleaning and editing began in the spring and continued through the end of data collection and into the early fall. The process consisted of several steps.  First, the survey forms were reviewed manually to correct entries and review written comments.  Then the data were key-entered and the accuracy of the data entry was verified. When a sufficient number of cases had been entered, staff began a review of responses to the open-ended questions and developed codes for them. The data were then checked for internal consistency and edited when inconsistencies could be logically resolved. In cases where data were missing on a school’s enrollment or counts of students by race/ethnicity or free lunch eligibility, the data were imputed by borrowing from the CCD, if available. When the data file had been cleaned and edited, item frequencies were calculated to identify items with high non-response that might cause problems in analysis. In the school questionnaire the overall rates of item non-response were low (below five percent) with the exception of subparts of item 13 on counts of staff by full and part-time status. These data were excluded from the analysis and only total head counts of staff were used.

After the data file had been cleaned and edited, all schools in the sample (2,125) were assigned a final interview status code for the purpose of calculating unit response rates and assigning weights. These status codes were derived from examination of individual cases on the data file.  In this process additional cases were identified as non-respondents to the full and critical items surveys. There were other cases that originally were considered respondents but were then identified as out of scope because they did not meet the criteria for regular K—12 public school. Because of these reclassifications the numbers reported below by final interview status code do not always match those reported in the previous section, Data Collection Methods and Procedures. 

The final interview status codes and the number of schools in each are as follows:

 

Final Interview Status	Number of Schools
1.Completed Full Survey	1,128
2.Completed Critical Survey Items	146
3.Refused but DID offer MH Services	87
4.Refused but DID NOT offer MH services	20
5.Districts refused participation of their schools	255
6.Schools refused	89
7.Did not respond	382
8.Out of scope	18

 

The cases with status codes 1, 2, 3, and 4 were included in estimates of the total number of regular K—12 schools in the U.S.  Schools that did not offer mental health services (status code 4) were used to estimate the percentage of all regular K—12 schools in the nation that do not provide mental health services.  Cases with status codes 1 and 2 were used in estimates of characteristics of schools that do provide mental health services. The procedures for estimation based on this sample are described in section IV, below.

 

The exhibits below show the overall response rates for the school survey with and without schools that completed the critical items survey. Not including respondents to the critical items survey, the response rate was 54 percent (Exhibit III.1).  When the critical items respondents were included, the response rate increased to 60.5 percent (Exhibit III.2).

Exhibit III.1.  School response rate without critical items data.

Without Critical Items Data	Unweighted Count	Unweighted Response Rate	Weighted Count	Weighted Response Rate
	979	46.46%	38,374	45.99%
Nonrespondent
Respondent	1,128	53.54%	45,071	54.01%
Total	2,107	100.00%	83,446	100.00%

Exhibit III.2.  School response rate including critical items data.

 

With Critical Items Data	Unweighted Count	Unweighted RR	Weighted Count	Weighted RR
Nonrespondent	833	39.53%	32,990	39.53%
Respondent	1,274	60.47%	50,456	60.47%
Total	2,107	100.00%	83,446	100.00%

 

The response rates by stratum without and with the critical items data are presented in Exhibits III.3 and III.4.below:

Exhibit III.3 School counts and response rates without critical items data, weighted and unweighted, by stratum.