U. S. Department of Education Office of Planning, Evaluation and Policy Development

C. Previous Findings

• 
  For the average eligible applicant, Upward Bound had little effect on most key high school outcomes, including credits, grades, and graduation. Myers et al. (2004) found Upward Bound had no effect on total credits and a small effect on credits earned in high school math. The program increased the number of math credits earned by 0.2 credits; that is, about one in five students completed an additional high school math course because of exposure to Upward Bound. Upward Bound had no effect on credits earned in science, English, social studies, or foreign language courses. Also, the program had no effect on honors and Advanced Placement credits, grades earned in high school, or high school graduation.
  
• 
  For the average eligible applicant, Upward Bound had few short-run effects on postsecondary outcomes, but may have increased enrollment at four-year colleges and universities. Upward Bound had little effect on enrollment and credits earned at two-year or vocational postsecondary institutions and on the receipt of college financial aid. Myers et al. (2004) found some inconclusive evidence that Upward Bound may have increased the percentage of treatment group members attending a four-year college or university. When all postsecondary enrollment information reported by sample members was included, the estimated effect was 6 percentage points and was statistically significant; however, when unverified enrollment information was excluded (as described in Appendix H of the report), the effect fell to 5 percentage points and was not statistically significant.
  
• 
  Upward Bound had positive effects for eligible applicants with lower educational expectations. For eligible applicants with lower educational expectations—those who did not expect to earn a bachelor’s degree when they applied to Upward Bound—Upward Bound increased Advanced Placement or honors credits as well as credits earned in core academic subjects in high school. It also had short-term effects on some postsecondary outcomes for this group, such as the likelihood of enrolling in a four-year college or university, total postsecondary credits, and credits earned at four-year colleges and universities. For eligible applicants who expected to obtain a bachelor’s degree or more, Upward Bound had little short-term effect on any of these outcomes.
  

II. Research Design and analytic issues

A. Research Design

1. Selection of Upward Bound Projects and Random Assignment

For the evaluation, we randomly selected 70 Upward Bound projects representative of all 395 regular Upward Bound projects operating in the 50 states and the District of Columbia that were hosted by a postsecondary institution, had operated for at least three years as of October 1992, and were not dedicated to serving only students with physical disabilities. Many different designs for selecting the sample of projects were considered. Several designs had relatively modest stratification and modest variability in sampling rates for different types of projects defined by potentially policy-relevant characteristics, including project size and type of host institution. Such a design would have supported precise estimates for many key subgroups

Of the 70 projects originally selected, 11 could not participate or had to be excluded for various reasons. For example, some did not plan to recruit new students for the 1992–93 school year, some had too few applicants to accommodate random assignment, and some did not have their Upward Bound grants renewed. We replaced eight of these 11 projects with similar, randomly selected projects, arriving at a total sample of 67 projects. See Appendix A for a detailed description of the sample selection and weighting procedures.

During the 1992–93 and 1993–94 school years, we randomly assigned eligible applicants from each project to either a treatment group, which was invited to participate in Upward Bound, or a control group, which was not invited to participate. Eligible applicants were defined as students who the projects had recruited and who met both the federal eligibility criteria (low-income or potential first-generation college student status) and any project-specific criteria for participation. All of the projects received more applications than they had openings, and all served the same number of students they would have normally served under their usual selection procedures.

We implemented random assignment over 14 months so that projects could use their standard recruiting procedures and enroll students in accordance with their usual enrollment schedules. Nationwide, the random assignment process resulted in a treatment group of about 1,500 students and a control group of about 1,300 students for subsequent impact analyses. Myers et al. (1993) presented a detailed description of the random assignment procedures.

To accommodate project wishes concerning the composition of the participants served by the program, such as sex, racial, or ethnic group balance, we used stratified random sampling to select the treatment and control groups (and weighted sample members appropriately to account for different random assignment probabilities). Nonetheless, random assignment may have led some Upward Bound projects to serve students they would not normally have served. Before random assignment, we asked project directors to rate each applicant as either most likely, somewhat likely, or least likely to have been selected under normal selection procedures; in this report, we assessed whether the effects of Upward Bound vary across these three groups. Appendix I provides little evidence that the effects on postsecondary enrollment and completion varied across groups; however, there is evidence of significant positive effects on attendance and completion at vocational schools for the somewhat likely to be selected group.

With random assignment, the only systematic difference between the treatment and control groups in the present evaluation is that treatment group members were offered the opportunity to participate in Upward Bound; otherwise, the two groups are statistically equivalent (Myers and Schirm 1997). On important demographic variables such as gender, race or ethnicity, and Upward Bound eligibility status, differences between treatment and control proportions are small. Statistically significant differences between the two groups exist within two categories of background variables: a student’s own educational expectations and the educational expectations held by his or her mother. Even in a randomized experiment, there will generally be a few differences between the groups purely due to chance; using a 10 percent level for statistical significance, we would expect to find significant differences for 10 percent of the comparisons. To adjust for the small differences between the treatment and control groups, we computed regression-adjusted estimates of program effects in which we statistically controlled for these and other background characteristics. We describe our estimation methods in more detail below.

2. Outcome Measures

Selectivity of four-year colleges and universities attended was measured by using school ratings from Barron’s Profiles of American Colleges (2003). If a school was rated as “most competitive,” “highly competitive,” or “very competitive,” we classified the school as more selective. If a school was rated as “competitive,” “less competitive,” “noncompetitive,” “special,” or unrated, or was excluded from Barron’s, we classified the school as less selective. According to the classification system, more selective colleges and universities generally accept less than 75 percent of applicants, and students at more selective institutions were generally in the top half of their high school class. Less selective postsecondary institutions generally admit more than 75 percent of their applicants. The values of the four-year college or university selectivity outcome variables are set to 0 for sample members who did not attend a four-year college or university, that is, such sample members are classified the same as sample members who attended less selective four-year institutions.

To measure these postsecondary outcomes, we use data from the fifth follow-up survey, as well as from administrative records. We describe these different data sources below, along with their strengths and weaknesses in providing valid information for measuring these outcomes of interest.

3. Data Sources

The response rate for the treatment group was 4 percentage points higher than for the control group. Given this small difference in response rates, the differences between marginal treatment respondents—treatment group members who would not have responded if they had been assigned to the control group—and other treatment respondents would have to be very large to have any perceptible effect on the impact estimates. Furthermore, we use the extensive baseline data available to incorporate an adjustment for nonresponse into the sample weights.

After each follow-up survey, we also collected transcripts from high schools and postsecondary educational institutions attended by sample members. Following the fifth follow-up survey, we requested postsecondary transcripts for 2,079 sample members and received transcripts for 1,772 of them (85 percent); Appendix B describes the data collection procedures.

4. Construction of the Outcome Measures

We use the different data sources because they have different relative strengths and weaknesses. In conducting the impact analysis for this report, our basic principle has been to utilize the maximum amount of information that is available on the sample members. While the follow-up surveys provide data on a broad range of outcomes, we face the problem of not having

data for survey nonrespondents, and the nonrespondents might be systematically different from respondents, potentially leading to nonresponse bias in our estimates. The NSC and the FSA data are two convenient resources to mitigate this problem, as we can get information on both survey respondents and nonrespondents from these administrative records.

However, these administrative sources have their own limitations. The NSC does not cover the entire universe of postsecondary schools, and does not cover all member schools for the entire relevant time period. Nationally, current rates of coverage are 87 percent for students attending a two-year institution and 90 percent for students attending a four-year institution. The coverage rates were lower in earlier years (the NSC data go back to 1993–94); in terms of total U.S. college enrollment, coverage by the NSC data rose from 57 percent in 1997 to 88 percent in 2002, with small increases in subsequent years. Thus, the NSC might be missing data for a sample member who attended and potentially completed his or her education at a postsecondary institution because the institution was not covered by the NSC during the relevant years.⁴ FSA records provide data on all sample members; however, they do not have information on postsecondary completion, and they provide information on enrollment for only some students (those who receive a Pell grant).