U. S. Department of Education Office of Planning, Evaluation and Policy Development

Study results

• 
  Upward Bound had no detectable effect on the rate of overall postsecondary enrollment or the type or selectivity of postsecondary institution attended for the average eligible applicant. About four-fifths of both treatment group members and control group members attended some type of postsecondary institution, including four-year institutions, two-year colleges, and vocational schools, and the estimated impact is an increase of less than 2 percentage points in the rate of enrollment (effect size = 4 percent). For enrollment at four-year colleges and universities, the estimated impact is 1 percentage point (effect size = 3 percent). These effects are not statistically significant.
  
• 
  Upward Bound had no detectable effect on the likelihood of applying for financial aid, or, the likelihood of receiving a Pell Grant. The 1 and 2 percentage point increases in the rates of financial aid application and Pell Grant receipt (effect sizes = 3 and 5 percent) are not statistically significant.
  
• 
  Upward Bound increased the likelihood of earning a postsecondary certificate or license from a vocational school. It had no detectable effect on the likelihood of earning a bachelor’s degree or the likelihood of earning an associate degree. While about 4 percent of control group members received a vocational certificate or license, nearly 9 percent of treatment group members did, implying an impact of 5 percentage points (effect size = 23 percent). The impacts on receiving any postsecondary credential and receiving a bachelor’s degree are 2 and 0 percentage points (effect size = 5 and 0 percent), respectively, and are not statistically significant.
  
• 
  Upward Bound increased postsecondary enrollment or completion rates for some subgroups of students. For the subgroup of students with lower educational expectations at baseline—that is, the students who did not expect to complete a bachelor’s degree—Upward Bound increased the rate of postsecondary enrollment and the likelihood of receiving a degree, license, or certificate by 6 and 12 percentage points, respectively, raising the overall postsecondary completion rate to about the level observed for students with higher expectations. Because targeting on the basis of lower educational expectations might be challenging if it creates an incentive for applicants to understate their expectations, further analyses were conducted to examine the effects of Upward Bound on subgroups that might be more readily targeted. According to these exploratory analyses, Upward Bound increased postsecondary enrollment rates for students who were in tenth grade or above at the time of application, students who took a mathematics course below algebra in ninth grade, and students with a ninth grade GPA above 2.5. The estimated impacts were 3, 7, and 3 percentage points, respectively. Additional analyses suggest that Upward Bound also had positive impacts on postsecondary outcomes for some other subgroups defined by student- and project-level characteristics.
  
• 
  Longer participation in Upward Bound was associated with higher rates of postsecondary enrollment and completion. An additional year of Upward Bound participation was associated with a 9 percentage point increase in the rate of enrollment at four-year institutions and a 5 percentage point increase in the likelihood of receiving a bachelor’s degree. Completing the Upward Bound program was associated with increases of 27 and 21 percentage points, respectively. These findings are based on nonexperimental methods, and the validity of causal inferences based on these estimates depends on the validity of strong assumptions.
  

One important design choice pertained to the length of the follow-up period for the evaluation. Considering the objective of Upward Bound to prepare students for entry into and success in postsecondary education, the Department of Education specified a long follow-up period that allowed sample members to be observed for many years beyond expected high school graduation. Although response rates to the evaluation’s follow-up surveys remained high, administrative data from the NSC and federal FSA files were obtained to assess and address the potential effects of survey nonresponse.

One set of sensitivity analyses examined alternative ways of combining data from the available sources—surveys, NSC, and FSA—to measure postsecondary enrollment and completion. While nonresponse is one potential limitation of survey data, measurement and coverage error are concerns with administrative data. Measuring postsecondary outcomes in different ways can shed light on how the relative strengths and weaknesses of the data sources affect the findings of the evaluation. For that reason, the sensitivity analyses examined 27 different measures of postsecondary enrollment. Estimates of Upward Bound’s impact on postsecondary enrollment across these 27 measures ranged from a negative 2.4 percentage points to a positive 2.8 percentage points, none of which were statistically significant.

The Upward Bound Sample Design Was Highly Stratified with Highly Variable Selection Probabilities

In designing the requirements for the Upward Bound evaluation sample, the Department of Education specified that the evaluation sample had to be nationally representative. It also required that the sample have substantial overrepresentation of some less common, but key types of projects, including, for example, projects serving predominantly Native American students. Alternative sampling schemes were considered, and a design was chosen to balance the competing needs of the evaluation. The chosen design had much higher selection probabilities for the relatively rare projects than for more common types of projects, leading to substantial undersampling and underrepresentation of the latter. This led to very unequal weighting of projects in the evaluation sample.

One implication of the sample design was that some of the most common types of Upward Bound projects had low selection probabilities and were substantially undersampled. This is true of one set of projects in particular—projects that were medium-sized, located in an urban setting, hosted by a four-year public institution, and not serving a group of students that is predominantly Asian, Native American, or Latino. This stratum of projects ends up accounting for about 26 percent of all eligible Upward Bound applicants nationwide. The final sample selected for the impact evaluation included only one project out of 56 projects in this stratum. The analysis weights the sample accordingly, and the sample members from this one project account for approximately 26 percent of the total weight.

Because one project and its students comprise such a large proportion of the weighted sample, two additional types of analyses were conducted. The first examined whether this one sampled project—labeled Project 69—is an outlier or unusual in any way. Data from a grantee survey sample on project-level characteristics found that Project 69 was similar to other projects in this stratum on a broad range of characteristics. Similarly, data from student surveys and NSC and FSA records indicated that the students from Project 69 did not have unusual characteristics.

The second type of analysis reduced the relative weight given to Project 69—in some cases by dropping the project entirely—when estimating impacts. The impact estimates were sensitive to substantial changes in weighting. Because Project 69 had below average impacts, reducing its weight relative to other projects resulted in larger overall impacts for most outcomes compared with the findings from the main impact analysis, which weighted all sample members according to their actual selection probabilities. Reducing the weight of Project 69 also underestimates the standard errors associated with the impact estimates. With larger impact estimates and reduced standard errors, many impact estimates become statistically significant when the sample weight for Project 69 is substantially reduced. When the standard errors more accurately reflect the precision of the sample design, many of these impact estimates are not statistically significant. Furthermore, impact estimates become smaller and fewer are significant when other projects with relatively large weights are dropped from the analysis along with Project 69. This illustrates an important consideration—the potential for influencing the findings through post hoc adjustments that deviate from the chosen design.

Another important consideration in interpreting results from analyses that omit Project 69 or otherwise change the weights of projects in any substantial way is that the resulting sample no longer represents the actual universe of Upward Bound projects. In particular, the sample does not appropriately represent the most common stratum of Upward Bound projects. Thus, such analyses do not answer the evaluation’s research questions about the impacts of the national Upward Bound program. Moreover, the estimates from such analyses do not generalize to urban projects, large projects, or any other well-defined subset of projects for which the findings might have policy implications.

In contrast, the findings from the main impact analyses, which include all projects weighted based on their selection probabilities, are intended to generalize to the national Upward Bound program. In assessing the implications of those findings, however, a statistical consideration is that as a consequence of selecting a single project from a large stratum—the stratum represented by Project 69—the estimates and inferences for that stratum and, therefore, the universe of projects will generally not be as robust as the estimates and inferences that would be obtained with an alternative design with much less variable project selection probabilities and with several projects selected from the large stratum. The lower robustness of the chosen sample design and the results from the extensive sensitivity analyses can be taken into account in determining the implications of the main findings.

Some Control Group Members Received Upward Bound or Upward Bound Math-Science Services and Some Treatment Group Members Did Not

After random assignment, project directors at some projects allowed a few control group members to receive regular Upward Bound. In addition, some control group members reported participating in the Upward Bound Math-Science program, which was not part of the random assignment evaluation. In total, 13.5 percent of the control group participated in either regular Upward Bound or Upward Bound Math-Science. In contrast, about 15 percent of students assigned to the treatment group did not participate in either program.

To account for this cross-over and no-show issue, the impact analysis estimated models of the effects of actual Upward Bound participation (as opposed to the opportunity to participate) on student outcomes, where Upward Bound participation is defined as receiving either regular Upward Bound or Upward Bound Math-Science. The impacts of actually participating are generally larger than the impacts of having the opportunity to participate. Most impact estimates are not statistically significant.