B. The Effect of Participation in Upward Bound on Selected Subgroups (CACE)
In the subgroups defined by the grade at application and ninth-grade GPA, the pattern of statistically significant CACE effects is exactly the same as the pattern of statistically significant ITT effects. Generally, most of the patterns of effects are similar for subgroups defined by ninth-grade math course. The exception is that the CACE effect on the completion of a bachelor’s degree or higher is statistically significant for sample members who took a course lower than algebra in ninth grade. For the subgroups of eligible applicants defined by self-reported educational expectations, the pattern of effects of participation in Upward Bound is again very similar to the pattern of effects of being offered the opportunity to participate in Upward Bound.
Although the point estimate of the CACE effect is generally larger than the ITT effect estimate, the standard error also increases, leading to similar results with regard to statistical significance. The larger standard error (seen for many outcomes) of the CACE effect is attributable to the estimation of additional parameters, specifically those relating participation in Upward Bound to assignment to the treatment or control group.
V. The Effect of Additional Upward Bound Participation
on Postsecondary Outcomes
It is possible that students would reap larger benefits from Upward Bound if they spent more time in the program. The typical participant remains in Upward Bound for a little more than a year and a half, and most participants—61 percent—do not complete the program. In this chapter, we describe the relationships between postsecondary outcomes and two measures of the extent to which students participated in Upward Bound—the duration of program participation and program completion. In measuring these relationships, we attempt to estimate the effects of additional participation on postsecondary outcomes. As we describe later, the estimated effects of additional participation may overstate the true effects due to selection bias. With this caution in mind, our findings suggest that keeping students in Upward Bound for longer periods may substantially improve their postsecondary outcomes.
A. Research Questions
To assess the potential for keeping students in Upward Bound for longer periods of time, we classified participants as low-duration (1 to 12 months of participation), medium-duration (13 to 24 months of participation) or high-duration (25 or more months of participation), and also as program completers (still participating in the spring of senior year) or noncompleters. Among Upward Bound participants, 36 percent participate for 1 to 12 months, 29 percent participate for 13 to 24 months and 35 percent participate for 25 or more months; 39 percent complete the program (see Table V.1). These figures suggest that there is considerable opportunity to increase the completion rate and the length of time that participants remain in the program.
To better understand the potential effects of Upward Bound retention on postsecondary outcomes, we address two research questions:
1. For low-duration and medium-duration participants, how would their postsecondary outcomes change if they participated for an additional year?
2. For noncompleters, how would their postsecondary outcomes change if they completed Upward Bound?
B. Research Methods
To answer these questions, we compared the outcomes for students with relatively low levels of participation to those for students with relatively high levels of participation. A simple comparison of students with different levels of Upward Bound participation, however, may fail to reveal the effects of additional participation. While random assignment ensures that there will be no systematic differences between treatment and control students, it does not ensure there will be no differences between completers and noncompleters or among students who choose to participate in Upward Bound for different lengths of time. The characteristics of students may influence how long they choose to participate in Upward Bound and whether they complete Upward Bound. If so, the average characteristics of students will vary with the level of Upward Bound participation. For example, we found that students who participate in Upward Bound for longer periods of time are more likely to be female and have higher educational expectations than students who participate for shorter periods; we find similar differences between completers and noncompleters. Therefore, we cannot infer the effects of additional Upward Bound participation simply from differences in average outcomes between shorter- and longer-duration participants and between completers and noncompleters.
Because students with different levels of Upward Bound participation have different characteristics, we used statistical matching to select samples of shorter- and longer-duration participants with similar observed characteristics and samples of noncompleters and completers with similar characteristics. Unlike random assignment, which ensures two statistically equivalent groups that are similar in terms of observed and unobserved characteristics, the matching procedures can only create groups that are similar in terms of observed characteristics.
To measure the potential effects of participating in Upward Bound for an additional year, we matched low-duration participants to similar medium-duration participants, and we matched medium-duration participants to similar high-duration participants. These matches allow us to simulate what the outcomes of low-duration participants would have been had they instead been medium-duration participants and what the outcomes of medium-duration participants would have been had they instead been high-duration participants. Likewise, to measure the potential effects of program completion, we matched noncompleters to similar completers. The matching process ensured that matched samples contained participants with similar observed characteristics; we used propensity score matching to select the matched samples (see Appendix F for additional details).
Once matching was completed, we estimated program effects using the same methods employed throughout this report. For example, consider the comparison of noncompleters to similar completers. The completer-noncompleter difference in mean outcomes provides an estimate of the potential benefit of additional Upward Bound participation. Mean outcomes were regression-adjusted with the same logistic model used throughout the report to compute the impacts of the opportunity to participate in Upward Bound.21 Although the matched samples were observationally similar, they may differ in ways not revealed by the data collected for the evaluation. For example, our matched samples of completers and noncompleters may differ in their motivation to attend college. Unobserved differences between matched samples may bias the estimates presented in the next section.
|
Yüklə 1.39 Mb.