IV. The Effect of Upward Bound on Postsecondary outcomes
for selected subgroups
Impact estimates for the full evaluation sample might conceal important differences in impacts across subgroups. If Upward Bound has an effect on the average eligible applicant, it might be heavily concentrated in, or could be much larger for, certain subgroups of applicants. Conversely, if Upward Bound has no effect on the average eligible applicant, the program might still have positive effects for some subgroups, counterbalanced by negative effects for other subgroups. Thus, estimates of subgroup impacts can help policymakers identify the persons for whom a program is most effective and, in some cases, help policymakers better target a program or better tailor its services. However, this targeting or tailoring must be done with caution, as it is unknown what the effects of Upward Bound would be if the composition of students enrolled in Upward Bound or the services provided were different.
The subgroups for which we present estimates here in the main text are based on the following characteristics: (1) grade at application to Upward Bound; (2) educational expectations at the time of application to Upward Bound; (3) level of ninth-grade mathematics class; and (4) grade point average in ninth grade. Appendix I also presents estimates of effects for subgroups defined by gender, race and ethnicity, an academic performance index, the Upward Bound eligibility criteria, and the likelihood of a student being served under normal selection procedures (rather than random assignment), as well as project and project host characteristics. As before, when an impact estimate is statistically significant, that is, significantly different from 0, we highlight the estimate in the table with one or more asterisks (*). In addition, when the difference in impact estimates between two subgroups is statistically significant, we highlight the estimate for one of the two subgroups with a pound sign (#). Because a large number of outcomes and subgroups are considered, some effects will be significant by chance. We thus focus on a few key subgroups, and interpret results with caution; however, we make no formal adjustments for multiple comparisons. In light of the evolving approaches to program targeting that were discussed in Chapter I, it is probably most appropriate to interpret the subgroup analyses as exploratory rather than confirmatory, providing suggestive evidence for further investigation.17
Smaller sample sizes make it more difficult to obtain precise impact estimates. Therefore, for small subgroups, we are likely to detect significant impacts only if the effects are relatively large. When we report that the effect for a small subgroup is not detectable (not statistically significant), it may mean that Upward Bound has no effect on that outcome for that subgroup or that there is an effect but that it was not large enough to be detected given the sample size. The evaluation would have had to have substantially more projects and students to reliably detect small impacts for each subgroup and small differences between subgroups.
A. The Effect of the Opportunity to Participate in Upward Bound for Selected Subgroups (ITT) 1. Grade at Application to Upward Bound
The grade at which a student applies to participate in Upward Bound may give us some insight into the motivation and attitudes of a student. Those students who learn about the program and apply earlier may be more motivated to continue their pursuit of education beyond high school. In some ways, this may be an indication of the educational expectations of a student, based on their actions rather than responses to questions. Additionally, applying earlier provides the opportunity for greater exposure to Upward Bound, the benefits of which are discussed in more detail in the next chapter.
For the 54 percent of students who applied in ninth grade or earlier, Upward Bound had no detectable effects on postsecondary enrollment or completion, either overall or by level (Table IV.1). Upward Bound increased the likelihood of receiving a Pell Grant, from 52 to 58 percent, for this subgroup of early applicants.
Relative to the effects for eligible applicants who applied early in high school, our findings suggest that Upward Bound had more detectable effects on postsecondary outcomes for the eligible applicants who applied in tenth grade or later. For this subgroup of later applicants, there is a significant 3 percentage point increase in any postsecondary enrollment, from 82 to 85 percent. There are also significant increases in financial aid applications (from 73 to 77 percent) and certificate or license completion (from 3 to 9 percent). None of the differences between impacts for the early and late applicant groups is significant.
Earlier evaluation reports have found that Upward Bound has larger effects on the eligible applicants who did not expect to obtain a bachelor’s degree at the time they applied for Upward Bound than for those who did. The findings in this report are less conclusive in terms of statistical significance of impacts—there are no statistically significant differences between the group impacts—but still suggest that the program may have some more favorable effects on eligible applicants with lower educational expectations (Table IV.2).
For the 20 percent of eligible applicants with lower educational expectations—those who did not expect to earn a bachelor’s degree—Upward Bound had a detectable effect on overall postsecondary enrollment, increasing it from 70 to 75 percent, similar to the 6 percentage point
increase in enrollment in vocational institutions.18 Unlike previous findings, treatment group members with lower expectations were not significantly more likely to attend a more selective four-year college or university than were control group members with lower expectations; although the estimated rate at which they attended highly selective institutions nearly doubled, this impact is not significant. Findings for postsecondary completion differed from the pattern for enrollment: there were separate increases in four-year, two-year, and other degrees that were each not statistically significant, but the overall rate of completion for sample members with lower expectations rose by a significant 12 percentage points, and reached the rate for sample members with higher expectations.
Our findings suggest that Upward Bound also had a few detectable effects on postsecondary outcomes for the 80 percent of eligible applicants with higher educational expectations—those who expected to earn a bachelor’s degree when they applied to Upward Bound. For this subgroup, there is a significant increase in any postsecondary enrollment of 3 percentage points and a significant decrease in associate degrees of 4 percentage points. There is also a significant increase in the likelihood of receiving a Pell Grant.
3. Level of Ninth-Grade Mathematics Class
Targeting Upward Bound services to students based on their self-expressed expectations at the time of application to Upward Bound presents potential challenges. If Upward Bound projects began to screen applicants based on their educational expectations and it became common knowledge that applicants with lower expectations were more likely to be admitted than were applicants with higher expectations, applicants might face a strong incentive to understate their educational expectations in the application process.
In light of these practical limitations, we also examine whether Upward Bound has effects on subgroups defined by an alternative measure of academic expectations—the level of ninth-grade mathematics class taken.19 In particular, we define subgroups based on whether a sample member’s ninth-grade mathematics class was below the level of algebra or at the level of algebra or above. Students who do not take algebra or a course above algebra in ninth grade would not generally be considered to be on a “college track,” so this can be used as an indication of whether the student expects and is preparing to attend a four-year college or university. Among eligible Upward Bound applicants, approximately two-thirds took algebra or above in ninth grade and approximately one-third took a course below algebra.20
We find that Upward Bound has significantly different effects on overall postsecondary enrollment and enrollment at two-year institutions for the two subgroups (Table IV.3). Among sample members who took a course below algebra, treatment group members were nearly 7 percentage points more likely than control group members to enroll in any postsecondary institution. For sample members who took algebra or higher, the impact on overall enrollment is an insignificant decrease of 1 percentage point. The impact on attendance at two-year institutions is a significant 6 percentage point decrease for this subgroup, whereas it is an insignificant 2 percentage point increase for the sample members who took a course below algebra in ninth grade. There were significant increases in the likelihood of receiving a Pell Grant for both subgroups.
4. Grade Point Average in Ninth Grade
This measure divides sample members into two roughly equal-sized subgroups: those with ninth-grade GPAs greater than or equal to 2.5 and those with ninth-grade GPAs less than 2.5. The results reveal several significant differences in impacts between the subgroups, and indicate that Upward Bound has larger effects on enrollment and financial aid for eligible applicants with higher academic performance in ninth grade and larger effects on completion for eligible applicants with lower academic performance in ninth grade (Table IV.4).
For eligible applicants with higher ninth-grade GPAs (above 2.5), Upward Bound had significant effects on overall postsecondary attendance, attendance at two- and four-year institutions, and attendance at more selective four-year institutions. In this higher GPA group, treatment group members were 3 percentage points more likely than control group members to enroll in any postsecondary institution. There were also significant shifts by type of institution, with an increase in attendance at four-year institutions (8 percentage points) and a decrease in attendance at two-year institutions (7 percentage points). Treatment group members with higher GPAs were also 6 percentage points more likely to enroll in a more selective four-year institution. Among the higher GPA group, treatment group members were more likely to apply for financial aid and receive a Pell Grant, both by 6 percentage points. The impacts on 4-year college or university enrollment did not translate to higher rates of postsecondary completion, with overall completion falling insignificantly, driven by a significant four percentage point drop in associate degrees and no significant increase in the likelihood of earning a bachelor’s degree.
In contrast, Upward Bound had no detectable effects on postsecondary enrollment, overall and by type, on financial aid application and receipt, and on completion of bachelor’s or associate’s degrees, for eligible applicants with lower ninth-grade GPAs (below 2.5). However,
treatment group members with lower GPAs were 6 percentage points more likely than control group members with lower GPAs to earn a certificate or license but no higher degree, leading to a significant 7 percentage point increase in postsecondary completion overall.
|
Yüklə 1.39 Mb.