U. S. Department of Education Office of Planning, Evaluation and Policy Development
|
|  Yüklə 1.39 Mb.
|
III. The Effect of Upward Bound on Postsecondary outcomesThe ultimate question for this study is whether the national Upward Bound program helps disadvantaged high school students attend and graduate from institutions of higher education. To address that question, we used information from the fifth follow-up survey, along with data from previous surveys, the National Student Clearinghouse (NSC), and Federal Student Aid (FSA) records. Using these data, we compared the postsecondary experiences of eligible applicants offered the opportunity to participate in Upward Bound (the treatment group) with the experiences of eligible applicants not offered that opportunity (the control group), over a period extending to about seven to nine years after they were scheduled to graduate from high school. These estimated effects are “intended to treat” (ITT) effects. Some sample members end up not “complying” with their experimental assignment: some treatment group members do not actually participate in Upward Bound, and some control group members do participate in Upward Bound (or Upward Bound Math-Science). Therefore, we also present estimates of the effects of actually participating in Upward Bound. These are “complier average causal effect” (CACE) estimates. This chapter presents these two kinds of estimates to assess Upward Bound’s effects on three main sets of outcomes relating to postsecondary educational attainment:
Our analyses of impacts on these outcomes drew on multiple data sources. We used information collected in follow-up surveys during 1998–99, 2001–02, and 2003–04, along with data from NSC and FSA administrative records. There are many ways to combine data from these sources to measure postsecondary enrollment or completion, and many different assumptions that can be made about enrollment or completion status when the data do not provide definitive evidence. Each approach reflects choices about how to resolve issues that arise from the nature of the data: whether to use data from previous surveys or only the latest,; which data sources to use, in which “order of preference” to combine the data when there are apparent contradictions; how to deal with cases for which the data do not provide definitive evidence of enrollment or completion; and whether to use all available data or only data covering a certain period. A detailed discussion of these issues is presented in Appendix B. The impact estimates presented in detail in this chapter are for one measure for each outcome, but other estimates are presented in Appendix C. The main estimates presented in this chapter are based on the fifth follow-up survey data along with NSC and FSA administrative records. The alternative estimates emerged from sensitivity analyses examining different measures of the outcomes based on different methods and assumptions. Additional sensitivity analyses pertaining to the weighting of the evaluation sample are presented in Appendix G. Findings from all of the sensitivity analyses are summarized at the end of this chapter. A. The Effect of the Opportunity to Participate in Upward Bound (ITT)As documented in detail in this section, Upward Bound had no detectable effect on overall postsecondary enrollment for the average eligible applicant, and did not affect the types of institutions eligible Upward Bound applicants attended, either in level or selectivity. Similarly, Upward Bound did not have a detectable effect on financial aid application or receipt. However, there is evidence that Upward Bound increased the completion of certificates or licenses. T With Measure 5B for enrollment, we code a sample member as an enrollee if he or she is found to be an enrollee in the full NSC data or is a Pell recipient according to the FSA data or said in the survey that he or she was enrolled at some time. The sample member is not an enrollee if he or she does not appear in the NSC data (and is therefore not an enrollee) and has not been a Pell recipient and said in the survey that he or she had never been enrolled. This leaves uncoded the survey nonrespondents who are not in the NSC data and did not receive a Pell grant. For them, we assume that they are not enrollees if they never applied for financial aid. If they did apply for financial aid, we code their enrollment status as missing. The sample members with missing enrollment status get dropped from the analyses of enrollment, and weights for the remaining sample members are adjusted to compensate, as described in Appendix A. For measuring completion, the FSA data do not provide information on the actual receipt of degrees, certificates, or licenses. Recognizing this limitation in constructing Measure 7B for completion, we code a sample member as a completer if he or she is a completer according to the full NSC data or said in the survey that he or she has completed a degree, certificate, or license. The sample member is not a completer if he or she has no evidence of completion in the NSC data and said in the survey that he or she had not completed a degree, certificate or license. This leaves uncoded the survey nonrespondents who have no evidence of completion in the NSC data. For them, we assume that they are not completers if they never applied for financial aid. If they did apply for financial aid, we code their completion status as missing. The sample members with missing completion status get dropped from the analyses of completion, and weights for the remaining sample members are adjusted to compensate. See Appendix A for more information on weighting, Appendix E for more information on estimation, Appendix B for a discussion of alternative measures of postsecondary enrollment and completion, Appendix C for analyses using the alternative measures to assess the robustness of our findings, and Appendix G for analyses pertaining to the weighting of the evaluation sample. 1. The Effect of Upward Bound on Postsecondary Enrollment and College SelectivityUpward Bound attracts students who are much more likely to enroll in postsecondary education than are similarly disadvantaged students. A comparison of overall postsecondary enrollment rates for Upward Bound applicants and students nationwide shows that Upward Bound applicants are just as likely to enroll in postsecondary education as the average eighth-grader in the United States, regardless of socioeconomic status. Serving this applicant pool of motivated disadvantaged youths, Upward Bound did not have a detectable effect on enrollment in postsecondary institutions within approximately seven to nine years after scheduled high school graduation. Approximately 81 percent of treatment group members and 79 percent of control group members attended some type of postsecondary institution (four-year, two-year, or vocational). The difference, which is the effect of the opportunity to participate in Upward Bound, is not statistically significant (effect size = 4 percent). Similarly, the opportunity to participate in Upward Bound did not significantly affect the type or selectivity of postsecondary institutions attended by eligible applicants. 2. The Effect of Upward Bound on Financial Aid Application and Pell Grant ReceiptAlthough academic improvement is the primary focus of most Upward Bound projects, addressing financial barriers is also important. Nearly one-half of all Upward Bound project directors reported in a survey of grantees that inadequate financial resources are the most serious obstacle to college completion (Fasciano and Jacobson 1997), and many projects aimed to link students to financial aid resources. Despite such efforts, Upward Bound did not have a detectable effect on the percentage of eligible Upward Bound applicants who at some point applied for financial aid or received a Pell Grant for postsecondary education. 3. The Effect of Upward Bound on Postsecondary CompletionUpward Bound had no detectable effect on the likelihood of completing a postsecondary credential in the seven to nine years after high school (effect size = 5 percent). The program did increase the percentage of sample members whose highest credential was a certificate or license, from 4 to 9 percent (effect size = 23 percent). It is possible that sample members could obtain a credential later. Approximately 7 percent of both treatment and control group members were still enrolled at the time of the survey. |
able III.1 presents the effect of the opportunity to participate in Upward Bound—the ITT impact—on a range of postsecondary outcomes. Each row in the table presents the results for one outcome, with columns for the regression-adjusted