A joint project of the Graduate School, Peabody College, and the Jean & Alexander Heard Library

Title page for ETD etd-10052009-210839

Type of Document Dissertation
Author Stuit, David Alan
Author's Email Address david.a.stuit@vanderbilt.edu
URN etd-10052009-210839
Title A comparative analysis of non-experimental methods for estimating the impact of magnet school enrollment on student achievement
Degree PhD
Department Leadership and Policy Studies
Advisory Committee
Advisor Name Title
R. Dale Ballou Committee Chair
Mark Berends Committee Member
Matthew G. Springer Committee Member
Thomas M. Smith Committee Member
  • selection bias
  • school choice
  • Quantitative research methods
  • magnet schools
Date of Defense 2009-09-25
Availability unrestricted
The objective of this dissertation is to understand the circumstances under which non-experimental methods yield unbiased estimates of the effect of magnet school attendance on student achievement. This dissertation has two main analyses. In the first analysis, non-experimental estimates (via multiple regression with observed covariates, analysis of covariance with student fixed effects, and propensity score matching) of the effect of attending one academically selective magnet school on 5th and 6th grade math and reading achievement are compared to experimental estimates found using lottery status as an instrumental variable (IV) for magnet school attendance. This analysis finds that multiple regression and propensity score matching yield estimates with sizeable positive bias that would likely lead a policymaker conclude the magnet school is more effective than it really is; in some cases, this bias represents over half a school year’s worth of learning. Student fixed-effects modeling performs the best of the non-experimental methods and in reading yields estimates of the magnet effect on 5th and 6th grade achievement that are not meaningfully different from the experimental IV estimates. The second analysis tests how well the experimental and non-experimental methods perform under various forms and rates of sample attrition. To investigate this issue I create a variety of samples with different forms and rates of artificial attrition among lottery winners, lottery losers, and non-participants and then run the experimental and non-experimental estimators on these simulated samples. The second analysis finds that the experimental IV estimates are less biased than the non-experimental estimates in almost all scenarios. The one exception is the student fixed-effects estimator, which performed as well or better than the experimental IV estimator as attrition rates exceeded 40%. Collectively, the findings raise caution against using multiple regression and propensity score matching to evaluate the causal impact of school choice programs, even under situations where attrition in the experimental sample is severe. Student fixed-effects modeling shows promise, particularly in reading, but only under high rates of sample attrition can one expect it to perform better than an analysis using randomly assigned comparison groups.
  Filename       Size       Approximate Download Time (Hours:Minutes:Seconds) 
 28.8 Modem   56K Modem   ISDN (64 Kb)   ISDN (128 Kb)   Higher-speed Access 
  Stuit_dissertation.pdf 713.65 Kb 00:03:18 00:01:41 00:01:29 00:00:44 00:00:03

Browse All Available ETDs by ( Author | Department )

If you have more questions or technical problems, please Contact LITS.