These pieces originally appeared as a weekly column entitled “Lessons” in The New York Times between 1999 and 2003.
[THIS ARTICLE FIRST APPEARED IN THE NEW YORK TIMES ON DECEMBER 13, 2000]
Judging Vouchers’ Merits Proves to Be Difficult Task
Monday’s federal appeals court ruling striking down Cleveland’s school voucher program had nothing to do with its educational merits. “Practical solutions to the problem of failing schools,” the ruling said, must respect separation of church and state.
But however the religious question comes out in a likely Supreme Court appeal, it is far from clear that vouchers are a practical solution. Evidence is sparse that low-income children gain from going to private schools.
Prof. Paul Peterson of Harvard University, a voucher advocate, has conducted creative research on them but does not find what most observers would consider a meaningful benefit. In New York City, black children’s average gain from vouchers was only four percentile points in two years. This performance is still considered “failing.”
Dr. Peterson and others have recently studied vouchers financed by philanthropists and corporations for low-income students to attend private, mostly parochial schools. The researchers have tried admirably to do experiments similar to medical drug trials. Of families who volunteered for private school scholarships, some were selected by lottery to receive them. The rest became a control group whose test scores could be compared with those of voucher recipients.
While the Peterson studies have not had normal academic peer review, Martin Carnoy, a professor of education at Stanford University, has done an informal review. Dr. Carnoy notes that in medical experiments, selection is “blind” – participants are unaware if they took a drug or placebo. Knowing if you are in a treatment or control group can subtly change behavior in ways researchers cannot observe. This can affect the drug’s impact.
Blind experiments are difficult to conduct in education. Parents know if their children got vouchers and this knowledge can affect results. For example, volunteers for vouchers, already more dissatisfied with public schools than others, may have their hopes raised, then dashed when they were not selected for a voucher. Sorely disappointed, they may then demand less of their children in public school. Meanwhile, those with a voucher know that its renewal might depend on their children’s performance. Thus, gains in scores of voucher recipients, compared with controls, may be exaggerated. With such small gains to begin with, positive results could disappear in a truly blind study.
Another problem is that the participation rate of the control group dropped in the Peterson studies. In drug trials, too, participants leave, and researchers cannot know if leavers differ significantly from stayers. But in valid drug trials, dropouts are fewer.
In the voucher experiments, leavers were numerous. Dr. Peterson based his conclusions on tests given to all volunteers before vouchers were issued, and then each year thereafter. But neither recipients nor controls were forced to take new tests. In Washington, only half did so. In New York City, it was two-thirds.
With such small gains to start with, this attrition could make results meaningless.
Dr. Peterson tried to estimate how those who did not show up would have performed. If students whose parents were high school dropouts did not show up, he assumed their scores would have been similar to children of dropouts who did take the test. Such adjustments are helpful, but do not solve the problem.
Most puzzling is the results’ inconsistency. Only black students made significant gains. Hispanic and white students with vouchers did no better than the public school control group.
Even more curious is that black students gained only in some grades in some cities. In New York, black students switching to private schools in the fifth grade made gains, but those switching in other grades did not. In Dayton, Ohio, students switching in sixth grade gained 17 points, but students switching in the eighth grade lost 15 points. Only in Washington were gains consistent across most grades. To report average gains poses the question for both policy makers and parents: With gains so arbitrary, how can one judge if vouchers would help?
Some scholarship students at private schools improve when surrounded by pupils with higher academic expectations. But if vouchers became public policy and more poor students attended private schools, positive peer influence would decline. If gains in score are small even with large peer effects, it is hard to see how a full voucher program could do much for overall achievement.
Interest in vouchers has often been based on a belief that they could solve the problem of failing urban schools. Certainly, more experiments like those of Dr. Peterson are called for. But if church-state problems are overcome, and vouchers still do not improve achievement, the attractiveness of this panacea may fade.
