Problems With Value Added Assessment by Diane Ravitch

January 18, 2011 The Pitfalls of Putting Economists in Charge Of Education/Building Bridges Blog

Dear Deborah,

A few weeks ago, Mike Rose posted a list of his New Year’s resolutions. One of them was that we should “make do with fewer economists in education.

These practitioners of the dismal science have flocked to education reform, though most know little about teaching and learning.” Mike suggested that so few economists were able to give useful advice about the financial and housing markets that we should now be skeptical about expecting them “to change education for the better.”

 

I agree with Mike. It is astonishing to realize the extent to which education debates are now framed and dominated by economists, not by educators or sociologists or cognitive psychologists or anyone else who actually spends time in classrooms. My bookshelves are chock full of books that analyze the teaching of reading, science, history, and other subjects; books that examine the lives of children; books that discuss the art and craft of teaching; books about the history of educational philosophy and practice; books about how children learn.

Now such considerations seem antique. Now we are in an age of data-based decision-making, where economists rule. They tell us that nothing matters but performance, and performance can be quantified, and those who do the quantification need never enter a classroom or think about how children learn.

So the issue of our day is: How do we measure teacher effectiveness? Most of the studies by economists warn that there is a significant margin of error in “value-added assessment” (VAA) or “value-added modeling” (VAM). The basic idea of VAA is that teacher quality can be measured by the test-score gains of their students. Proponents of VAA see it as the best way to identify teachers who should get merit pay and teachers who should be fired. Critics say that the method is too flawed to use for high-stakes purposes such as these.

Last July, the U.S. Department of Education published a study by Mathematica Policy Research, which estimated that even with three years of data, there was an error rate of 25 percent. A few months ago, I signed onto a statement by a group of testing experts, which cautioned that such strategies were likely to misidentify which teachers were effective and which were ineffective, to promote teaching narrowly to the test, and to cause a narrowing of the curriculum.

None of these cautions has stemmed the tide of rating teachers by student test scores and releasing the ratings. Last year, the Los Angeles Times published an online database that rated 6,000 teachers as to their effectiveness (one of them, elementary school teacher Rigoberto Ruelas, committed suicide a few weeks later). New York City is poised to make a public release of the names and ratings of 12,000 teachers, if the courts give the go-ahead (in the first trial, a judge ruled that the data could be released even if it was inaccurate).

This trend did not just happen. It was encouraged by the Obama administration’s Race to the Top, which urged states to develop quantitative measures of teacher effectiveness. Secretary of Education Arne Duncan has issued statements endorsing the publication of teachers’ names and ratings, although few testing experts agree with this practice.

The bulk of studies warn about the inaccuracy and instability of these measures, but the Gates Foundation recently released a study called “Measures of Effective Teaching” (MET) that supports the use of VAA and VAM. As is customary for the Gates Foundation, it hired an impressive list of economists at institutions across the nation to give the gloss of authority to its work. Among its key findings was this one: “Teachers with high value-added on state tests tend to promote deeper conceptual understanding as well.” Ah, said the proponents of measuring teacher quality by the rise and fall of student test scores, this study vindicates these methods and effectively counters all those cautionary warnings.

But now comes a re-analysis of the Gates study by University of California-Berkeley economist Jesse Rothstein, which says that the MET study reached the wrong conclusions and that its data demonstrate that VAA misidentifies which teachers are more effective and is not much better than a coin toss. Even the claim that teachers whose students get high scores on state tests will also get high scores on tests of “deeper conceptual understanding” is flawed, writes Rothstein.

The correlation between the two tests was actually modest: About 29 percent of the teachers in the bottom quintile on the basic skills tests were rated above average on the tests of reasoning and critical thinking; these are the teachers who would be fired if the Gates Foundation had its way.

 

“Interpreted correctly,” writes Rothstein, the analyses in the Gates’ report “undermine rather than validate value-added-based approaches to teacher evaluation.” Jesse Rothstein is not just any economist; not only has he studied VAA in the past, but he served as senior economist for President Barack Obama’s Council of Economic Advisors and chief economist at the U.S. Department of Labor. So he is well-equipped to take on the entire stable of Gates-funded economists, mano a mano.

 

For another take on these issues, I recommend a lively blog debate between economist Dan Goldhaber and teacher John Thompson. Thompson warned Goldhaber that economists should think seriously about the damage their methods will wreak on teachers and children and schools, especially low-income schools, where it is harder to get big test-score gains. In one of Thompson’s perceptive comments, he wrote, “If teachers have a one-sixth, one-fifth, or one-third chance per year of being wrongly indicted as ineffective, none will ever have any peace of mind. Effective teachers with self-respect will flee those schools for lower-poverty schools.” I was reminded of a comment by Rutgers economist Bruce Baker, who asked whether you would buy a car if the salesman assured you that it would explode only once every five times you turned the ignition key.

 

If we step back a bit, Deborah, don’t you think there is a certain kind of madness in thinking that economists who never set foot in a classroom can create a statistical measure to tell us how best to educate children? It seems some will never be satisfied until they have a technical process to override the judgments of those who work in schools and are in daily contact with teachers and children. I don’t know of any other nation in the world that is so devoted to this effort to turn education into a statistical problem that can be solved by a computer. It is not likely to end well.

Diane

Posted by Diane Ravitch