Problems With Value Added Assessment by Diane Ravitch

January 18, 2011 The Pitfalls of Putting Economists in Charge Of Education/Building Bridges Blog

Dear Deborah,

A few weeks ago, Mike Rose posted a list of his New Year’s resolutions. One of them was that we should “make do with fewer economists in education.

These practitioners of the dismal science have flocked to education reform, though most know little about teaching and learning.” Mike suggested that so few economists were able to give useful advice about the financial and housing markets that we should now be skeptical about expecting them “to change education for the better.”

 

I agree with Mike. It is astonishing to realize the extent to which education debates are now framed and dominated by economists, not by educators or sociologists or cognitive psychologists or anyone else who actually spends time in classrooms. My bookshelves are chock full of books that analyze the teaching of reading, science, history, and other subjects; books that examine the lives of children; books that discuss the art and craft of teaching; books about the history of educational philosophy and practice; books about how children learn.

Now such considerations seem antique. Now we are in an age of data-based decision-making, where economists rule. They tell us that nothing matters but performance, and performance can be quantified, and those who do the quantification need never enter a classroom or think about how children learn.

So the issue of our day is: How do we measure teacher effectiveness? Most of the studies by economists warn that there is a significant margin of error in “value-added assessment” (VAA) or “value-added modeling” (VAM). The basic idea of VAA is that teacher quality can be measured by the test-score gains of their students. Proponents of VAA see it as the best way to identify teachers who should get merit pay and teachers who should be fired. Critics say that the method is too flawed to use for high-stakes purposes such as these.

Last July, the U.S. Department of Education published a study by Mathematica Policy Research, which estimated that even with three years of data, there was an error rate of 25 percent. A few months ago, I signed onto a statement by a group of testing experts, which cautioned that such strategies were likely to misidentify which teachers were effective and which were ineffective, to promote teaching narrowly to the test, and to cause a narrowing of the curriculum.

None of these cautions has stemmed the tide of rating teachers by student test scores and releasing the ratings. Last year, the Los Angeles Times published an online database that rated 6,000 teachers as to their effectiveness (one of them, elementary school teacher Rigoberto Ruelas, committed suicide a few weeks later). New York City is poised to make a public release of the names and ratings of 12,000 teachers, if the courts give the go-ahead (in the first trial, a judge ruled that the data could be released even if it was inaccurate).

This trend did not just happen. It was encouraged by the Obama administration’s Race to the Top, which urged states to develop quantitative measures of teacher effectiveness. Secretary of Education Arne Duncan has issued statements endorsing the publication of teachers’ names and ratings, although few testing experts agree with this practice.

The bulk of studies warn about the inaccuracy and instability of these measures, but the Gates Foundation recently released a study called “Measures of Effective Teaching” (MET) that supports the use of VAA and VAM. As is customary for the Gates Foundation, it hired an impressive list of economists at institutions across the nation to give the gloss of authority to its work. Among its key findings was this one: “Teachers with high value-added on state tests tend to promote deeper conceptual understanding as well.” Ah, said the proponents of measuring teacher quality by the rise and fall of student test scores, this study vindicates these methods and effectively counters all those cautionary warnings.

But now comes a re-analysis of the Gates study by University of California-Berkeley economist Jesse Rothstein, which says that the MET study reached the wrong conclusions and that its data demonstrate that VAA misidentifies which teachers are more effective and is not much better than a coin toss. Even the claim that teachers whose students get high scores on state tests will also get high scores on tests of “deeper conceptual understanding” is flawed, writes Rothstein.

Ranking Teachers Riddled With Problems

Hurdles Emerge in Rising Effort to Rate Teachers

By SHARON OTTERMAN Published: December 26, 2010 NYTimes

For the past three years, Katie Ward and Melanie McIver have worked as a team at Public School 321 in Park Slope, Brooklyn, teaching a fourth-grade class. But on the reports that rank the city’s teachers based on their students’ standardized test scores, Ms. Ward’s name is nowhere to be found.

Melanie McIver, a teacher at Public School 321 in Park Slope, Brooklyn, with Elizabeth Phillips, background, the school principal. Both women have seen issues related to the city’s system of ranking teachers, which is at the heart of a lawsuit in State Supreme Court in Manhattan.

“I feel as though I don’t exist,” she said last Monday, looking up from playing a vocabulary game with her students.

Down the hall, Deirdre Corcoran, a fifth-grade teacher, received a ranking for a year when she was out on child-care leave. In three other classrooms at this highly ranked school, fourth-grade teachers were ranked among the worst in the city at teaching math, even though their students’ average score on the state math exam was close to four, the highest score.

“If I thought they gave accurate information, I would take them more seriously,” the principal of P.S. 321, Elizabeth Phillips, said about the rankings. “But some of my best teachers have the absolute worst scores,” she said, adding that she had based her assessment of those teachers on “classroom observations, talking to the children and the number of parents begging me to put their kids in their classes.”

It is becoming common practice nationally to rank teachers for their effectiveness, or value added, a measure that is defined as how much a teacher contributes to student progress on standardized tests. The practice was strongly supported by President Obama’s education grant competition, Race to the Top, and large school districts, including those in Houston, Dallas, Denver, Minneapolis and Washington, have begun to use a form of it.

Respected Teacher Commits Suicide Following LA Times Published Ratings of All Los Angeles Public School Teachers

Rigoberto Ruelas was rated “less effective than average” by the school district. This was published in the Los Angeles Times.

Teacher’s Death Exposes Tensions in Los Angeles

By IAN LOVETT Published: November 9, 2010 NYTimes

LOS ANGELES — Colleagues of Rigoberto Ruelas were alarmed when he failed to show up for work one day in September. They described him as a devoted teacher who tutored students before school, stayed with them after and, on weekends, took students from his South Los Angeles elementary school to the beach.

When his body was found in a ravine in the Angeles National Forest, and the coroner ruled it a suicide, Mr. Ruelas’s death became a flash point, drawing the city’s largest newspaper into the middle of the debate over reforming the nation’s second-largest school district.

When The Los Angeles Times released a database of “value-added analysis” of every teacher in the Los Angeles Unified School District in August, Mr. Ruelas was rated “less effective than average.” Colleagues said he became noticeably depressed, and family members have guessed that the rating contributed to his death.

On Monday, a couple hundred people marched to the Los Angeles Times building, where they waved signs and chanted, demanding that the newspaper remove Mr. Ruelas’s name from the online database.

“Who got the ‘F’? L.A. Times,” chanted the crowd, which was made up mostly of students, teachers and parents from Miramonte Elementary School, where Mr. Ruelas taught fifth grade.

The value-added assessments of teachers — which use improvements in student test scores to evaluate teacher effectiveness — has grown in popularity across the country with support from the federal Department of Education, which has tied teacher evaluations to the Race to the Top state-grant program.

But their use remains controversial. Teachers’ unions argue that the method is unfair and incomplete and have fought its implementation across the country.

The Los Angeles Times compiled its database using seven years of standardized test scores obtained through a public records request.

A. J. Duffy, president of the union, United Teachers Los Angeles, which helped organize Monday’s event, held up Mr. Ruelas as an example of the problems with value-added assessments.

“Value-added assessments are a flawed system,” Mr. Duffy said. “This was a great teacher who gave a lot to the community.”

The newspaper has refrained from commenting on the issue beyond a statement issued after Mr. Ruelas’s death: “The Times continues to extend our sympathy to Mr. Ruelas’s family, students, friends and colleagues. The Times published the database, which is based on seven years of state test scores in the L.A.U.S.D. schools, because it bears directly on the performance of public employees who provide an important service, and in the belief that parents and the public have a right to judge the data for themselves.”

Teachers’ unions have largely opposed moves away from the tenure system, in which layoffs are based on seniority, not performance.

Recently, in Washington, where the school chancellor, Michelle Rhee, used comprehensive teacher evaluations to fire hundreds of “ineffective” teachers, their unions poured hundreds of thousands of dollars into a campaign to unseat her main supporter, Mayor Adrian M. Fenty. Mr. Fenty lost the Democratic primary in September, and Ms. Rhee resigned the next month.

Despite opposition from the teachers union, Education Secretary Arne Duncan came out in support of greater transparency in teacher evaluations, and the New York City Department of Education is also preparing to release data reports on its teachers, pending the result of a court hearing later this month.

In Los Angeles, where the school district has moved toward significant reforms, like handing control of some chronically low-performing campuses to charter school operators, members of the school board have increasingly pushed to implement value-added assessments.

“Not including value-added measures is not acceptable,” said Yolie Flores, a board member of the Los Angeles Unified School District. “But it also has to be part of a more comprehensive system of evaluation.”

Eric A. Hanushek, a senior fellow at Stanford’s Hoover Institution who studies school accountability systems, said the value-added assessments should be combined with other factors. But he said the tenure system did not offer any meaningful evaluation of teacher performance.

“Now that The L.A. Times has published these scores, I think the genie is out of the bottle, and parents are going to want this information,” Mr. Hanushek said. “I presume the union’s opposition is a last effort of the teachers’ union to say that you should never evaluate teachers. This is their attempt to take a tragic situation and turn it into one that they can use for their own political advantage.”

But Randi Weingarten, president of the American Federation of Teachers, argued that reliance on value-added assessments actually hindered efforts to carry out comprehensive teacher evaluations.

“Our union has proposed a comprehensive system of teacher evaluation that more than 50 districts have adopted,” Ms. Weingarten said. “The good work we’re doing trying to make comprehensive teacher evaluations will actually be hurt by this fixation on a value-added system.”