students, improved work environment for teachers.
A new evaluation of a teacher pay-for-performance program in Little Rock finds that it produces significant gains in student performance on standardized tests and a more positive work environment for teachers. The Achievement Challenge Pilot Project at Meadowcliff and Wakefield elementary schools offers teachers and staff bonuses that vary based on the magnitude of increases in student achievement.
Gary Ritter, who holds an endowed chair in education policy at the University of Arkansas, will present the findings of this study at a news conference at 2 p.m. Tuesday, Jan. 16 at the State Capitol, Old Supreme Court Meeting Room. The full report is available at http://www.uark.edu/ua/der/Research/merit_pay.html
According to researchers in the UA department of education reform, these bonuses led to significantly greater learning gains than had been achieved by the same students prior to the program and by students at comparison schools. The introduction of merit bonuses produced an additional 3.5 normal curve equivalent points on a nationally normed math test. This gain in achievement after one year’s time is roughly equal to one-sixth of the test score gap between white and black students, on average, nationwide.
That is, if the observed benefit of the merit pay program were to compound for six years, it would close the black-white test score gap.
“As Arkansas policy-makers consider ways to recruit, retain and reward effective public school teachers, they may well consider the responses from the teachers in these Little Rock elementary schools,” said Joshua Barnett, a lead researcher of the study. “These teachers do reveal dissatisfaction with the status quo and a desire to be rewarded fairly for their work.”
In the merit pay program, teachers could earn a bonus worth as much as $11,000. In 2005-06, teachers and staff at Meadowcliff were awarded bonuses totaling $200,926, while those at Wakefield received $228,300 in performance bonuses. For the 2006-07 school year, the program was expanded to include three more elementary schools: Geyer Springs, Mabelvale and Romine. The program began at Meadowcliff in 2004-05, and Wakefield was added in 2005-06.
A survey of teachers at participating and comparison schools found a more positive work environment at schools where the merit bonuses were available. In addition, teachers at schools participating in the merit pay program were not more likely to report counterproductive competition among faculty.
The study was supervised by Ritter, director of the Office for Education Policy in the department of education reform. Jay P. Greene, who holds the endowed chair in education reform, and graduate students Joshua Barnett and Marcus A. Winters also participated in the evaluation. The research team will continue to examine the program next year with a larger sample size of students and teachers to determine whether these initial benefits persist. The program is funded by the Little Rock School District, the Little Rock Public Education Foundation, the Walton Family Foundation, the Hussman Foundation and the Brown Family Foundation. Financial support for the evaluation was provided by the Walton Family Foundation.
“The Achievement Challenge Pilot Project merit pay system should be considered as one policy option to improve the status quo and compensate teachers, and then be subject to rigorous evaluation to continually test its effectiveness,” Ritter said.
National Education Association Response:
The National Education Association issued a preliminary response Monday, Jan. 22, to a review of the beginning of a merit pay experiment for teachers in Little Rock elementary schools:
Synopsis and Review
“Evaluation of Year One of the Achievement Challenge Pilot Project in the Little Rock Public School District” by Joshua H. Barnett, Gary W. Ritter, Marcus A. Winters, and Jay P. Greene
The Study
This is the first installment of a multi-year study of elementary schools in the Little Rock School District that are participating in the Achievement Challenge Pilot Project (ACPP), a merit pay program. ACPP pays classroom teachers and school employees cash bonuses based on gains made by students on norm-referenced test scores for math, reading and language. The project started with funding from the Public Education Foundation of Little Rock (the Walton Family Foundation is among its donors), which continues as its major source of support. According to sources at the Arkansas Education Association, this study has the same financial backer, which raises the concern of how credible it is for the same entity that underwrites the merit pay program to be responsible for evaluating its effectiveness. Surely, an evaluation of the program could benefit from an independent analysis.
The ACPP program currently operates in five of Little Rock’s 34 elementary schools, with three of the schools beginning in 2006-07. The five schools all have high percentages of minority students and students who qualify for free and reduced lunches, as well as low academic achievement. This study encompasses two ACPP schools, one which started in 2004-05 and one that started in 2005-06. Three “comparison schools” were also included in the study.
The authors undertook two analyses:
1) a statistical analysis (fixed effects model) of changes in norm-referenced test scores for fourth and fifth graders. Due to “data limitations”, only math scores were used and only one of the two ACPP schools (along with all three comparison schools) was included in the study. _2) A survey of teachers, asking them about behaviors and attitudes related that the authors considered related to merit pay. Teachers in the two ACPP schools and three comparison schools were surveyed.
The Study’s Findings
The authors contend that:
1. Their statistical model (a “fixed-effects” model) estimated that “schools where the ACPP operated in 2005-06 showed an improvement of 3.5 normal curve equivalent points. For the average student, this gain represents an improvement of nearly 7 percentile points.”
2. The comparison of teacher surveys in the ACPP vs. non-ACPP schools suggested that that they: _- were no more innovative than comparison teachers._- were no more likely to work harder than comparison teachers._- were more satisfied with their salaries than comparison teachers._- reported no more counterproductive competition than comparison teachers._- their work environment became more positive than comparison teachers._- were less likely than comparison teachers to agree that low-performing students were a “burden” in the classroom._- were more likely than comparison students to report that the academic performance of their students had improved over the past year.
Overall, the authors concluded that “while the results from this first year study suggest positive impacts of the ACPP, we believe the second year study with five schools involved in the ACPP will greatly assist in expanding on and explaining the first year findings.”
Analysis
While headline of the press release announcing the study boldly states that says the “Report Finds Merit Pay Has Positive Effects,” upon closer examination there are several reasons to question whether the reported findings live up to the hype. Overall, it is important to keep in mind that these are preliminary findings based on incomplete data from the first year of a multi-year study. At best, it is premature to be calling ACPP a success in achieving any of its stated goals. More probably, it is irresponsible to make any claims about the effectiveness of ACPP on the basis of this study.
1. Modest Effects
One of the authors described the findings as “modest” and while encouraging to those who carried out the study, they “certainly wouldn't say we're thrilled by any stretch.” (Associated Press, 1/16/07)
2. Narrow Scope
The study is very limited in its scope. Due to “data limitations,” only math scores were included in the analysis (p.6). Reading and language test scores were omitted. Further, “due to the limits of available data,” the estimates are based on students in only one of the two ACPP schools. (p.7). Thus, “even though nearly 2,000 students attend the schools involved in the evaluation, approximately 500 students were part of the student achievement evaluation.” (p.12) These data limitations alone make the results highly questionable.
It is difficult for other trained statisticians to further evaluate the results of the study of the impact of ACPP on test scores because the specifications, equations, coefficients, and significance tests are not shown in the report and are only partially and briefly discussed.
_It is also worth noting that there is no discussion of each school’s status in school improvement plans under No Child Left Behind (NCLB) guidelines, which could be a very significant factor in affecting the behavior and attitudes of teachers in each of the schools. While the percentages of students rated as “proficient” in math and literacy tests were considered as criteria in selecting the comparison schools, the authors did not make clear the distinction between norm-reference tests (which were used in their analysis) and criteria-referenced tests that are used as benchmarks for determining school improvement status. Thus, according to sources at the Arkansas Education Association, a paradoxical situation exists where teachers in ACDD schools earned bonuses based on norm-referenced tests while their same schools were put on improvement plans based on the results of criteria-related tests.
3. Lack of causality
The comparison of survey data of teachers in schools with and without ACPP does not take into account any other factors that might explain differences between the two groups. While the authors conduct simple significance tests on averages for survey answers for schools in and out of the ACPP program, this alone is not sufficient to infer that these differences can be attributed to ACPP.
For example, the authors also imply that greater “satisfaction with salaries” can be attributed to the presence of merit pay while it may actually just be an effect of higher pay. It is not possible for teachers in ACPP schools to have pay deducted because of low test scores. It is only possible to receive a bonus on top of regular pay for improved test scores. Though the actual percentage of teachers receiving bonuses was not reported, it was reported that over was $200,000 in bonuses was divvied up in one ACPP school and nearly $230,000 was awarded in the other school in 2005-06. No additional pay was awarded in any of the comparison schools. Thus, any differences in pay satisfaction might be more attributable to higher pay than any incentive effect.
Setting aside this overall lack of causality, it is interesting to note that while the authors hypothesize that merit pay may “motivate current teachers to improve the performance of students through additional effort and innovation, where teachers work to learn and implement new effective teaching strategies,” their survey found no difference between the two groups of teachers in how “innovative” they were or in reporting that they worked “harder.” The authors do not discuss this apparent lack of support for key behavioral theories about why merit pay would lead to higher student achievement in the first place.
You have read this article with the title Report Finds Merit Pay Has Positive Effects. You can bookmark this page URL http://universosportinguista.blogspot.com/2007/01/report-finds-merit-pay-has-positive.html. Thanks!
No comment for "Report Finds Merit Pay Has Positive Effects"
Post a Comment