The following is the testimony of Mark Simon, EPI research associate, in a hearing before the DC Council’s Education committee on the National Research council of the National Academies Summary Report: “An Evaluation of the Public Schools of the District of Columbia: Reform in a Changing Landscape” on Monday, June 22, 2015.

Chairman Grosso and Members of the DC Council:

My name is Mark Simon. I have been a long-time active DCPS parent, a lifelong educator, an education policy analyst at the Economic Policy Institute, and I’ve been before you in the past on the DCPS budget, teacher evaluation and other topics.

The Summative Evaluation from the National Research Council opens the door to a serious conversation based on its analysis of the effects of PERAA changes in school governance since 2007. They took their role as outside evaluators seriously, bringing a knowledge base of national research and best practices. However they admit that their role was limited to answering whether the PERAA law was implemented as intended and what the results seem to be. It is up to us to take their critique and documentation, and to draw conclusions about whether the strategies being pursued under PERAA are bearing fruit. We have to go beyond the limited purpose of the report, using their very useful attempt to document results.

On the subject of Mayoral Control they state: “A change in governance structures does not deliver educational results on its own.” It is “shaped not just by the person the mayor chooses to lead the day-to-day operations of the school system but also by how much attention the mayor chooses to devote to the schools.” They pointed out that research on urban school reform in 11 major U.S. cities has shown that the effectiveness of mayoral control depends on forging “ongoing relationships with disparate groups, ranging from business elites and labor unions to grassroots community activists, and to draw on broader civic capacity.” If done right, mayoral control can bring:

• Increased school accountability
• Better citywide coordination
• Better mobilization of civic capacity on behalf of schools
• Better student results, particularly for the neediest kids

The report then systematically documents how PERAA, as it was implemented by DCPS, by the charter sector, by deputy mayors of education, and by OSSE, has actually failed to achieve each and every one of these four goals.

The conceit of the report is to cloak findings in language stating that although there are big problems, progress is being made. But the descriptions of the problems go to the core of the original stated goals of the governance reforms. Stated “progress” provides thin cover to the more dramatic revelations about failure of the specific strategies employed to meet the goals:

1. Accountability – Much data has been kept but the agencies in charge of keeping it have frequently not provided it to the NRC researchers. Data has been withheld from the public and only analyses that city agencies have chosen to focus on, like the percentage of students above and below proficiency, is readily available. OSSE in particular, was skewered by the evaluators as, if not outright incompetent, then at least an overstaffed agency dedicated to spin, not accountability or transparency of results.
2. Better Citywide Coordination – The advent of 60 Local Education Agencies (LEAs) and a reluctance on the part of city government to exercise the authority they have over the charter sector has left us less coordinated, with less transparency about what different schools are doing with kids and whether the conditions of learning are getting better since PERAA. Lines of authority between the DME, Chancellor, State Superintendent and Charter Board have been confusing, creating the opposite of what was promised.
3. Mobilization of Local Civic Capacity – The different decision-making players have each exhibited a go-it-alone approach, with chancellors, the PCSB, and OSSE each separately circling the wagons and considering anyone not on the inside with them an enemy. What the reviewers didn’t say is that parents and education activists in particular seem to have been considered enemies, compared with how the public had been engaged previously. Local universities and the unions representing teachers, principals and others have all been considered the enemy. Instead, experts from national consulting firms, often with little or no experience in education have been brought in to advise on policy.
4. Improved Student Achievement – Student results judged by test scores show no progress for the city’s neediest students and students of color – the vast majority of our youth – with a widening achievement gap based on race, class and ward. Confounding factors related to changing demographics – the city becoming whiter and richer – and changes in the way data has been kept (since 2012 non-poor students have been counted as poor in schools with over 40% FARMS rates), may make the true results even worse.

What were the reform strategies implemented under the high degree of control provided by PERAA and are those specific strategies working or are they problematic in some way?

The reviewers are clear that their mission was to look at whether expectations of the PERAA law have been met and whether changes have led to improved coordination, efficiency and accountability; the extent to which the actions of school leaders were consistent with research and best practices; and changes that have occurred in the conditions for learning in schools and outcomes for students 7 years after the governance changes enacted under PERAA. They specifically did not draw conclusions about the efficacy of the education reforms enacted under those governance changes. However, their evaluation document does provide evidence that sheds light on the reform strategies that have been pursued.

1. A Growing Charter Sector – According to the NRC Reviewers PERAA covers the charter sector and they devote significant space to the problems of accountability and coordination between the two sectors. PERAA defined the responsibilities of the PCSB at a time when there were a small number of charter schools. The reviewers ask the question whether the roles, responsibilities and accountability of the PCSB need another look now that they are in charge of more than half the schools.
2. IMPACT – The centerpiece strategy for DCPS has been the IMPACT teacher evaluation system. It is used for two primary purposes: as the basis for additional compensation and as the basis for dismissal. The reviewers used only quantitative measures to analyze the results to see if IMPACT is working to retain effective teachers and get rid of ineffective ones. Their conclusion is that there is some statistical evidence that more highly effective teachers are staying and more ineffective teachers are leaving, but that is a measure of correlation, not causation. No one bothered to ask teachers what they think of IMPACT. The reviewers also found problems with their own method of analysis:

• Changes in IMPACT since its initial implementation make it hard to determine whether teacher effectiveness ratings have improved over time.
• One-year Value Added scores are used which testing experts say are less reliable
• There is little quality control in the judgement-based ratings
• IMPACT observation rubrics include a limited range of classroom practices
• Teachers in high poverty schools get significantly lower IMPACT scores than teachers in low-poverty schools which may mean there’s an evaluation bias, not that worse teachers teach in more challenging schools.

What’s missing from the reviewers analysis is any way of determining whether IMPACT is improving the actual quality of teaching and learning in the school system. No attempt was made to get a subjective assessment of whether teachers feel they are benefitting from IMPACT or whether IMPACT has had a positive or negative effect on keeping good teachers or helping struggling ones. The only DCPS personnel interviewed by the evaluators were three Associate Superintendents who are in charge of supervising principals. Meanwhile,

• One third of the teachers granted bonuses in the first year refused them and a significant number still refuse their bonuses.
• The WTU conducted a survey in 2012 and over 90% of teachers wanted IMPACT ended.
• There is little coordination between IMPACT and school-based coaches charged with teacher training and improving instruction
• IMPACT master educators observe between 80 and 100 teachers as clients, ensuring that they are able to provide little or no support, only judgements.
• The investment in IMPACT for central office-based Master Educators and bonuses for teachers rated highly effective is huge. It was subsidized by foundation grants for the first three years but now competes with other school needs and staffing.
• None in the charter sector has adopted IMPACT as their model

Several of these factors were mentioned in the NRC conclusions, but did not lead to summative conclusions about the success or failure of the approach in IMPACT. As Vicki Phillips, Director of the Gates Foundation’s education programs says, ”If a teacher evaluation system is not valued and trusted but teachers themselves, it fails.”

3. Centralized Budget and Human Capital Control in DCPS/ Highly Decentralized Unaccountable Charter Schools – Under PERAA, the two sectors have been able to engage in opposite extremes of central control and de-centralized laissez-faire. Neither one has built dramatically improved conditions of learning, and neither one has achieved significant improvement in student achievement. The emphasis has been on degrees of managerial control and governance, not on the more important issues of how best to respond to student needs that may be different in different Wards.

According to the NRC reviewers, after eight years of PERAA:

• “The charter sector did not (and does not) have any overarching strategy to improve teacher quality (or any other factor in education)”
• Most of the factors the NRC reviewers wanted to examine could not be evaluated in either sector because neither sector collects the information the reviewers requested on the conditions of learning.
• As of 2014 “the US Department of Education reports that D.C is still among the worst school systems in the nation in providing appropriate educational opportunities for students with disabilities.”
• Still, in 2014, DCPS has a “crisis of school absenteeism” that has been obfuscated by shoddy record-keeping by OSSE, the PCSB and DCPS. Charter schools still rely on expulsions and suspensions as discipline strategies, and DCPS seems to report only incidents involving weapons and violence.
• Early childhood education is a clear area of success for D.C, but those investments in pre-K for 3 and 4 year began long before PERAA and so cannot be credited to it.
• AP Courses have grown under PERAA but highly inequitably across schools and Wards. OSSE seems to be skewing or obfuscating the data on the percentage of students scoring 3 or higher on AP exams and the quality control of courses labeled AP across the Wards.
• “DC would benefit from having much more systematic information about students who struggle in school at every level, beginning with pre-K and the ways schools support them.”
• The new weighted funding formula enacted by the D C Council in 2014-15 is a promising effort to finally pay more attention to needy students. But it is making up for the fact that this has not been an emphasis under the first seven years of reform.
• “Most of the Ward Level information we were able to find was from sources other than the city.” This statement in the report speaks volumes about the lack of emphasis in the era of PERAA reforms on responding to very different needs across the socio-economic divides in our city. It just has not been an emphasis.

The reviewers had to recommend in their conclusions that the city: ”…maintain and make publicly available, systematic data for three topics:

1. Students with particular needs, including those with disabilities, English-language learners, and students in poverty.
2. School climate, including discipline, attendance, safety, and facilities: topics to monitor include trends over time; the nature and magnitude of problems, distribution of problems across schools, Wards, and LEA’s etc.
3. Academic supports for learning: topics to monitor include equity of access to rigorous coursework at all grade levels; access to supports for struggling students; etc.“

The fact that the reviewers had to make these recommendations is itself an indictment of the lack of emphasis on these things in the theory of action under DC’s PERAA reforms.

• Missing from the NRC Report is any mention of facilities modernization. Concurrent with PERAA reforms but not part of it, and outside the control of the education agencies. It has been the city’s first major school modernization campaign, radically improving school facilities, sports fields and recreation facilities across the city. This likely had more of an impact on the conditions of learning and the return of students to DC Public Schools than any of the factors the NRC did look at under PERAA.

4. A relentless focus on short term strategies to improve proficiency rates in student standardized test scores – The reviewers failed to identify this as a reform strategy, but the emphasis on getting proficiency rates up under chancellors Rhee and Henderson has transformed what is taught, how it is taught and has consumed the attention of educators. It is itself a strategy that, judging by the results, has failed.

• The reviewers criticize the “many downsides of focusing on a single point in the score distribution.” But they fail to adequately understand the degree to which schools have focused on “the bubble kids” to the exclusion of meeting the needs of more needy students. Reporting and rewarding the single factor of proficiency rates distorts the picture of learning, creating an illusion of general improvement that can be subject to manipulation. The focus on proficiency rates may have contributed to the growing achievement gap by ignoring progress below the level of proficiency by the neediest students.
• The reviewers did discuss the controversy within the educational testing research community over the use of Value Added Measures (VAM) but in the end took no position. In fact, there is now a near consensus among education testing experts that attaching the kind of stakes DCPS has attached to VAM scores, particularly with just one year of data, should not be done.
• Although the Reviewers take up the issue of alleged cheating in 2009-10, they largely accept DCPS’ contention that “an investigation was done.” In fact, the “investigation” was so limited in scope that it’s fair to say that there was not a real investigation. The reviewers ignored the evidence from the USA Today investigation, McGraw Hill analysis, and other sources that cheating by adults likely took place in 103 schools. They also fail to examine the supposedly legitimate strategies schools use to get test scores up in the short run. When students test scores show an inaccurate picture of their actual abilities, the repercussions can be disastrous, on everything from teacher evaluation to choices about curriculum or whether and how to respond to student needs that might have little impact on test scores. The reviewers did attempt to take the issue up, but in the end, they accepted OSSE’s assertion that “the alleged violations were likely not widespread enough to have affected citywide scoring levels, which removed the primary reason for our concern.” This may have been the greatest failure of the NRC Final Evaluation Report.
• Not having identified the focus on standardized tests as itself a strategy, the NRC Final Report misses an inevitable conclusion that needs to be drawn. The reviewers say “The outcomes we examined present a mixed picture of students’ achievement in DC,” with “noticeable achievement in math scores” and “less progress in reading.” In fact, progress had been more robust in the years before PERAA. The achievement gap based on race and class has widened in reading and math under PERAA. A more honest look at the focus on standardized tests would have to conclude that the wrong strategies have been used, not, as the reviewers seem to contend, that “DCPS and PCSB have made choices that show promise.”
• Other important measures have not been the focus. “Graduation rates have fluctuated in the years since PERAA” (read: progress has not been made. DC’s Graduation rates continue to be some of the lowest in the nation.) And still, “The D.C. (test score) rates remain disturbingly low for black and Hispanic students, those with disabilities, and those eligible for free or reduced-price lunches, and English-language learners.”
• Lastly, the report is replete with examples of how the reviewers’ requests for data were ignored or they were told the data they wanted did not exist. In terms of measures of student achievement, the reform strategy under PERAA seems to have attempted to narrow the measures, restrict access to data, and actually limit the public’s right to hold the system accountable. The reviewers failed to mention the fact that the PERAA law was amended in 2010 to eliminate language requiring annual reports to the DC Council and striking language requiring a summative evaluation in 2012. This report is three years late. These amendments to the law were made, without public debate or discussion, by former Mayor Vincent Gray when he was chairman of the DC Council as part of the routine budget process in 2010. From start to finish, therefore, PERAA reforms in DC Public education have hindered the goal of transparency and accountability to the public. With their emphasis on how difficult it was to get data, their powerful criticisms of OSSE, and their recommendations that new data be kept and made publicly available, this report makes an important contribution to the public’s right to know about public education reform in our city.

Where do we go from here?

The question is not whether any progress is being made or whether the good news outweighs the bad news in the results of PERAA reforms. The question is whether the PERAA law has been used to implement the right reforms, whether modifications are in order, or whether a different approach would have achieved better results. One comes away from the NRC report not really knowing the answer, but rather with a list of actions the reviewers recommend to gather better and more independent data analysis, to begin better citywide coordination and planning, and to engender greater public engagement and trust. The Council, the Mayor, DCPS, OSSE, and the PCSB would do well to support the following next steps.

1. Engaging the Public – At the very least, the DC Council needs to take the lead in convening stakeholders and community activists who have been marginalized and denied information under the highly centralized decision-making process that PERAA has allowed over the last eight years. The Council and the Mayor need to institutionalize a continuing advisory group of ward based representatives of parents, teachers, and community stakeholders as was recommended in the final report of the Student Assignment Work Group chaired by the DME in 2014. The DC Council needs to implement, or instruct the DME to implement this plan. The task of assessing whether the reforms are working is still before us, and the voices of stakeholders are crucial. What do parents, educators, and community activists think about whether the stakes attached to standardized test scores are doing more harm than good? Is the design of IMPACT right? Should there be a limit on the continued growth of the charter sector and the unfettered authority of the PCSB? The NRC report makes clear the DC Council allowed the PCSB that unfettered control when it had the power to do otherwise.
2. Transparency of Data – The NRC Report makes very specific recommendations about changes in what data is collected and how that data needs to be made transparent for other people’s analysis. The DC Council needs to ensure that this happens immediately. OSSE, DCPS, the DME and PCSB, have assiduously avoided collecting certain kinds of data and have been more interested in spin than accountability to the public. Recent efforts by OSSE and the charter sector to limit data that would allow researchers to evaluate IMPACT flies in the face of the NRC recommendations to make more and better data about IMPACT available. Privacy considerations must be surgically addressed allowing for maximum data to be available to those interested in determining the efficacy of the still highly controversial IMPACT program.
3. Honesty About Inequities Based on Income, Race, and Geography – The inattention to learning conditions in schools, the inequity in conditions across Wards, and the lack of focus on the neediest students all need to be re-thought. The DC Council must make clear their expectation that those in charge take a more honest approach to poverty and make addressing unequal learning conditions a priority.
4. Citywide Coordination and Planning – There needs to be a new coordinating body across the two sectors and between the various agencies. Whether and where new schools are opened cannot be a decision that the PCSB or DCPS make on their own. We have a crisis of lack of coordination! There needs to be a moratorium on school location decisions that could negatively affect communities or existing schools until the planning and coordination functions are structured, the authority is clear, and elected bodies accountable to the public have control. The NRC reviewers imply that OSSE is not capable of playing such a role, but they state clearly that a coordination center is needed. It is up to the citizens of the District through their representatives in the DC Council and the Mayor to sort this out and to make it happen.

This NRC Summary Report is a goldmine of long overdue independent analysis. Teachers, principals, and central administrative staff work incredibly hard in DC’s public schools, particularly those who work with the neediest students. Honesty and transparency about whether policies and programs are working is essential if the school systems are to engender trust and commitment from those who work with students. As the NRC reviewers state over and over in this report, much of the data they looked for was not available and the analysis and conclusions that need to be dawn await further study. This summary report is inconclusive. On the one hand, the reviewers conclude that “DCPS and PCSB have made choices that show promise.” But on the other hand the report concludes that accountability, transparency, coordination and trust are seriously lacking. The achievement results for low income, African American and Latino students are disappointing. This first independent analysis of the 2007 education DC education reforms has opened the door to a conversation about changes that are in order.

There is nothing more demoralizing than teaching or being a student under policies or mandates that seek to move discrete outcome metrics without improving working or learning conditions. This report is far from a ringing endorsement of the current governance structures, policies and programs. Rather, it is a call for changes – better data, greater transparency, better coordination and planning, and more attention to the perspectives of teachers, principals, parents and the community. The PERAA law anticipated annual evaluations and a five year summative report. But for eight years there has been no independent evaluation. Finally, five years late, the National Research Council of the National Academies has been able to spend three years tapping researchers with a wide range of relevant national expertise. It is up to elected officials, accountable to the public, to read carefully and internalize this independent summary report.