Q&A: New Orleans Education Expert Doug Harris Walks Through His Research Showing a Decade of Student Gains

This article is part of a series commemorating the 10th anniversary of Hurricane Katrina and the efforts to rebuild schools in New Orleans. To read more about this topic, please visit our comprehensive coverage and essays. We also recommend watching our three-part documentary series that explores the past, present, and future of education in New Orleans.

There has been a long-standing debate about the impact of school reforms that were implemented in New Orleans following Hurricane Katrina. The majority of traditional public schools were converted into charters run by nonprofit organizations. Economist Doug Harris, who leads the Education Research Alliance at Tulane University, has conducted extensive research on the effects of these reforms on student outcomes. In a previous report, I summarized the key findings of his latest study that was presented at a conference in May.

In a recent interview in mid-July, Professor Harris and I discussed his studies in detail. We covered various topics, including the improvement in test scores in New Orleans, changes in the teacher workforce, and the challenges of replicating these reforms in other cities. The transcript of our conversation is provided below, with minor edits for clarity and length.

The Seventy Four: To begin, could you please explain the specific reforms that were implemented in New Orleans?

Professor Doug Harris: I see the reforms as a combination of changes rather than a single initiative. They were based on a few fundamental principles, namely accountability and autonomy for schools. Let me elaborate on that.

Regarding accountability, the reforms introduced what economists refer to as "market accountability." The traditional attendance zones, which determined students’ school assignments, were eliminated. This means that schools had to compete for students, thereby being accountable to the market. Additionally, the government held charter schools to performance contracts. If a school failed to achieve certain test score targets, it could be closed down. Many schools have indeed been shut down, taken over, or merged with others. This high level of accountability is based on test scores.

The second principle is autonomy. Schools were given the freedom to determine how to achieve the set goals. While the government established the benchmark for test scores, schools had control over their budgets. The funding was allocated per pupil, similar to a business model where more students meant more funding. Schools were also granted flexibility in how they utilized their budgets. Moreover, the union contract was abolished, which had a significant impact on the hiring, firing, and evaluation of teachers. This aspect was another crucial element of autonomy.

Are test scores the sole metric used for accountability purposes?

Test scores are the primary metric used for accountability, especially at the high school level. However, other indicators like high school graduation rates are also considered.

Did these reforms "work"? In other words, what evidence supports whether the New Orleans reforms were successful in improving student learning and other outcomes?

When I first arrived here, people primarily talked about the increases in outcomes. Regardless of the measure used, there has been a substantial improvement, particularly in comparison to the state as a whole. Concerns arise because changes over time or state policies unrelated to the reforms might influence the outcomes. To address this, having a comparison group is beneficial. In any case, New Orleans has shown improvement across various measures, including state test scores for grades 3-8, ACT scores, high school graduation rates, and college enrollment. However, it is essential to consider other factors that may contribute to these improvements, not just the reforms.

Over the past year and a half, my team and I have been working to disentangle the effects of the reforms from these other factors. I will provide you with an overview of what we have examined.

One concern was the changing population. Some public housing projects were temporarily closed, and the neighborhoods most affected by flooding were also the lowest-income ones. There is evidence suggesting that low-income populations did not return as quickly. These circumstances alone could account for the increase in scores due to the strong correlation between income and educational outcomes. It is possible that the perceived improvements are simply a result of these demographic changes.

We began by examining census data and then conducted simulations using additional data to explore the relationship between income and test scores. We obtained this information from a nationally representative federal dataset and performed statistical analysis. We also integrated data on how income changed in New Orleans compared to the surrounding districts. Our findings suggest that New Orleans has a slight socioeconomic advantage or improvement relative to the other districts, although the effect is minimal and cannot explain the observed increases in outcomes.

The first factor we considered is the change in populations.

The second factor is the distortions caused by test-based accountability. This refers to practices such as focusing solely on test preparation without actually enhancing learning in a substantial way. It was challenging to analyze this aspect, but there is a strong likelihood that it is a more significant issue in New Orleans compared to other districts. Two reasons support this claim. Firstly, charter schools, which are prevalent in New Orleans, place a heavy emphasis on test performance as part of their educational philosophy. Secondly, these schools faced the risk of closure if they failed to demonstrate positive results. Therefore, test scores had a direct impact on the livelihoods of educators in these schools.

To investigate this, we analyzed the test scores based on the stakes involved. In Louisiana, during the period we examined, the science and social studies tests had lower stakes compared to math and English language arts tests. If the response was driven by incentives, we would expect to observe greater improvements in math and language arts scores. However, our findings indicate that the effects on science and social studies scores are similar to those in math and language arts. This initial analysis suggests that teaching to the test was unlikely to be a significant factor.

We also conducted an analysis of "bubble effects," which refers to the possibility that schools might prioritize students who are close to the proficiency threshold while neglecting those who are significantly below or above it. This could have been influenced by the school performance score, which initially placed more weight on proficiency ratings. However, we did not find any evidence of this phenomenon specifically in New Orleans, suggesting that it does not contribute to the observed outcomes more than in other districts.

These were the only two measures we could take. While it is still possible, and I remain skeptical, that some test-based distortions influenced the outcomes, we also examined state reports on high school graduation and college outcomes in New Orleans.

College outcomes hold great significance for two reasons. First, from a long-term perspective, college achievement is a stronger predictor of students’ overall life outcomes compared to test scores. We utilize test scores as an immediate and easily accessible measure. When exploring college outcomes and high school graduation rates, we found positive trends. The percentage of high school graduates who immediately pursue college education is increasing, alongside a rise in the overall high school graduation rate.

Are these trends occurring at the same rate?

Yes, in fact, the fact that they are moving in the same direction is remarkable. Typically, a higher graduation rate is associated with a lower percentage of students going on to college, as the additional graduates may include lower-performing students who might not choose to pursue higher education. The magnitudes of the trends are similar, but the alignment of their directional movement is even more noteworthy.

SCHOOL DISCIPLINE

What about school discipline? As you know, there is concern that these schools may be suspending and expelling more students, suggesting potential costs in terms of student treatment within schools.

There could be several reasons for this. One possibility is that the actual number of incidents has decreased. This could be attributed to the implementation of strict discipline policies. Just because there are fewer suspensions doesn’t mean that schools are not strict; it could mean that the strictness is resulting in fewer incidents.

Has any research been conducted to analyze whether there is a correlation between strict schools in New Orleans and their achievement outcomes or suspension rates?

We have not conducted such research partly because it is challenging to accurately interpret the numbers. Schools that enforce particular discipline policies also have specific teaching methods, and it becomes difficult to separate these factors.

SCHOOL FUNDING

Next, let’s discuss the role of money. Some people argue that the increase in funding is what explains the improved outcomes in New Orleans.

While it is possible, it is difficult to isolate the impact of any single change, including the reforms and financial investments. I can provide an overview of how the funding situation changed. Initially, New Orleans was spending approximately $8,000 more per pupil compared to similar districts. This was almost a doubling of the spending. However, it gradually converged back to the normal or near-normal rate. Currently, they are spending around $1,000 more per pupil than similar districts, whereas before the hurricane, their spending was similar to those comparison districts. It’s important to note that these numbers exclude construction funds.

So, what does this interpretation suggest? Does it mean that investing more money into urban school districts will lead to the improvements witnessed in New Orleans?

It is highly unlikely that this is the sole explanation. When we examine evidence from other places, we do not see robust relationships between rapid increases in spending and substantial score improvements. This is not to say that there is no relationship at all. In fact, I believe that the increase in spending was undoubtedly a necessary factor in the reforms. To attract talented individuals, competitive salaries were required, and resources were needed to initiate the reforms. It is doubtful that everyone would have come and performed as well with the previous level of funding. I believe that every element of the reform package, including the change in spending, played a role, but it is unlikely that it was solely about the money.

Final question regarding funding. What do you make of the argument that New Orleans could have invested in pre-K, reduced class sizes, and increased salaries for experienced teachers who remained in the system? Could this have improved academic outcomes while causing less disruption to the community?

Certainly, it would have caused less disruption. The current system is markedly different from what people were accustomed to. It took considerable time to establish and implement, and it continues to involve disruption due to school closures and elevated teacher turnover. However, the degree of disruption should not be the primary criterion. Ultimately, our focus should be on whether the schools have improved or not.

Once again, I would acknowledge that providing more funds to schools does have some positive effects. However, based on previous research, there is no reason to believe that solely implementing these measures would have generated the same level of impact. In fact, we have compared the effects of our reforms to interventions such as early childhood education and smaller class sizes. I have also conducted previous studies comparing increased teacher salaries to smaller class sizes.

TEACHER EFFECTIVENESS

Can you discuss the changes that have occurred in the teaching profession and the demographics of teachers? Specifically, I am referring to the paradox that has emerged – the teaching force has become more homogeneous in terms of race, less experienced, and less qualified. Based on previous research, one would expect a decline in teacher effectiveness due to these factors.

This is a significant topic and New Orleans holds particular importance because it encompasses various aspects of the educational reform debate. It touches on financial considerations, perceptions of the teaching workforce, school choice, and test-based accountability – all within one context.

Descriptively, you are correct. The percentage of certified teachers has decreased by 20%, the percentage of teachers with 10 or more years of experience has also dropped by 20%, and teacher turnover has doubled. Certification, experience, and low turnover are generally considered indicators of a high-quality school and workforce.

I believe there is a relatively simple resolution to this paradox. Firstly, alternative certification programs have played a role in the changing demographics of teachers. These programs have a decent track record, if not a good one. For example, Teach For America (TFA) has shown evidence suggesting that its teachers are as effective, if not more so, than traditionally trained teachers when it comes to student achievement. Many of the new teachers in New Orleans are coming from programs like TFA.

Secondly, the school models in New Orleans require teachers to work exceptionally long hours – it is not uncommon for them to work 80-hour weeks. The traditional career path does not accommodate such demands. Teachers often start families in their twenties or thirties, making it nearly impossible to work such long hours unless they have no other choice. By bringing in less experienced teachers who do not intend to make teaching a long-term career, the education system in New Orleans can make up for their lack of experience by having them work longer hours.

Do you think that the selection and deselection of teachers through the evaluation system, such as the use of pay for performance models in charter schools, contributed to these changes?

Yes, that is one of the factors I was going to mention. Let me mention the other factor, which is the size of the city. New Orleans is not a large city like New York, Miami, Los Angeles, or Chicago. It is a smaller city, with approximately 90 schools. Many people, not just local residents, wanted to contribute to the rebuilding of New Orleans and be part of this unprecedented reform movement. There was a high level of interest, but only a limited number of job openings. As a result, there was an oversupply of candidates for the available positions. Schools were able to be selective and choose who they wanted to hire, and they also had the ability to fire teachers and have a long list of replacements waiting. This is a situation that is unlikely to be replicated in most other places. I believe this is a significant factor contributing to what is happening in New Orleans, and it needs to be considered when thinking about the implications for the rest of the country.

REPLICATING NEW ORLEANS

So, let’s discuss this! What does the situation in New Orleans mean for the rest of the country?

From my perspective, the results of the reforms in New Orleans are undeniably positive, no matter how you analyze them. Even the potential negative consequences appear to be minimal or non-existent. There is no evidence of increased segregation. All subgroups of students, including those with special education needs – a group of particular concern – seem to have benefited. The number of disciplinary incidents has not increased; in fact, out-of-school suspensions have significantly reduced. There is a lot of good news coming out of New Orleans.

One thing this tells us is what can be achieved through this type of reform. I believe that what has been accomplished in New Orleans represents the upper limit of what can be achieved using this approach. If other places attempt similar reforms, they may not achieve results of the same magnitude. However, this should not dissuade them from trying. Instead, they should view New Orleans as an example of what might be possible.

In regards to skepticism, as previously mentioned, I believe that the unique conditions in New Orleans played a significant role in the observed effects. Unlike other places, people from various parts of the country were motivated to come to New Orleans to contribute to school reform efforts. New Orleans is renowned as the hub of education reform, attracting highly talented individuals who are passionate about improving the education system. This phenomenon may not be replicated in other locations. It’s similar to how individuals aspiring to build a successful tech company would flock to Palo Alto, as it is recognized as the center of the industry. Likewise, those interested in school reform gravitate towards New Orleans. Such dynamics may not apply elsewhere; there is only one Silicon Valley.

These are important considerations to bear in mind. While the same effects might not be observed elsewhere, it doesn’t mean that there aren’t valuable lessons to be learned from New Orleans.

Now, let’s consider a hypothetical scenario where there is an urban school district facing challenges. This district may not have any charter schools or have a limited charter sector. They approach you seeking advice on replicating the reforms implemented in New Orleans. What factors would you advise them to consider, and how likely would you be to recommend the New Orleans reforms?

Allow me to address this question by adding another aspect to my previous statement about the conditions in New Orleans. One of the defining characteristics of New Orleans was that it was an extremely underperforming district. It was regarded as a disaster by almost every measure. Initially, I was inclined to believe that these claims were exaggerated because everyone kept repeating them, but over time, I came across more objective evidence supporting these assertions. The level of corruption within the school board was so rampant that the FBI had to set up an office to investigate it. Finally, after years of effort, I obtained a report from the Council of Great City Schools, which evaluated the district’s human resources management. The findings were shocking, with a long list of issues plaguing the district’s operations. Unsurprisingly, the outcomes of the district were abysmal. It was the second lowest-performing district in the second lowest-performing state in the nation.

If a district is already functioning relatively well, the New Orleans reforms may not necessarily be the optimal choice. New Orleans serves as a case study highlighting the dire condition of a district that was performing poorly across various metrics. It is important to note that not all urban districts are inherently terrible solely based on their scores. Many factors such as demographics and socioeconomics play a significant role. Urban areas can house excellent schools and districts that do a commendable job. It would be unwise to dismiss their achievements based on a possibly erroneous interpretation, especially if the starting point of the students is not taken into account.

In such cases, I would advise districts or individuals advocating for the reforms to thoroughly assess the functioning of the traditional district before embarking on that path.

Furthermore, I would stress the importance of considering all the necessary conditions and actions required to ensure the success of these reforms. It is misguided to believe that simply increasing the number of charter schools is the solution. Detroit’s experience serves as a cautionary tale in this regard. Despite having a significant market share of charter schools, the system in Detroit is in shambles due to poor coordination.

One notable distinction between New Orleans and other places is the presence of two authorizers. In contrast, Michigan has dozens of authorizers, resulting in an uncoordinated system. It is possible to grant autonomy to individual schools while maintaining coordination at the system level. A potential lesson to be learned from New Orleans is that an unregulated approach, where every school operates independently, is unlikely to yield positive outcomes. Effective government intervention, ensuring the functioning of the overall system of schools, without overly interfering with every aspect of school management, might be a more appropriate approach.

Considering all these factors, what do you anticipate will be the primary methodological criticisms of your paper?

From my perspective, the primary critique will likely not revolve around methodology. I don’t expect people to argue that we should have employed a different methodology. We have extensively shared our paper with numerous individuals and incorporated sensible suggestions, leading us to reach the same conclusions.

Another aspect that may come up is that when we examine the initial few years, specifically up until 2009 when the system was still in its early stages, the impact is generally positive but not statistically significant. Some individuals have looked at this result and questioned its significance, even though our primary concern is the year 2012, which is the most recent year under study. When creating a new system from scratch, it is important to allow it time to develop and observe its long-term effects. What we observe from the data is that schools consistently improve year after year. The differences in 2009 may not be statistically significant, but if we look at 2012, they are always statistically significant. We have estimated various ways and it is very difficult to eliminate the effect of 2012.

Moving on to the aspect of test-based accountability, in addition to the direct examination of high-stakes and low-stakes subjects, the fact that we are witnessing a significant increase in high school graduation and college entry rates in state reports suggests that it is not solely a case of teaching to the test, as that would not result in students entering college.

Personally, I find the data to be quite convincing. Honestly, I did not expect this outcome at all. I anticipated a complex situation with factors such as population changes, distortions due to test-based accountability, and other potential issues, leading to unclear conclusions. However, after going through an extensive checklist, we consistently obtained positive results, making it difficult to interpret these findings negatively.

Is there anything else you would like to mention?

Another point worth discussing is the process involved, as much as we are focusing on the results. There is a genuine concern in New Orleans regarding how the system was implemented and how it is being managed. The fact that there is now a smaller percentage of African-American teachers, following the schools being taken away from the local district, which was elected by a majority African-American city, and handed over to the state, has caused racial tension. The state leaders do not represent a significant African-American majority. There is a sense of bitterness, even among families whose children attend charter schools and may even have positive opinions about their individual schools. They have concerns about the system’s operations and how it was initially established.

Considering this perspective provides some insight. There is survey data available from the Cowen Institute, which conducts surveys every few years. Generally, the results show majority support, both from the white community and the African-American community. However, it is not unanimous support by any means.

But I also don’t think the survey questions were addressing the points I mentioned earlier. They focused more on opinions about whether the current system is effective. If they had asked whether people thought it was justified for a small group of individuals to make the decision to run the schools in this way while everyone was being evacuated, almost everyone would have answered no, that it was not a good thing.

I believe the process is important. During one of the sessions at our conference, Howard Fuller participated in the discussion. He has been a staunch supporter of this kind of reform for a large part of his life but is deeply concerned about the process and how the reforms are perceived in the local community.

As much as I prioritize evidence, it is not solely about evidence, and we must keep that in mind.