More Evidence on Charter Schools

(Photo: Alex Starr)

Writing at Slate, Ray Fisman reviews the latest research on the efficacy of charter schools.  The study focuses on students at six Boston schools that had previously demonstrated an ability to improve students’ test scores on the Massachusetts Comprehensive Assessment System.  This time, however, the researchers wanted to evaluate whether the schools really improved student outcomes or just mastered the art of “teaching to the test.” Here’s the breakdown:

The study examines the college readiness of Boston public school students who applied to attend the six charter schools between 2002 and 2008, with projected graduation dates of 2006–2013. In just about every dimension that affects post-secondary education, students who got high lottery numbers (and hence were much more likely to enroll in a charter school) outperformed those assigned lower lottery numbers. Getting into a charter school doubled the likelihood of enrolling in Advanced Placement classes (the effects are much bigger for math and science than for English) and also doubled the chances that a student will score high enough on standardized tests to be eligible for state-financed college scholarships. While charter school students aren’t more likely to take the SAT, the ones who do perform better, mainly due to higher math scores.

The upshot of this improvement in college readiness is that, upon graduation, while charter and public school students are just as likely to go on to post-secondary education, charter students enroll at four-year colleges at much higher rates. A four-year college degree has historically meant a better job with a higher salary, making a spot in one of Boston’s charter schools a ticket to a better life for many students. (We’ll presumably know in a few years whether things actually turn out that way in the longer run for the cohort the researchers are following.)

Brian O'Connor

I would be very surprised if charter schools did not have a higher percentage of students going on to four-year colleges. Considering that students with special needs are rarely admitted to charter schools, or are matriculated out due to the stress of the AP focus of these schools, charter schools already have a significant advantage over typical public schools in terms of high performing students. Moreover, these lotteries are many time paired with entrance exams that students are required to pass prior to admittance. So, schools that steal the top 1 or 2 percent of the highest achieving students very well should have higher college entrance and completion rates.

Quite frankly, even if these students did not attend a charter school, they would very likely move on to college.

Enter your name...

They aren't comparing those who attended the charter school to those who didn't.

They are comparing those who "applied" and had the opportunity to attend (regardless of whether they actually did attend or not) to those who also, equally, "applied" and did not have the opportunity to attend (regardless of what school they eventually attended). The sole difference between these two groups is whether or not their name was randomly drawn out of the bucket sooner or later in the lottery.

You should read which directly says that in Massachusetts, there are no entrance exams or other ways to cherry-pick students. Either everyone who wants to attend gets in, or—if there isn't enough space—everyone who wants to attend has an exactly equal chance at getting in, with no favoritism and no rules against students with special needs.


It seems that this study is a bit misleading. Really all it is showing is that good schools have good outcomes, not that charter schools have good outcomes. Not exactly a mind blowing result. What is clearly true is that all schools need to improve and the distinction between charter and traditional is a red herring.

The troubling thing for me is the lack of sharing between charter and traditional schools. The initial intent of charter schools was to be a lab for testing new practices and identifying effective ones that could be disseminated back into the traditional schools. I have not seen much evidence of that happening.


Is there really a lack of SHARING between public and charter schools, or are the public schools just not interested in improving on their traditional practices?


You've told us nothing about the student populations in the comparison groups. For example, how many students in each group are below, at and above the poverty line. How many are Special Needs, how many are classified as LEP or ELL?

Your analysis is only helpful if you are okay with segregating our students and returning to pre-Brown vs. Board of ed type schooling.

Enter your name...

They've told you everything that you need to know about the population, which is this:

All the students in this population were randomly assigned to one of two groups: those who were randomly assigned a lottery number that made admission to the free charter school possible, and those who were randomly assigned a lottery number that made admission to the free charter school impossible.

That's the only initial difference between the two groups. Following the standard scientific method, when you take a large group of subjects and randomly assign individuals to one of two groups, and then treat the groups differently, you are then entitled to assume that any future difference between the two groups is likely to have been caused by the differences in the way that you treated them, rather than in pre-existing differences. That's the point behind randomized controlled trials.


I would be highly interested in seeing a Bill James-esque take on this situation. There needs to be some form of a SAT+ (akin to the ERA+ and OPS+) that can adjust students' performance for the context of their life situation.
Also, there is a tendency to group these three criteria together:
-Chartered Schools
-Unionized vs. Non-Unionized Teachers (many, but not all, charter schools are non-unionized staffs)
-School Choice

While all three of these elements can have an affect on a student's school experience, we need to be sure which element we are analyzing.

Kim J

1) To what degree do charter schools promote the vision that their students will go to college?
2) What sort of effect does getting into a "special" school have on how the child sees him or herself?
3) What are the student/teacher ratios of the charter and regular schools in the study?
4) What effect does working in a charter school have on the self-esteem of the teacher, and does this change how he or she teaches, feels about students, or approaches material?
This country is known the world over for being the seat of innovation. The more we move away from the free, inclusive, public education system that this country was built upon, the faster we will be out innovated and out smarted by other countries who paid attention to our original model.

Enter your name...

> 2) What sort of effect does getting into a “special” school have on how the child sees him or herself?

If this is a significant positive factor, then everyone should get to attend a "special" school.


That's the theory behind school choice. Whether or not the schools are actually "elite", if the people who attend them and work there believe that they are special there could definitely be benefits. This is also known as the Pygmalion effect.


The ideal way to test would be to randomly select 50% of the students in a given population and put them in Charter schools while the rest go to normal public schools. Any student that is forced out of the charter school for reasons of non-performance / special needs etc. should count as a drop-out and still appear in the denomiantor while calculating college attendance percentages. That would be the real test to see if a charter school is better, else it is just a case of kicking out the most vulnerable students and then claiming success by pretending that the good students are good because of your intervention


Thats their performance TODAY. I bet eventually the charter schools will take a profit maximizing approach just like hospitals and medical centers do today. As more charter schools proliferate, their roadmap will change from sending your kids to a great college, thereby attracting more students to getting the most bang for their buck.

It will all be about maximizing the return on their real estate, cutting labor costs, filling in as many slots as possible. As long as the schools aren't a complete disaster everything will be fine.

NOW the schools are on their best behavior. In the future, they will run out of control just like any capitalist industry.


"Moreover, these lotteries are many time paired with entrance exams that students are required to pass prior to admittance."
Actually that's not true. Charter schools by definition are open to anyone and do not discriminate on any basis. The lottery system, while not all that fair, is not paired with any sort of testing. If it does then you're probably talking about a private school. Charter schools receive tax dollars and therefore are not allowed to turn students away based on any sort of academic standards or exams.

Enter your name...

You're right about the situation in this study, but some other states permit charter schools to use admission criteria other than random lottery numbers. For example, my friend's child is graduating from a charter school this month that only accepts students who are considered "academic failures". There's another in my area that requires 50% of the students to speak Spanish at home (to the disgust of the many, many non-Latino families who want their children to be bilingual: the waiting list for English-only speakers is far longer than the waiting list for Spanish-only speakers). But, as I said, that's not the case in Boston.


" ... the researchers wanted to evaluate whether the schools really improved student outcomes or just mastered the art of 'teaching to the test.'"

"Teaching [students] to the test" is like Toyota building to a quality and performance spec -- both result in better products.


As other readers have stated, there are several key differences between charter schools and typical public schools that are rarely, if ever, taken into account when these studies are done. The most obvious is the difference in the student population. Charter schools do not have the same ratios of special ed students, and they have the ability to expel students for not meeting academic standards (generally speaking). Additionally, there is the factor of outside influences. In order to get into a charter school a parent must apply, which only parents with a genuine interest in their child's education will do. I suspect that if one were able to separate test results in neighborhood schools by parental involvement, the results would be similar to those of charter schools.


The phrase that comes to mind is: Exogenous Variables.

Could it be that the parents of the kids that are enrolled in charters are more likely to emphasize education; therefore, are present for homework supervision, etc...? Or that the parents are more likely to be more well educated and their children have better educational foundations because of their home-life?

Not to mention that charter schools have an advantage over traditional public schools: They can control their population. Public schools don't get to dismiss poorly performing students they way charters can.


My question is, which causes which. Is it that

a) charter schools improve student performance for those that apply


b) higher performing students are more likely to apply to charter schools

It seems that the discussion here seems to be focused on a. But isn't b a possibility?


@Stacey, @Tim, @dullgeek: So many people are posting to this discussion without actually reading the several posts that have explicitly mentioned that both the experimental AND control groups consist of students that applied to charter schools. Therefore, you CANNOT make the arguments you are trying to make. Both groups had parents motivated enough to apply to charter schools.

I am still skeptical, though, having had some very negative experiences with charter schools.

First, it is correct that charter schools can and do boot students that aren't up to snuff. Even if it is technically against the rules, there are charter schools that go to unethical lengths to remove problem students from the school. This could be an unfair advantage that the charter schools have, when comparing them to public schools.

Second, it is probably not possible, at this point, to say how much of the student success can be attributed to the psychological effect of randomly winning the lottery. I am reminded of the studies where teachers were told they were working with gifted students, who then performed better because of higher expectations. Something similar could be happening here, where the event of winning the lottery makes the students and their families feel special, lucky, or more in control of their situations, which could lead to increased performance, regardless of whether or not the school is any better. It would be difficult, but not impossible to set up experiments to test for this effect.

Finally, I am concerned about the methods used to select the charter schools that are being studied. These nine schools may, indeed, be better schools. But, that does not mean they are representative of charter schools, in general. From my experience, the success of a charter school seems to be strongly influenced by how much excitement and (more importantly) money can be raised by the people behind the school. And, there are many charter schools that have neither. It is possible that the main factor that makes these schools more successful is simply a placebo effect, and they might not actually end up teaching us anything useful about effective pedagogy.