How to Fix a Broken High-Schooler, in Four Easy Steps (Replay)

Hey, podcast listeners. Last week we brought you an episode from our archives called “Is America’s Education Problem Really Just a Teacher Problem?” Today we’re bringing you the follow-up episode. It’s called “How to Fix a Broken High-Schooler, in Four Easy Steps.” It focuses on the demand side of the education equation — the students — as opposed to the supply side, the teachers. I hope you find the topic as interesting as we do, whichever side of the equation you’re on.

CAROLYN ACKER: My name is Caroline Acker. I was the executive director of the Regent Park Community Health Center.

Regent Park, in downtown Toronto, is known for having one of Canada’s oldest, biggest and worst housing projects.

ACKER: So, we’re doing all this work, we’re investing more and more dollars. When I went to Regent Park Community Health Center in 1992, the budget was about $2.8 million. By about 1996, ‘97, the budget was close to $6 million. Instead of things improving, things were getting worse in terms of crime and murder and violence — this kind of thing. We were very distressed over what was happening to our young people, and we didn’t really understand it. We were doing more and more,  always investing more, and we weren’t seeing an improvement.  There were about nine murders in Regent Park in 2000, which was the year before we started Pathways To Education.

Pathways to Education was a voluntary program for high-school kids in Regent Park. It wasn’t an education program, exactly; it was more like life support.

PHILIP OREOPOULOS: The Pathways Program, they say, has four pillars. Those are: counseling, academic, social, and financial.

That’s Philip Oreopoulos. He’s an economist at the University of Toronto, with a particular interest in education. Over the years, Oreopoulos had heard about Pathways to Education — that it was something of a miracle cure for low-performing high-schoolers. He wondered if that could possibly be true.

OREOPOULOS: Pathways to Education had a pro bono study done in the mid-2000s by a consulting firm. And the director that did the pro bono study was a member of the board of Pathways and came out with a report—

DUBNER: I can feel your antenna as an empirical economist already going up, right? A nice report done by a consultant — this was Boston Consulting Group I believe, right?

OREOPOULOS: It was Boston Consulting Group, and—

DUBNER: Who are good and reputable, but still, a pro bono report commissioned by someone who’s also sitting on the board of the nonprofit that’s running the thing. You might be a little bit skeptical, yes?

OREOPOULOS: Well, what was striking about the report was it suggested that before Pathways, the dropout rate was 56 percent, and very soon after Pathways was introduced, the dropout rate was 10 percent. So you had a 46-percentage point fall in the dropout rate, and the report was attributing it to the introduction of Pathways. And this type and magnitude of effect is virtually unheard of in the education literature. It’s like the holy grail of programs that try to improve outcomes, especially among disadvantaged households. And if these results were true, we should try to figure out exactly how to replicate them across the country and in the U.S. because they’re so large it would solve a lot of our problems.

So, on today’s program: were the results true? If so, how did it happen? And, most important, where do the rest of us sign up?

Carolyn Acker was running a community health center in a Toronto housing project.

ACKER: I was driven to break the cycle of poverty. How do you break the cycle of poverty? Well, education and income are the two most powerful determinants of health.

We talked about this in our last episode — the huge returns to education, especially if you’ve got a good teacher. We also talked about how U.S. teachers, on average, aren’t as good as we might like, and that it’s not necessarily their fault.

But teachers, and schools, are only one side of the education equation. There’s the students and their families. And the schools don’t have much to say about how well-equipped or motivated, those kids may be.

So here’s the paradox: the returns to education are huge, but the kind of kid who really need those big returns — a kid from a poor, maybe broken family — is least likely to get a good education. As much as the school system might want to help, they’re not really set up for it. So what about a community health center in a housing project? It wasn’t an obvious idea, in the beginning at least, even to Carolyn Acker.

ACKER: And the community had quite a stigma, and it was sensationalized; it made a lot of headlines, right? Regent Park is this horrible place, and look what’s happening there. Now, we never felt that way. We felt a privilege to be able to serve the community, and very distressed by the fact that we were working harder and expending more efforts and investments to improve the health of the community and we weren’t seeing it improve at all.

They kept trying new things.

ACKER: And we were doing our third or fourth strategic plan.

Finally, they couldn’t help but notice the relationship between school and all the bad stuff happening in Regent Park.

ACKER: So we worked with the Toronto District School Board. They told us we had a 56 percent high school dropout rate, and that only 20 percent of the kids were going on to postsecondary. And when the director of the program came and told me that, I said, “Now I know why these kids are shooting each other. They have no hope and no future.”

Acker and her colleagues changed their focus. Rather than dwelling on all the bad outcomes in their neighborhood, what about the good outcomes, as rare as they might be?

ACKER: And we brought in focus groups — we did the young people who grew up in Regent Park who had made it through university, were doing well. We asked them what made the difference in their lives and every one of those people told us somebody took them under their wing. It was either a teacher, it was either a Parks and Recreation person, and they taught them and they helped them all the way. Now, this only happened to a handful of people spontaneously, but we were listening — listening is exceptionally important — and we determined they lack academic support. There is absolutely no one they can go to for help with homework. They lack social capital. They have no networks whatsoever. They go from one house of poverty to another. There’s no aunt or uncle that can help them get a job. So I was thinking, “I have to make an old boys’ network here.”

OREOPOULOS: And she and her staff came up with basically the Pathways to Education model with the idea and the motivation being that if they could break the cycle by reducing the dropout rates and encouraging individuals within the community to go on to doing well in their careers and then give back to the community that they came from that would be a way to try to turn the community around.

Pathways to Education would be a voluntary program, funded primarily by the government and non-profits. All you had to do to qualify was be a high-school kid in the neighborhood and sign a contract, along with a parent, that you’ll do everything you’re supposed to do.

In 2001, when Pathways began, 115 incoming freshmen signed up. Every year, a new cohort was added; within a few years, more than 600 kids were participating, and the program spread to two other housing projects in Toronto. The take-up rate was about 85 percent.

OREOPOULOS: It’s unique in that the origins of the program are neighborhood-based.  It’s not directly, explicitly tied to the schools. And yet they coordinate a lot with the schools. The neighborhood source is important because it helps give the students a sense of community even away from the school that they’re part of this group on or off the school grounds. And that it’s the community really paying attention to them, it’s the community trying to get them to sign up for the program, and it’s the community providing the support.

That’s what Carolyn Acker means by an “old boys’ network.” To get Regent Park kids to go to school, to do well in school, it would take more than just one or two lines of intervention. In fact, it took four: counseling, academic tutoring, social activities, and financial incentives.

OREOPOULOS: On the counseling side, every student that enters grade nine gets assigned a student/parent support worker. And they call them SPSWs.  The SPSWs are paid advocates who meet on a regular basis with the students to check in with how they are doing and it’s their job and role to help ensure the academic success of the student. The next pillar is on the academic side, where tutoring services are available up to four nights a week.  Students have to visit their SPSW twice a month. And they’re expected to attend tutoring services twice a week unless if their grades are above a certain amount, around 70 percent.  And as incentives to do those things, there’s the financial pillar. So as long as a student continues to participate in all the activities in Pathways, they’re eligible to receive free public transportation that can get them to school and back and also a college scholarship that builds up to $4,000 by the end of high school.

DUBNER: $4,000 for each year of college, or total?

OREOPOULOS: Total.  

DUBNER: So financial incentives, but let’s be honest, not huge. Free transportation is certainly not huge, but something. And $4,000—

OREOPOULOS: And yet, when interviewing the students participating in Pathways, it’s clear that the public transportation is a big deal to them, that they feel that they have to participate in order to get the public transportation tickets, or else that they would be walking to school.  And when they see their friends getting the free public transportation as well, I think that they see that as a big advantage that they want to do.

ACKER: We have a contract — parents sign it, because we needed permission to look at the grades, look at the marks. So that when a parent called up and said, “You didn’t give my Johnny enough bus tickets” we would say, “Well. Johnny skipped school on Wednesday. That’s why he didn’t get two for that day.” “Oh, he did, did he?!” The kids told me they’d get the bus tickets and they’d sell them for cigarettes only the first time because they would learn right away — now they can’t get to school.

OREOPOULOS: And then the last pillar is the social pillar, where Pathways students are eligible and required to go to at least one group activity a month. And these group activities are designed, one to keep them out of trouble and encourage them to hang out with also academically interested students. And two, to have fun, or even learn something along the way, develop a new hobby. And so the activities can range from a sporting event, or bowling night. Sometimes they can even be more sophisticated where they will have cognitive behavioral therapy and encourage students to think about managing their stress.  

Coming up on Freakonomics Radio: we’ll learn just how effective Pathways to Education was — and, what we really want to know: of all these different interventions, which ones were doing the heavy lifting?

*     *     *

Philip Oreopoulos had heard that Pathways to Education was phenomenally successful at helping kids stay in school, and do well. But the source of this rumor was a report that had been commissioned by a Pathways board member. Oreopoulos, as a research academic, wanted to look at the data for himself.

OREOPOULOS: Right, so I started engaging Pathways on possibly working with their data a while back. It took me a few years to convince them that I was genuinely trying to evaluate the program and look into the program from a non-partisan perspective.  And I think to their credit that they were also interested in having a third party evaluate the program.

Oreopoulos was able get hold of some publicly available data, but it was when Pathways started feeding him their data that he could really measure what was going on.

OREOPOULOS: Just prior to the introduction of the Pathways program in 2001, the entering grade-nine students in 2000, the eventual graduation rate, five-year graduation rate for them, was 40 percent. And two years later it does jump up quite a bit to almost 60 percent, so a 20 percentage point increase in the graduation rate for students who were entering from Regent Park. And keep in mind the program was phased in starting from grade nines entering in 2001. So grade 10s in 2001 were not eligible, but grade nines were. And then all the grade nines entering afterwards were eligible for Pathways. And so you compare that to the change in the graduation rates for the students from other housing projects. The increase in graduation was less than five percentage points. So you had the 20-percentage point increase over a two-year period for students from Regent Park, and then you had only a three- or four-percentage point change in the other projects. So the estimated impact of the program was about 15- to 16-percentage point increase in the high school graduation rate.

DUBNER: Which is huge, yeah?

OREOPOULOS: It’s huge. It’s still, it’s not the large effect that the Boston Consulting firm was finding, but still it’s a large enough effect to get really excited about. So not only do I find large increases in the high-school graduation, but also about a 20-percentage point increase in college-going as well.  

DUBNER: So, one would think that educators and politicians around the world are looking at this as a beacon of light and saying, “Hey, let’s replicate, replicate, replicate, or adjust to our own needs.” Is that the case, or no?

OREOPOULOS: Well, sometimes I describe the Pathways program to people outside of Canada as the Harlem Children’s Zone of Canada. We have this interesting program that’s relatively young that people are becoming more aware of that seems to be successful, at least in the settings that have been so far examined. I think the program is certainly intriguing to want to consider a possible expansion both in Canada and the U.S.  I think overall, there is some effort to try and disentangle what parts of the program — what are the key ingredients that are really driving these positive impacts and taking from that knowledge ways to expand the program.

DUBNER: Philip, in the paper, you with your coauthors write, “It is not possible to tell from the results,” in this paper, “whether only a few components of the program drive the results, or whether its integration is crucial.” So, can you talk to me about that for a minute, why you can’t tell which of these different interventions are the ones that are moving the needle, and which of these  perhaps you suspect are driving the gains?

OREOPOULOS: Because the analysis simply looks at what happens to these graduation rates or college-going rates before and after the program compared to some comparison groups, the main estimated effects are on the overall impact of the program, what happened when the program was introduced. But of course that doesn’t really say even with these large impacts what’s driving that, whether it’s just one component or one pillar of the program or whether it’s really important that all these services are provided at the same time. I think  it’s virtually impossible to even imagine setting up the ideal experiment to investigate that. The title of my paper is “An Integrated Approach for Helping Disadvantaged Youth.” “Integrated” because I think it may well be that a student who goes to see his SPSW gets advice to make sure that they’re going to tutoring, and while they’re going to tutoring, they meet some friends who are students, and then they get encouraged to go to the group activity. And then they keep doing this because they want the incentives. So everything is sort of working together to help ensure the participation.

DUBNER: Philip, you write that the cost of this program is nearly $5,000 per student, per year.  This of course is on top of the cost already going to the public school, through taxes and so on. And that doesn’t include pro bono tutoring and so on. So in some places, you know, $5,000 a year on top of all those other things that are not counted in a dollar fashion here, that buys a kid a good private school education. So considering the cost of this program per head, what can you tell us about overall ROI? So you’ve said that you’ve found that Pathways to Education really increases high school completion and postsecondary enrollment a lot. But as large as those gains seem, you know first of all it’s relatively expensive. And second of all it’s still barely half of these kids are finishing high school even so and/or going to college.

OREOPOULOS: If the impact that we’re finding from Pathways is really correct, in terms of a 15 percentage point increase in the high school graduation rate, a 20-percentage point increase in the college going rate, if these translate to higher earnings, say 10 percent a year, 15 percent a year and possible non-pecuniary benefits as well — better health, lower crime rates, more social participation — these benefits are huge because they’re added up over a lifetime. Whereas the cost of Pathways is, you know, only for a four-year or five-year period. The potential impact from just a small increase in rate of return from these would lead to an increase in lifetime wealth that is likely to be quite substantially more than the $20,000 total cost of the program. And in fact, with some back-of-the-envelope estimates, I find that the tax revenue alone from the higher earnings is enough to cover the present value of the program.

Philip Oreopoulos wanted to measure the efficacy of Pathways to Education not only because he’s an economist who pays attention to education, but because he thinks that most economists who pay attention to education are missing the point.

OREOPOULOS: For the longest time, economics has modeled education as a well-thought-out investment. And so all decisions are the right ones. If a student decides that he or she doesn’t want to study, or she wants to drop out of high school, our traditional models say that that’s the right thing to do, that they’re doing the best that they can under their circumstances and their ability, and whatnot. And more recently, economics has started incorporating elements of psychology, and sociology, and neuroscience to recognize that we don’t always get it right when thinking about long-term investment decisions, especially when it has to do with ourselves, and these kind of things.

DUBNER:  In one of your papers you make the point that young people, while needing education to do well later, are kind of the worst candidates for understanding how much they need education to do well later. And you open with a quote from Aristotle, which I’ve never read, which I love: “The roots of education are bitter, but the fruit is sweet.”

OREOPOULOS: Right, my father always mentioned that quote when I was a kid to remind that however bitter the experience of education was right now, the effort from staying home on a Friday night, or doing something that doesn’t really seem that exciting or fun and requiring concentration and thinking, however tedious, the benefits are long-term and they’re big. And  the difficulty in making these types of decisions is especially true for children and adolescents whose brains are just not even developed at that point when the cortex, the parts of the brain that are really focused on trying to imagine the future and think about the consequences are taking many years to develop.

DUBNER: Is this, do you think, a problem that we’re only starting to realize now, or is this something — look Aristotle knew it. Did we forget it for a few thousand years and we’re getting back to it, or is it something you think education folks and ed. reform people have known all along?

OREOPOULOS: I think we’re starting to pay more attention to the possibility that a more behavioral model might be a better way of describing how individuals are making schooling decisions. And that opens up to a lot of possibilities for policies. So on the one hand, students might not be getting it always right, not to stay that they always get it wrong. But on the other hand, there’s room with this knowledge to come in and think about alternative ways to help them, that we haven’t thought about before.

DUBNER: So if you think of education as a form of investing — and, as we know when it comes to financial investing, there are all kinds of mistakes to be made — what kind of mistakes do students make in thinking about education?

OREOPOULOS: I guess I boil it down to four points. One is students are often too focused on the present. They deemphasize the future or they don’t think about it as much as they should. The second is that students tend to over rely on routine, just keep doing what they’ve been doing and not realizing maybe other routines might be better for them. Three is that students sometimes think too much about negative identities. They focus on what they’re not good at and they might hang out with the wrong crowd and be influenced negatively that way. And the fourth one is that mistakes are more likely when in stressful situations or in situation where there’s not enough information.

DUBNER: Right, so when you talk about that integration and you talk about all the different components of it, there’s the social worker that’s attached to the student and the parents, there’s the academic tutoring, there’s the financial incentives, there’s the group activities, and the social pillar.  When you describe it like that and when I read about the program itself, it strikes me that basically this program is performing the function of a community and the family within the community. So I’m just curious, I know it’s attractive because it works well and because it’s relatively unusual, but did it strike you at any point that holy cow, what this program is doing is essentially backfilling for what society and families used to already be doing on their own without a special program designed to achieve it?

OREOPOULOS: It’s possible. It’s possible that the program is succeeding because there’s not enough support at home or in other parts of the community. I think all households to some degree or other sometimes miss out on providing encouragements or that nudge towards using services that students themselves may not feel that they want to use. I think one interesting possibility is that the program is working essentially by mandating that some students see someone on a regular basis to talk about their academic goals, to get reminded about why they’re in school and where they are going. And to get mandated to use tutoring services even if they don’t really feel that they have time or that they need the help.

Carolyn Acker, who helped set up Pathways to Education, was instrumental in making the tutoring mandatory. And she’s not even a behavioral economist. Nope, her background was nursing. Maybe that’s where she learned to listen so well.

ACKER: The kids actually told us to make tutoring mandatory. After our first year, which we called a pilot year, we brought in the youth, we did focus groups with the tutors, the mentors — how do we improve this? And we asked the young people, “How do we increase attendance at tutoring?” And the naughtiest boys of all looked at us and said, “You have to make it mandatory!” So, you see, the young people want structure, and they want to know that somebody’s caring about them. We care if you skip school!

*     *     *

Freakonomics Radio is produced by WNYC Studios and Dubner Productions. Today’s episode was produced by Suzie Lechtenberg and David Herman, with help from Joel Werner. The rest of our staff includes Arwa Gunja, Jay Cowit, Merritt Jacob, Christopher Werth, Greg Rosalsky, Kasia Mychajlowycz, Alison Hockenberry and Caroline English. If you want more Freakonomics Radio, you can also find us on Twitter and Facebook and don’t forget to subscribe to this podcast on iTunes or wherever else you get your free, weekly podcasts.