04.23.14On Poverty and Schools
I’m a big fan of Paul Bruno’s- love reading his posts on Alexander Russo’s This Week in Education blog at EdWeek, follow him on Twitter, find him spot-on when I agree with him and thought-provoking and sensible when I don’t, even though I suspect I might be one of those people who he dismisses as being “reformy” (ouch!).
So I sat up pretty straight in my chair when his most recent blog post announced that improving education didn’t really reduce poverty. Who Told Us The Education Fights Poverty, Anyway? Paul is an empirical guy (one of the reasons I like reading him) and presented a data set appearing to demonstrate that high PISA scores not only didn’t correlate to lower levels of poverty but actually correlated to higher rates of poverty. Holy smokes! I’ve always believed that this was one of the primary values of effective schools and have read lots of studies suggesting it is true (for example, Eric Hanushek’s long term study that found that achievement rates accounted for 80% of differences in GDP growth over 40 years).
We had a brief conversation about it via twitter–conversation is a strong word for what anyone can do via Twitter–and I promised a follow up to explain why I didn’t buy his analysis. With apologies for blogging when i promised an email, here are some thoughts. My goal is not so much to “win”–i.e. convince doubters including but not limited to Paul that good schools do in fact help reduce poverty–but to start a more robust conversation about Paul’s hypothesis and how you would measure the idea to understand it more fully. After all the goal of using data is, to me, less to prove a point than to understand the factors behind a trend so you can make policies and institutions better.
First, here’s Paul’s data. He writes:
One preliminary way to look at the evidence would be to see if countries with better academic performance also have lower poverty rates.
Out of curiosity I decided to take a first crack at that using results from the 2012 PISA, which tested 15-year-olds in reading, math, and science.
To get an overall picture of each country’s performance I added up the three subject area test scores. (This may not be an OECD-sanctioned method.)
I then compared each country’s overall PISA performance to its pre-tax, pre-transfer poverty rate. (That’s the poverty rate before the government “steps in” by taking money in the form of taxes and redistributing it in the form of welfare payments, etc.)
In general, countries with higher PISA scores also tend to have higher pre-transfer poverty rates. (We’re the red dot.)
I think there are a couple of problems with the approach here. First, I want to restate what we’re looking for a correlation between–the 2012 test scores of 15-year-olds and the poverty rates of their nations in that same year.
The first thing that jumps out at me is that if the data had in fact turned out a strong correlation, I’d be highly skeptical. In 2012, 15-year-olds the world over, like 15-year-olds in most years, had very little ability to mitigate their own poverty, no matter how attentive in school. If you wanted to see an effect you’d need to look at the correlation between national PISA scores for 15-year-olds and poverty rates at some point in the future… 20 or 30 years out?
And while you were at it you probably wouldn’t want to look at a lot of data before drawing any big conclusions. That is, I’d expect to see a national economic effect (whether or not that showed up as a reduction in poverty) if schools were better for a sustained period of time… five years at a minimum to affect something on such a large scale as poverty rates.
I’d also want to think about controlling for other factors since there is so much “noise” here–ie other variables that could have large effects. So I’d ideally want to test how changes in PISA scores over time affected changes in poverty rates in countries taking the test.
One other thing that jumps to mind also has to do with a comment that Paul made in our twitter exchange. I referenced the noted economist Eric Hanushek’s findings on the strong correlation between test scores and future economic growth:
“If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate.
That may not seem like much, but, as Hanushek notes, world economic growth averages about 2 to 3 percent of GDP annually, so a difference of a full percentage point in growth is massive. Here’s a chart of Hanushek’s studying the correlation between test scores and GDP growth for 50 countries over 40 years (on the left side).
Anyway, Paul responded to my comments by noting that GDP growth (which is the increase in overall national prosperity) was different from poverty rates (which is a question of distribution).
That made me also think about the correlation between PISA scores and poverty rates. In the US, I reminded myself, not finishing high school or finishing high school with skills that don’t allow you to go on to college is one of the fastest way to increase your likelihood of ending up in poverty. (Would anyone dispute that? The trend in fact has steepened recently as middle class industrial jobs erode and “knowledge work” is the overwhelming way to the middle class and above) But that also reminded me that only a minority of PISA test takers are those who end up in poverty; mostly they’re a measure of those who end up participating in the economy productively and framing the question of how productive they’re able to be. So if i really wanted to look at education’s effect on poverty I’d probably want to look at much at distribution of scores as I would average scores. Something like this:
Finally, it’s important to look at what we’re looking at when we talk about “poverty.” My understanding is that Paul may have used GINI as a proxy for poverty but according to the World Bank, GINI is a measure of income distribution rather than of poverty. And this could be a major confounding factor. For example, roughly speaking, being in the bottom quintile of incomes in the United States makes you “poor”…relatively speaking. I do not intend to mitigate the suffering or difficulties that come with that poverty. (In fact, I work primarily with high poverty schools because i care so much about it) but being in the bottom quintile in the US would not make you poor in India or El Salvador. Your income and quality of life would be far above the mean. So your opportunity and quality of life even at the bottom of the economic hierarchy in the US is far better than that of people far higher on a relative ranking of poverty within other countries. So if i improved my nation’s schools and GDP growth went up steeply, everyone would become more prosperous and better off and poverty would be mitigated but my country’s GINI scores might not change. In fact they’d be just as likely to become more pronounced–poverty and income distribution are different things.
Anyway, I’m sure i got some of this wrong. And i don’t mean to call Paul out. But I believe that schools do influence poverty rates and so I think it’s important to engage a discussion about how we’d assess that because 1) it’s important to understand how and for who and 2) If it’s NOT true, and if having better schools does not mitigate poverty and change people’s access to opportunity, we’d better make some serious changes and all of us had better get pretty “reformy” pretty quickly.
Lots of people understand this stuff better than me. Hope you’ll weight in!
In our educational system, students must take on significant debt to acquire a college degree. For students coming from poverty, this would seem to keep even college educated youth at lower income levels.
As a fellow empirical minded person and a fellow admirer of Paul Bruno, I have to say that I am generally skeptical of correlations that use countries as data points. Sure you can look at test scores correlated with economic growth 30 years later, but there are so many factors about the country that are also correlated with test scores that might account for that economic growth. I am sure Hanushek has multiple regression analyses that control for several other potential factors, and make the claim that test scores (educational performance), and not years of schooling, or any number of other anti-poverty efforts have a huge impact on future economic growth, but I find it hard to imagine that those variables truly can be controlled for, because they are often so highly correlated with everything else. Hanushek makes similar claims about teacher “quality” mattering far more than resources for schools, and I am similarly dubious that his regression analyses can isolate teacher quality from working conditions and student variables. Teachers are not randomly distributed to schools, and neither are students, and while we can act as if we can statistically control for the massive selection bias, I just don’t think that statistical controls justify the conclusions he draws.
Variables such as universal preschool, or free school lunch programs, or food stamps, or other social safety net programs do not vary randomly, but often fit in with that country’s political orientation and culture. Each country is not a natural experiment. I’m not saying that economic research is impossible, I just don’t like treating these correlations as if they are natural laws (high educational performance leads to economic growth) rather than highly provisional relationships from the past that may or may not hold true in the future.
Thanks for your comment. I tend to agree with you. There are so many other variables to consider. My gut is it’s impossible to prove either way–so i am heartily skeptical of Paul’s claim that better schools “do not” affect poverty. It would be hard to prove the converse conclusively but I do think the evidence is generally much better on the side of “good education drives economic growth” at a minimum.
Ultimately, I believe that the quality of schools do influence poverty rates–but how is that truly assessed? Is it the number of students that go on to attend college? Graduate from college? Earn above the poverty level based on federal guidelines? Is it possible that a good (or great) school actually influences the level of poverty in a neighborhood? I don’t necessarily buy into his analysis because I feel like there are too many mitigating factors in each country, given what the median level of poverty is. I couldn’t agree with you more in that the disparity of income–being “poor” or in poverty in the U.S., is a far cry from what it would be in India.
I’m not sure that in this type of analysis you can separate the effects of school from other investments in social infrastructure and human wellbeing (very technical terms, I know). Seems like a country that’s building a stronger school system is likely building a stronger society at the same time. In fact, knowing what we do about the relevance of non-school factors in test scores, I’d be inclined to say that the rising test scores and later economic growth are both effects that provide some evidence of a stronger, healthier society. Are there nations that have undertaken significant school improvement efforts while taking a neutral position on health care, housing, social services, transportation, etc.? Seems unlikely. Meanwhile, if a society is willing to invest in all of the above, a bump in the test scores would seem reasonable for children whose families and home lives are in better condition.