Doug Lemov's field notes

I’m a big fan of Paul Bruno’s- love reading his posts on Alexander Russo’s This Week in Education blog at EdWeek, follow him on Twitter, find him spot-on when I agree with him and thought-provoking and sensible when I don’t, even though I suspect I might be one of those people who he dismisses as being “reformy” (ouch!).

So I sat up pretty straight in my chair when his most recent blog post announced that improving education didn’t really reduce poverty.  Who Told Us The Education Fights Poverty, Anyway?  Paul is an empirical guy (one of the reasons I like reading him) and presented a data set appearing to demonstrate that high PISA scores not only didn’t correlate to lower levels of poverty but actually correlated to higher rates of poverty. Holy smokes!  I’ve always believed that this was one of the primary values of effective schools and have read lots of studies suggesting it is true (for example, Eric Hanushek’s long term study that found that achievement rates accounted for 80% of differences in GDP growth over 40 years).

We had a brief conversation about it via twitter–conversation is a strong word for what anyone can do via Twitter–and I promised a follow up to explain why I didn’t buy his analysis. With apologies for blogging when i promised an email, here are some thoughts.  My goal is not so much to “win”–i.e. convince doubters including but not limited to Paul that good schools do in fact help reduce poverty–but to start a more robust conversation about Paul’s hypothesis and how you would measure the idea to understand it more fully. After all the goal of using data is, to me, less to prove a point than to understand the factors behind a trend so you can make policies and institutions better.

First, here’s Paul’s data. He writes:

One preliminary way to look at the evidence would be to see if countries with better academic performance also have lower poverty rates.

Out of curiosity I decided to take a first crack at that using results from the 2012 PISA, which tested 15-year-olds in reading, math, and science.

To get an overall picture of each country’s performance I added up the three subject area test scores. (This may not be an OECD-sanctioned method.)

I then compared each country’s overall PISA performance to its pre-tax, pre-transfer poverty rate. (That’s the poverty rate before the government “steps in” by taking money in the form of taxes and redistributing it in the form of welfare payments, etc.)

In general, countries with higher PISA scores also tend to have higher pre-transfer poverty rates. (We’re the red dot.)

I think there are a couple of problems with the approach here. First, I want to restate what we’re looking for a correlation between–the 2012 test scores of 15-year-olds and the poverty rates of their nations in that same year.

The first thing that jumps out at me is that if the data had in fact turned out a strong correlation, I’d be highly skeptical. In 2012, 15-year-olds the world over, like 15-year-olds in most years, had very little ability to mitigate their own poverty, no matter how attentive in school. If you wanted to see an effect you’d need to look at the correlation between national PISA scores for 15-year-olds and poverty rates at some point in the future… 20 or 30 years out?

And while you were at it you probably wouldn’t want to look at a lot of data before drawing any big conclusions. That is, I’d expect to see a national economic effect (whether or not that showed up as a reduction in poverty) if schools were better for a sustained period of time… five years at a minimum to affect something on such a large scale as poverty rates.

I’d also want to think about controlling for other factors since there is so much “noise” here–ie other variables that could have large effects.  So I’d ideally want to test how changes in PISA scores over time affected changes in poverty rates in countries taking the test.

One other thing that jumps to mind also has to do with a comment that Paul made in our twitter exchange. I referenced the noted economist Eric Hanushek’s findings on the strong correlation between test scores and future economic growth:

“If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate.

That may not seem like much, but, as Hanushek notes, world economic growth averages about 2 to 3 percent of GDP annually, so a difference of a full percentage point in growth is massive.  Here’s a chart of Hanushek’s studying the correlation between test scores and GDP growth for 50 countries over 40 years (on the left side).

Anyway, Paul responded to my comments by noting that GDP growth (which is the increase in overall national prosperity) was different from poverty rates (which is a question of distribution).

That made me also think about the correlation between PISA scores and poverty rates.  In the US, I reminded myself, not finishing high school or finishing high school with skills that don’t allow you to go on to college is one of the fastest way to increase your likelihood of ending up in poverty. (Would anyone dispute that? The trend in fact has steepened recently as middle class industrial jobs erode and “knowledge work” is the overwhelming way to the middle class and above)   But that also reminded me that only a minority of PISA test takers are those who end up in poverty; mostly they’re a measure of those who end up participating in the economy productively and framing the question of how productive they’re able to be.  So if i really wanted to look at education’s effect on poverty I’d probably want to look at much at distribution of scores as I would average scores.  Something like this:

Finally, it’s important to look at what we’re looking at when we talk about “poverty.”  My understanding is that Paul may have used GINI as a proxy for poverty but according to the World Bank, GINI is a measure of income distribution rather than of poverty. And this could be a major confounding factor. For example, roughly speaking, being in the bottom quintile of incomes in the United States makes you “poor”…relatively speaking. I do not intend to mitigate the suffering or difficulties that come with that poverty. (In fact, I work primarily with high poverty schools because i care so much about it) but being in the bottom quintile in the US would not make you poor in India or El Salvador.  Your income and quality of life would be far above the mean.  So your opportunity and quality of life even at the bottom of the economic hierarchy in the US is far better than that of people far higher on a relative ranking of poverty within other countries. So if i improved my nation’s schools and GDP growth went up steeply, everyone would become more prosperous and better off and poverty would be mitigated but my country’s GINI scores might not change. In fact they’d be just as likely to become more pronounced–poverty and income distribution are different things.

Anyway, I’m sure i got some of this wrong. And i don’t mean to call Paul out.  But I believe that schools do influence poverty rates and so I think it’s important to engage a discussion about how we’d assess that because 1) it’s important to understand how and for who and 2) If it’s NOT true, and if having better schools does not mitigate poverty and change people’s access to opportunity, we’d better make some serious changes and all of us had better get pretty “reformy” pretty quickly.

Lots of people understand this stuff better than me.  Hope you’ll weight in!

Schools and Poverty