Amanda Ripley Author of The Unthinkable

A few people have asked me to explain in more detail why I think the PISA index of socioeconomic status is a better way to compare the performance of rich and poor kids around the world (versus the breakdown of scores based on how many kids qualify for free or reduced price lunch at a US school). So I’ll do my best for those of you looking to get deep in the weeds on this….

OK, first let’s talk about the PISA index on socioeconomic status. The data for that index is indeed self-reported by the students taking the test, as some of the commenters have noted. I can see why people would wonder if that is reliable. In fact, I had the same question when I first heard about this.

Two things:

First, the research suggests that students are surprisingly accurate when asked specific questions about their family’s situation (for example, see this report on students’ reliability on such questionnaires.)

Secondly, the students are not asked to give their parents’ income per se; they are asked a long list of questions about their parents’ education levels, occupations, the number of books and computers in the home, etc.—all things that give a holistic sense of SES (and some of which, including education level, can better predict educational success than income alone).

Alright, as for the Free/Reduce Price Lunch (FRPL) breakdown of the PISA data referenced by people who insist our low-poverty schools are “No.1” in the world: this data comes from a totally different survey done in the U.S. only. Principals at U.S. schools where some number of students took the PISA were asked this question. They were told to respond in reference to the entire school—not just the students who took the test. So this is already a different unit of measurement than the average PISA scores for, say, Finnish students.

Moreover, the number of principals who said that between 0 and 10% of their students are eligible for FRPL is small; only about 10% of the 2009 U.S. PISA sample attended these schools.

But that’s all well and good. This FRPL data surely gives us a sense of the huge gap between the performance of the 10 percenters and the rest of the schools in the U.S.

But those last three words are key. This data is collected only to look at variance in the U.S. I agree that it would be fascinating to compare these figures to the same figures in Finland and around the world. However, we don’t have that information. We don’t know how Finnish schools with 0-10% of students from families earning less than 185% of the U.S. poverty level do on PISA.

We do know that Finland overall has far less poverty than the U.S. But the oft-cited figure—that Finland has about 4% child poverty—refers to a totally different definition of “poverty” than the FRPL definition. That 4% figure refers to the percent of people who earn less than 50% of the median income in Finland. (The comparable figure for poverty in the U.S. is about 20%—whereas under the FRPL definition of “poverty,” it’s about 40%, to give you a sense of the difference.)

Just to be sure, I spoke to the data experts who crunch this FRPL data in the U.S. and know it far better than I ever will, and they confirmed that it is inappropriate to use this data in the way that Ravitch is using it. You can’t compare the FRPL data from US schools to an entire country; it’s apples to oranges. The best option that I know of to compare apples to apples is PISA’s own ESCS index. And again, on that index, our richest kids do fine in reading—and not well in math and science.

OK, now back to writing the book! If you’ve read this far, you are probably trying to procrastinate doing something, too… Thanks for the company!

The other day, I posted the country rankings you never hear about—the only legitimate ones to show how countries’ most privileged 15-year-olds do on the PISA test of what kids know around the world.*

Our richest kids rank No. 7 in reading. OK, so it is not No.1, as others keep insisting, and we spend way more money per student to get there. But I’ll take it. No. 7 is still a perfectly respectable performance—well above the OECD average for rich kids.

But it got me thinking: What about math and science? How did our most privileged kids (who are, by the way, more privileged than most countries’ well-off children) do in math and science?

Oh Lord…Brace yourselves, suburban parents:

With thanks to the folks at the Education Trust who helped me ferret out this data from the PISA results, here we go:

MATH ACHIEVEMENT of the most privileged teenagers around the world:

1. Belgium

2. Netherlands

3. South Korea

4. Finland

5. New Zealand

6. Japan

7. Switzerland

8. Czech Republic

9. Canada

10. Australia

11. Germany

(Still going…)

12. Denmark

13. France

14. Sweden

15. Austria

16. Hungary

17. Slovak Republic (!)

18. Iceland

(Hang in there…)

19. Luxembourg

20. Ireland

21. Norway

22. Poland

23. UNITED STATES

There it is, No. 23 out of 29 countries in math, according to the 2003 PISA exam (which was the last time math was the primary focus of the test, yielding enough data to make such comparisons).

Wow. How to explain this? Our most privileged kids attend, on average, the most well-resourced schools in the world with some of the smallest class sizes and among the most credentialed, experienced, well-paid teachers. They have educated parents, books at home and computers to use, and this sample includes our private-school students.

And yet they score below the OECD average in math when compared to other countries most-privileged students. What is going on here?

SCIENCE ACHIEVEMENT of the most privileged teenagers around the world:

1. Finland

2. New Zealand

3. Netherlands

4. Canada

5. Australia

6. Germany

7. United Kingdom

8. Czech Republic

9. Belgium

10. Switzerland

11. Japan

12. France

13. Austria

14. Hungary

15. Ireland

16. Sweden

17. South Korea

18. UNITED STATES

In science, our most privileged students ring in 18th out of 30 countries, per the 2006 PISA test (the last one that had science as its primary focus.) This is, as in math, just below the OECD average for similarly affluent kids.

Why does it matter?

I bring this up just to point out that it is possible for kids to learn at much higher levels than our kids are learning—even our most-advantaged kids. I am not (repeat, not!) saying that poverty doesn’t matter; it obviously matters enormously. Let’s just stop talking about poverty as if it is some dark force that acts in isolation from the rest of our institutions.

Even if we could magically eliminate poverty in America (which would be a beautiful thing and something we should try much harder to do), then we still would not have world-class education outcomes.

Anyone care to offer a theory for why our most affluent kids score 23rd in math and 18th in science? Is it a lack of motivation? An overabundance of wealth? If so, why aren’t we below average in reading, too?

*And remember, before you send me links to wildly misleading blog posts and demand a recount: these rankings listed here rely upon PISA’s own carefully administered survey of students’ socioeconomic status known as the index of Economic, Social and Cultural Status—not a hijacked table regarding free-or-reduced price lunch ratios that was never ever intended to be used for international comparisons.

This is the story of how wishes come true in the strange, upside-down world of education.

Edu-pundits like Diane Ravitch like to say that America’s education problems have everything to do with poverty. This is actually a debate that goes back centuries in American schools. It takes different forms at different times, but it almost always follows the same equation: poverty (or race) is a problem so intractable that schools cannot be expected to overcome it. (Fun fact: the same debate was used to defend low-performing, segregated public schools in New York City in the 1960s. Check out this New York Times story from 1963.)

Interestingly, this is not the kind of talk you hear in places with higher-performing education systems. In those countries, the very same countries that Ravitch says should be models for US schools, educators also think poverty is a big problem. But they think it is their problem. They think it is a problem so intractable that our schools must be outstanding in order to help overcome it. See the difference?

Of course, if we think about it calmly for more than 5 minutes, we can probably agree that poverty interacts with schools, like a chemistry experiment. Bad schools make poverty worse, and great schools make it possible to overcome poverty. In fact, great schools are among the most effective anti-poverty measures known to humanity. Neither schools nor poverty work in isolation.

And yet this debate rages on, with a stunning lack of sophistication. To show you what I mean, let’s consider the latest talking point.

Ravitch and others have been saying over and over again that America’s low-poverty (i.e. affluent) schools do even better than Finland. “Low poverty schools, low poverty districts in the US perform just as well in the US as schools with similar demographics in the top nations in the world. They’re number 1. In fact, our children are number 1 in the low poverty districts.”

So if we took away our pesky poverty problem, we’d rank at the top of the world! This point is meant to defend all schools, but mostly it makes upper-income parents feel better about their own kids’ schools.

Too bad it’s not true.

The most respected international tests of teenagers around the world (PISA) has consistently shown that our most-affluent kids do not perform as well as the most-affluent kids in the highest-performing countries around the world (even though our rich kids are richer than their rich kids). PISA measures students’ economic, social and cultural status to get a sense of their socio-economic background. In reading, American kids’ best subject, our most affluent students still rank behind the most affluent kids in six other countries. (Even though we spend far more money per student than all of those countries.)

Rich Kids Ranking (PISA Reading 2009)

1. New Zealand

2. Korea

3. Belgium

4. Finland

5. Canada

6.  Australia

So where is Ravitch coming from? She is, after all, a professor at New York University. Surely she can’t just make these things up, right?

Here’s what’s happening. Bear with me, because it is revealing.

Ravitch’s claim can be traced back to a small table on page 15 of a government report that broke down the PISA results based on the percentage of kids who qualify for free or reduced-price lunch. When you do that, you see that kids at U.S. schools where less than 10% of the students qualify for free/reduced-price lunch score on average very high—indeed higher than the average for, say, Finland.

But then she makes the magical leap. She says that since Finland has less than 10% poverty, and those schools do, too, then…ta da! Our low-poverty schools are best in the world—when you compare rich kids to rich kids.

Here’s the problem: she is using 2 different definitions of “poverty.”

The free/reduced-price lunch figure measures the number of kids from families making 185% of federal poverty line, right? So that means a family of 4 needs to make less than about $40,000 to qualify. Under this measure, roughly 40% of American kids qualify as “poor.”

OK. Then the other measure is the measure usually used in international comparisons of poverty. That is the percentage of kids from families earning less than 50% of the median income in that country. (In the US, this comes out to about 22%. NOT 40%.)

In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).

I don’t have tenure, but even I know you can’t mix and match data like this. Unless you are really, really desperate to find a certain answer, that is. [Edit: After this post went up, an alert reader informed me that Ravitch does not have tenure either; NYU confirms that she is a nontenured “research scientist.”]

Conversely, PISA’s own measure of socio-economic background, the one you can find detailed in Table II.3.1 in PISA Volume II, offers a more valid comparison. And yet Ravitch does not cite it—because it does not show what she wants it to show.

I have been to Finland, Korea and Poland working on this book, and I have the luxury of spending hours reading PISA results. Most writers do not. They just repeat what Ravitch and others say. And so the magical thinking continues.

On Friday, David Sirota repeated this myth in Salon. And it ran again yesterday in the Oregonian.

As 2011 draws to a close, we can confidently declare that one of the biggest debates over education is — mercifully — resolved. We may not have addressed all the huge challenges facing our schools, but we finally have empirical data ruling out apocryphal theories and exposing the fundamental problems.

We’ve learned, for instance, that our entire education system is not “in crisis,” as so many executives in the for-profit education industry insist when pushing to privatize public schools. On the contrary, results from Program for International Student Assessment exams show that American students in low-poverty schools are among the highest achieving students in the world.

What interests me is not so much that fiction gets reported as fact. That is an old story. What interests me is why so many people—particularly liberals—seem to want to believe that poverty is even more intractable than it is…  Why would this be?

I’ll be in Vegas this week giving a talk to the International Association of Emergency Managers. Incredibly, I’ve never been to Vegas before. I know, I know. I’m an embarrassment to my country. So it’s time, and I’m looking forward to catching up with old friends and sources. Plus, what better place to talk about risk, denial and panic?

Here’s a little screed I wrote about testing in Finland and other countries for NBC’s Education Nation blog, the Learning Curve. As Congress debates a rework of No Child Left Behind and our own culture of testing, it’s worth considering what really matters.

Before Jesus Christ was born, human beings were taking tests. Civil service exams date back to China’s Han Dynasty (206 BC – 220 AD.) Hiring test-prep tutors - and cheating - go back about as far, by the way.

U.S. students now take more standardized tests than ever. Under the federal No Child Left Behind law, our kids get tested in grades three through eight, and at least once between tenth and twelfth grade.

Have we lost our minds? Many teachers and critics of school reform insist that we have, citing other, higher-performing nations as evidence of our relative insanity…

First of all, let’s be clear: Finland does have standardized testing. They have had it for at least 159 years. They have less of it, for sure. (Which is not to suggest that they have less testing overall, but more on that later.) In fact, in every high-performing nation, tests are embedded in the wiring of schools - particularly in high schools. In the developed world, 76 percent of students attend high schools that use standardized tests, according to the OECD.

Read more here.