Calling All Data Nerds…

A few people have asked me to explain in more detail why I think the PISA index of socioeconomic status is a better way to compare the performance of rich and poor kids around the world (versus the breakdown of scores based on how many kids qualify for free or reduced price lunch at a US school). So I’ll do my best for those of you looking to get deep in the weeds on this….

OK, first let’s talk about the PISA index on socioeconomic status. The data for that index is indeed self-reported by the students taking the test, as some of the commenters have noted. I can see why people would wonder if that is reliable. In fact, I had the same question when I first heard about this.

Two things:

First, the research suggests that students are surprisingly accurate when asked specific questions about their family’s situation (for example, see this report on students’ reliability on such questionnaires.)

Secondly, the students are not asked to give their parents’ income per se; they are asked a long list of questions about their parents’ education levels, occupations, the number of books and computers in the home, etc.—all things that give a holistic sense of SES (and some of which, including education level, can better predict educational success than income alone).

Alright, as for the Free/Reduce Price Lunch (FRPL) breakdown of the PISA data referenced by people who insist our low-poverty schools are “No.1” in the world: this data comes from a totally different survey done in the U.S. only. Principals at U.S. schools where some number of students took the PISA were asked this question. They were told to respond in reference to the entire school—not just the students who took the test. So this is already a different unit of measurement than the average PISA scores for, say, Finnish students.

Moreover, the number of principals who said that between 0 and 10% of their students are eligible for FRPL is small; only about 10% of the 2009 U.S. PISA sample attended these schools.

But that’s all well and good. This FRPL data surely gives us a sense of the huge gap between the performance of the 10 percenters and the rest of the schools in the U.S.

But those last three words are key. This data is collected only to look at variance in the U.S. I agree that it would be fascinating to compare these figures to the same figures in Finland and around the world. However, we don’t have that information. We don’t know how Finnish schools with 0-10% of students from families earning less than 185% of the U.S. poverty level do on PISA.

We do know that Finland overall has far less poverty than the U.S. But the oft-cited figure—that Finland has about 4% child poverty—refers to a totally different definition of “poverty” than the FRPL definition. That 4% figure refers to the percent of people who earn less than 50% of the median income in Finland. (The comparable figure for poverty in the U.S. is about 20%—whereas under the FRPL definition of “poverty,” it’s about 40%, to give you a sense of the difference.)

Just to be sure, I spoke to the data experts who crunch this FRPL data in the U.S. and know it far better than I ever will, and they confirmed that it is inappropriate to use this data in the way that Ravitch is using it. You can’t compare the FRPL data from US schools to an entire country; it’s apples to oranges. The best option that I know of to compare apples to apples is PISA’s own ESCS index. And again, on that index, our richest kids do fine in reading—and not well in math and science.

OK, now back to writing the book! If you’ve read this far, you are probably trying to procrastinate doing something, too… Thanks for the company!

EducationAmanda Ripley