Reality Distortion Field
This is the story of how wishes come true in the strange, upside-down world of education.
Edu-pundits like Diane Ravitch like to say that America’s education problems have everything to do with poverty. This is actually a debate that goes back centuries in American schools. It takes different forms at different times, but it almost always follows the same equation: poverty (or race) is a problem so intractable that schools cannot be expected to overcome it. (Fun fact: the same debate was used to defend low-performing, segregated public schools in New York City in the 1960s. Check out this New York Times story from 1963.)
Interestingly, this is not the kind of talk you hear in places with higher-performing education systems. In those countries, the very same countries that Ravitch says should be models for US schools, educators also think poverty is a big problem. But they think it is their problem. They think it is a problem so intractable that our schools must be outstanding in order to help overcome it. See the difference?
Of course, if we think about it calmly for more than 5 minutes, we can probably agree that poverty interacts with schools, like a chemistry experiment. Bad schools make poverty worse, and great schools make it possible to overcome poverty. In fact, great schools are among the most effective anti-poverty measures known to humanity. Neither schools nor poverty work in isolation.
And yet this debate rages on, with a stunning lack of sophistication. To show you what I mean, let’s consider the latest talking point.
Ravitch and others have been saying over and over again that America’s low-poverty (i.e. affluent) schools do even better than Finland. “Low poverty schools, low poverty districts in the US perform just as well in the US as schools with similar demographics in the top nations in the world. They’re number 1. In fact, our children are number 1 in the low poverty districts.”
So if we took away our pesky poverty problem, we’d rank at the top of the world! This point is meant to defend all schools, but mostly it makes upper-income parents feel better about their own kids’ schools.
Too bad it’s not true.
The most respected international tests of teenagers around the world (PISA) has consistently shown that our most-affluent kids do not perform as well as the most-affluent kids in the highest-performing countries around the world (even though our rich kids are richer than their rich kids). PISA measures students’ economic, social and cultural status to get a sense of their socio-economic background. In reading, American kids’ best subject, our most affluent students still rank behind the most affluent kids in six other countries. (Even though we spend far more money per student than all of those countries.)
Rich Kids Ranking (PISA Reading 2009)
1. New Zealand
2. Korea
3. Belgium
4. Finland
5. Canada
6. Australia
So where is Ravitch coming from? She is, after all, a professor at New York University. Surely she can’t just make these things up, right?
Here’s what’s happening. Bear with me, because it is revealing.
Ravitch’s claim can be traced back to a small table on page 15 of a government report that broke down the PISA results based on the percentage of kids who qualify for free or reduced-price lunch. When you do that, you see that kids at U.S. schools where less than 10% of the students qualify for free/reduced-price lunch score on average very high—indeed higher than the average for, say, Finland.
But then she makes the magical leap. She says that since Finland has less than 10% poverty, and those schools do, too, then…ta da! Our low-poverty schools are best in the world—when you compare rich kids to rich kids.
Here’s the problem: she is using 2 different definitions of “poverty.”
The free/reduced-price lunch figure measures the number of kids from families making 185% of federal poverty line, right? So that means a family of 4 needs to make less than about $40,000 to qualify. Under this measure, roughly 40% of American kids qualify as “poor.”
OK. Then the other measure is the measure usually used in international comparisons of poverty. That is the percentage of kids from families earning less than 50% of the median income in that country. (In the US, this comes out to about 22%. NOT 40%.)
In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).
I don’t have tenure, but even I know you can’t mix and match data like this. Unless you are really, really desperate to find a certain answer, that is. [Edit: After this post went up, an alert reader informed me that Ravitch does not have tenure either; NYU confirms that she is a nontenured “research scientist.”]
Conversely, PISA’s own measure of socio-economic background, the one you can find detailed in Table II.3.1 in PISA Volume II, offers a more valid comparison. And yet Ravitch does not cite it—because it does not show what she wants it to show.
I have been to Finland, Korea and Poland working on this book, and I have the luxury of spending hours reading PISA results. Most writers do not. They just repeat what Ravitch and others say. And so the magical thinking continues.
On Friday, David Sirota repeated this myth in Salon. And it ran again yesterday in the Oregonian.
As 2011 draws to a close, we can confidently declare that one of the biggest debates over education is — mercifully — resolved. We may not have addressed all the huge challenges facing our schools, but we finally have empirical data ruling out apocryphal theories and exposing the fundamental problems.
We’ve learned, for instance, that our entire education system is not “in crisis,” as so many executives in the for-profit education industry insist when pushing to privatize public schools. On the contrary, results from Program for International Student Assessment exams show that American students in low-poverty schools are among the highest achieving students in the world.
What interests me is not so much that fiction gets reported as fact. That is an old story. What interests me is why so many people—particularly liberals—seem to want to believe that poverty is even more intractable than it is… Why would this be?
The Fact Checker said on December 12, 2011 at 6:15 pm
Facts are inconvenient to you, apparently. But that doesn’t make them ironclad facts:
http://www.dissentmagazine.org/online.php?id=504
The most recent data come from the 2009 Program for International Student Assessment, released in December 2010. PISA tested fifteen-year-olds in sixty countries (plus five non-state entities such as Hong Kong) in reading, math, and science. Consider the results in reading, the subject assessed in depth in 2009: U.S. students in public schools with a poverty rate of less than 10 percent (measured by eligibility for free or reduced-price lunches) scored 551, second only to the 556 score of the city of Shanghai, which doesn’t release poverty data. The U.S. students outperformed students in all eight participating nations whose reported poverty rates fall below 10 percent. Finland, with a poverty rate of just 3.4 percent, came in second with a score of 536. As the level of student poverty in U.S. public schools increased, scores fell. Because of the high overall child-poverty rate (20.7 percent), the average reading score for all U.S. students was 500 (fourteenth place). In short, poverty drags down our international standing (see this Department of Education site).
Amanda said on December 12, 2011 at 6:50 pm
Yeah, this is the very point I was refuting in this blog post. You are using 2 different measures of poverty here.
First, you are citing free/reduced price lunch poverty levels in the U.S.; then you are citing international poverty rates re: Finland. Two wildly different definitions. As I explain. In detail. Above.
Fact Checker said on December 12, 2011 at 7:47 pm
More incredibly painful facts for you. Again, facts can be inconvenient - but just because you don’t want them to be around, doesn’t mean they aren’t facts:
http://www.nytimes.com/2011/12/12/opinion/the-unaddressed-link-between-poverty-and-education.html
Sad that you refuse to acknowledge the indisputable truth.
Donna Gratehouse said on December 12, 2011 at 9:16 pm
Can you really do an apples-to-apples comparison of poverty between nations based strictly on household income? The US has a public assistance safety net but it’s meager in comparison to, say, Finland.
Marie Lawrence said on December 13, 2011 at 11:14 am
Amanda, great piece and good point. I would add that the comparison is additionally unfair because Ravitch et al are trying to compare the scores of kids in rich U.S. *schools* to all kids in Finland. The units of analysis are totally off here. Due to local control over U.S. schools’ curricula, funding, and a whole host of other issues, such a comparison puts U.S. students in schools with the most resources up against (in the aggregate) kids in average schools in Finland. Such a comparison ignores how the respective education systems function and is, I agree, untenable.
Amanda said on December 13, 2011 at 12:01 pm
Thank you for making that point, Marie. You are absolutely right. The free/reduced price lunch data comes from a survey of principals at schools where some kids took the PISA. Only the PISA ESCS index data tracks the socio-economic background of the individual student (and does it in a holistic way that is not based simply on income, which helps mitigate for the problem that Donna raises above.)
Paul Thomas said on December 13, 2011 at 1:36 pm
Most of the data and claims in this piece are false, provably false.
PISA comparisons taking relative poverty into account:
http://nasspblogs.org/principaldifference/2010/12/pisa_its_poverty_not_stupid_1.html
Or look at EVERY data set of the SAT ever compiled—direct and consistent correlations between scores and home income and education levels of parents.
Out-of-school factors account for about 60-86% of measurable student outcome. FACT supported by piles of evidence.
Amanda said on December 13, 2011 at 1:55 pm
Dear Paul,
Again, your link includes the exact same claims that I am refuting above. Which makes me think that perhaps I am not explaining myself well.
Let me try again: Using the PISA scores based on school free lunch data is a fine way to show that poverty correlates with poor school outcomes. With that point, I agree totally. I also agree that out-of-school factors are hugely impactful on student learning, of course.
What I am arguing with generally is the simplistic idea that schools cannot mitigate against those factors to some degree—as they do in many countries around the world.
What I am arguing with specifically is the clumsy attempt to use PISA scores aggregated by schools’ free-lunch data to compare how our affluent kids perform to how affluent kids perform internationally. I won’t repeat the whole explanation here, but that comparison—and the absurd conclusion that our low-poverty schools are “No. 1 in the world”—would not hold up in a basic statistics class for all the reasons I mention (and more.)
OK, enough said! Thanks for reading.
Amanda
Donna Gratehouse said on December 13, 2011 at 1:57 pm
Why so much emphasis on comparing nations when we can compare low poverty schools in the US with high poverty schools in the US and see the stark difference in reading scores? A lot of people seem to be highly invested in the belief that it can’t be poverty when it obviously is.
Paul Thomas said on December 13, 2011 at 6:49 pm
Amanda:
Your response to my comment is much more measured that your original blog and may seem that we have little to disagree on. . .but I posted to support the accurate apples-to-apples comparison of relative poverty/scores from country to country. As the link I provided shows, US outcomes fair well when comparisons take poevrty into account, BUT that isn’t even what we should be discussing. Let’s note two things:
(1) Your strawman: “the simplistic idea that schools cannot mitigate against those factors to some degree—as they do in many countries around the world.” First, NO ONE is saying the first part of your premise, and there is no evidence that schools anywhere are genuinely producing the “miracles” advocates claim about addressing educational outcomes ONLY by school reform. Finland raised its status by implementing SOCIAL reform, reducing childhood poverty, and reforming schools (which is exactly what Ravitch and others are calling for).
(2) Belaboring the connection between test scores and school/teacher quality is in fact cheating everyone. It should stop.
Stuart Buck said on December 14, 2011 at 2:38 pm
Paul: “I posted to support the accurate apples-to-apples comparison of relative poverty/scores from country to country.”
Did you not even understand that the whole post is explaining why your “apples to apples comparison” is nothing of the sort? Disagree with Amanda Ripley if you like, but you have to come up with some argument beyond blithely repeating the exact same thing that she just refuted.
George said on December 14, 2011 at 4:01 pm
The comment section is unreal . . . the blogger debunks this apples-to-oranges comparison of “free-lunch-less” schools in the US versus all schools in Finland, and then these mindless “bots” wander in and try to debunk her by citing the exact same study! Wow, our education system really is falling apart . . .
JJ said on December 15, 2011 at 10:55 pm
“In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).”
Uh, no. She is comparing school districts that have less than 10% of kids getting free lunch. THOSE DISTRICTS HAVE KIDS IN POVERTY! Just fewer than other districts.
If you want to open the Pandora’s box of comparing different countries’ definitions of “poverty,” fine. Hope you take universal health care and many other societal infrastructures into account.
ron said on December 16, 2011 at 2:20 am
Rest assured JJ—Ms. Ripley is not interested in considering different experiences of poverty and how social welfare systems in different countries mitigate or dont mitigate those experiences because she is (in her own words) “desperate to find a certain answer.” She has a book to sell, after all. Her book will probably include some interesting and useful research but it will be based on a faulty premise so she needs to dismiss this data from the NCES report even though it uses the “free and reduced lunch” metric used in almost all education literature—such as the US Department of Education determining that the number of “High Poverty Schools” in the US increased from 12% in 200 to 17% in 2007-08. (One wonders how many High Poverty Schools there are in New Zealand, Finland, and Belgium…I would love for Amanda to make our understanding of international comparisons more sophisticated by looking into that)
Using standardized test score data gives the illusion of objectivity because simple numbers and rankings can be used to categorize the countries and we can lament the fact that we are 17th or 23rd or 7th or celebrate being 1st with disaggregated data but these types of tests really tells us very little about individual students and the international comparisons are even more problematic—even the comparisons from “The most respected international tests of teenagers around the world”—which is not without international critics, by the way: http://www.dissentmagazine.org/online.php?id=128
I’ll be interested to see if Amanda will address your suggestion that universal health care and other social safety net infrastructure should be taken in to account lest we compare apples and oranges when we look at PISA data or if she can see what percentage of schools in leading test score countries have high percentages of low-income students…
Stuart Buck said on December 16, 2011 at 11:09 am
“Uh, no. She is comparing school districts that have less than 10% of kids getting free lunch. THOSE DISTRICTS HAVE KIDS IN POVERTY! Just fewer than other districts.”
You’re completely missing the point. The point is whether we should congratulate ourselves because our districts with only a few kids in poverty are doing OK compared to other entire countries.
JJ said on December 16, 2011 at 12:04 pm
No, Stuart, you’re missing MY point. Ripley takes Ravitch to task for comparing, in Ripley’s words:
“...the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).”
Except Ravitch is NOT doing that; she’s is comparing districts with low numbers of poor kids to countries with low numbers of poor kids. Poor kids are included in both sets. We can pick at nits all day as to whether this is a valid comparison or not, but there’s no denying Ripley has misstated Ravitch’s comparison.
Further: no one is for “congratulating themselves” on our current situation. But Ravitch, myself, and others who are pushing back on corporate reform are trying to make a point: poverty matters, and it undoubtedly colors international comparisons. If you control for poverty, our comparative disadvantage melts away. Ripley is arguing that isn’t correct, because Ravitch is comparing only wealthier American kids to both wealthier and poorer kids internationally.
Except, again - Ravitch is NOT doing that.
Corporate reformers are constantly saying that we use poverty as an excuse not to deal with reform. I say you are using reform as an excuse not to deal with poverty.
Stuart Buck said on December 16, 2011 at 2:28 pm
she’s is comparing districts with low numbers of poor kids to countries with low numbers of poor kids.
Um, the whole point is that you can’t make such a comparison unless you’re using the same definition of poverty as to the U.S. districts and the other countries. Ravitch is not, and doesn’t even seem aware of the issue.
An analogy: Do you understand why it’s not very meaningful to compare proficiency rates on state tests in different states? The reason is that different states have set different standards for proficiency, and without a common standard, comparisons are meaningless.
The same is true here. You can’t say you’re comparing low-poverty US school districts to low-poverty European countries, when you’re using two completely different definitions of poverty: 1) number of kids within 185% of the U.S. poverty line in real dollars; 2) number of kids under half of whatever the median income is in a given country.
Paul Thomas said on December 16, 2011 at 3:28 pm
Prof. Stephen Krashen Discusses Education Reform, High-Stakes Testing, and Poverty [VIDEO]
http://www.ctunet.com/blog/professor-stephen-krashen-discusses-education-reform-high-stakes-testing-and-poverty-video#.Tut2vTpz6Dc.twitter
Income and Education as Predictors of Children’s School Readiness
http://www.brookings.edu/reports/2011/1215_school_readiness_isaacs.aspx
JJ said on December 16, 2011 at 4:38 pm
Stuart, that is an secondary point she is making, and one that is open for debate, as I acknowledge. But, as I said, I don’t think the argument falls well to your side; More here:
http://jerseyjazzman.blogspot.com/2011/12/poverty-shmoverty-part-ii.html
My first point, however, is that Ripley is misstating Ravitch’s position when she says:
“In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).”
Ravitch is not. This should be corrected.
Stuart Buck said on December 16, 2011 at 9:10 pm
OK, JJ, to be more precise, maybe she should have said:
“Ravitch is comparing the test scores of districts in the U.S. where only TEN PERCENT or fewer of the kids come from families earning less than $40,000, to the scores of Finnish kids FIFTY PERCENT of whose families earn less than $40,000.”
Doesn’t make Ravich look any better. She’s still comparing apples and oranges.
JJ said on December 16, 2011 at 10:54 pm
First of all: Stuart, I accept your apology.
Second: Ripley says Ravitch should have pointed to the PISA report, Vol II, Table II.1.3. That table, on p.165, shows the scores based on QUARTILES of SES. It does NOT take into account the shape of income distribution! The top 25% may be far less different - and probably is - in many other OECD countries than in the US.
So now who’s comparing apples and oranges?
This whole argument is ridiculous on its face. Ripley’s contention that the schools in other developed countries are so much better equipped to deal with poverty is a gross violation of Occam’s Razor. They have a strong safety net; we don’t. They have far less income inequity. They don’t have nearly the immigrant population we do, or endemic racism. Are you telling me that doesn’t account for the majority of the problem? Instead, the problem is our horrible, horrible schools? Especially when school-based effects account for, at most, 20% of test-based outcomes?
Seriously?
The notion that we should dismiss this argument because our poor MAY be wealthier than their poor is sophistry of the highest order (again, no mention of the distribution of wealth, or all of the things we pay for in the US that are “free” in other countries).
Ravitch is making a simple point, and to try to dismiss it by saying, “Well, our poor aren’t REALLY that poor” and “they have more people we’d call poor in their non-poor population” is just plain old dumb. While you all pick nits, this sort of reformy logic is poised to usher in an era of privatization and union-busting that has no track record of working.
Enough already. Poverty counts. Address it. Stop the nonsense.
Paul Thomas said on December 21, 2011 at 2:49 pm
Amanda:
Consider this: http://www.dailykos.com/story/2011/12/21/1047543/-Christmas-Miracle-2011:-Poverty-Matters!
efavorite said on December 22, 2011 at 9:56 am
Without getting into the statistical analysis, it seems like the point of this article is to discredit Ravitch.
So, beyond that, what to do about US public schools, with your assertion that poverty does matter: “Bad schools make poverty worse, and great schools make it possible to overcome poverty.”
I disagree that schools can have such an effect. I think schools are just a piece of the puzzle. Here in DC, where school reform has been in full force for almost 5 years, achievement gains have leveled off or dropped and the achievement gap has widened.
Adults pointing fingers at each other gets kids nowhere, but adults honestly analyzing data that counts and acting on it truly in the best interests of the children could make a huge difference.
Unfortunately, it seems to me that the corporate reformers are mainly keeping quiet about the damning data now available about their reform efforts and using their high intellect to obfuscate and protect their own interests, instead of focusing on helping the neediest kids.
PhillipMarlowe said on December 23, 2011 at 10:48 am
One must give Miss Ripley recognition for not repeating Michelle Johnson bogus claim of 90/90. Instead Amanda settled for:
But she did not question the then Ms. Rhee about her 90/90 miracle claim.
Did Amanda ever accuse Michelle Johnson of playing fast and furious with data?
CarolineSF said on December 23, 2011 at 11:23 am
Handling Rhee-Johnson’s obvious resume lies in such an ineffectual manner was extremely unsound journalistic judgment.
Donna Gratehouse said on December 23, 2011 at 3:39 pm
As Bob Somerby of the Daily Howler noted recently, maybe they should fly some reporters to Finland to figure out how they manage the miracle of spending half what we do per capita to provide universal health care to their citizens. $3226 to our $7290 (with millions still uninsured).
James said on June 14, 2012 at 12:46 pm
Wow. Lots of great debate on the integrity and relative comparisons of international test scores. I almost hate to pull the rug out from under all of you, but WHO CARES? Why have we gotten so bogged down in test scores? Where is the evidence that doing well on the PISA means anything at all?
The first international tests were given in the mid 60’s, and the US came in dead last. We’ve been middlin’ to last in every international contest since then, but I would argue that in the intervening 50 years American has done very well by many objective measures: GDP, number of nobel laureates, numbers of patents issued, quality of our universities/research facilities, quality of life, etc.
Why, then, are we bogged down in a debate about how America does on international tests that have not been shown to matter?
The importance of international test scores is yet another one of those “generally accepted” truths about which there is NO EVIDENCE.
Amanda said on June 14, 2012 at 1:48 pm
Actually, there is evidence.
1. PISA scores have been shown to better predict who will go to college than report-card grades.
2. Students who score poorly on the PISA reading test are much more likely to drop out of high school.
3. There is an almost one-to-one match between PISA scores and a nation’s long-term economic growth.
For more on Nos. 1 & 2, see this OECD report:
http://www.oecd.org/document/20/0,3746,en_32252351_32236191_44571668_1_1_1_1,00.html
For more on No. 3, see this one: http://www.oecd.org/document/58/0,3746,en_32252351_32236191_44417722_1_1_1_1,00.html
Ed Fuller said on March 17, 2013 at 1:50 pm
Not surprisingly, document #3 presents absolutely no evidence for your third claim. Even if it did, it relies on 2006 scores. How longitudinal could the study be? It takes a generation for educational improvements to affect economic output. And, if education is so terrible in the US, why is our economy the envy of the world (at least from the owners/investors point of view)?