Column: How Mississippi gamed its national reading test scores to produce ‘miracle’ gains

The term has been shooting around the education field and news reports lately with increasing frequency: “the Mississippi miracle.”

The reference is to that benighted state’s surprising success in improving reading scores for its fourth-graders through a focused program of literacy instruction for teachers and pupils alike. It’s now 10 years old, an anniversary that may have inspired the most recent assessments.

Statistics show that Mississippi’s children have gone from having almost the worst scores on the standardized national reading test for fourth-graders in 2013 to narrowly exceeding the national average in the most recent test, administered last year.

The Associated Press was quick to use the M-word in describing Mississippi’s achievement in a May 17 article.

The turnaround there (and smaller gains in Louisiana and Alabama, which enacted similar literacy programs) “has grabbed the attention of educators nationally, showing rapid progress is possible anywhere, even in areas that have struggled for decades with poverty and dismal literacy rates.”

Education writers and the New York Times jumped on the bandwagon. (“Mississippi Is Offering Lessons for America on Education,” was the latter’s headline.)

As an old journalism adage has it, the story is “interesting, if true.”

A close examination of the numbers suggests that it’s not true. Bob Somerby and Kevin Drum, two of the most adept mythbusters in the blogosphere, have done yeomen’s work deconstructing the statistics. Their conclusion is that Mississippi’s program isn’t nearly as successful as its fans assert and may not have produced any improvement at all in fourth-grade reading scores. The apparent gains may be a statistical illusion.

There is a lesson here, however. It’s not about how effectively to teach kids to read, but how difficult it is to teach journalists how to scrutinize statistical claims.

We’ll get to the specifics in a moment. First, it’s important to consider the social and political landscape in Mississippi.

Given Mississippi’s record on social issues, anyone claiming that its education policies have produced a miracle should check their skepticism meter for a possible malfunction. The state hasn’t been a black hole only in educational achievement but in many other respects.

It should not be overlooked that the 2022 case the Supreme Court’s right-wing majority used to overturn five decades of progress in women’s reproductive healthcare rights, Dobbs vs. Jackson Women’s Health Organization, originated as a challenge to Mississippi’s draconian anti-abortion law.

As I observed when the Court handed down the Dobbs decision, Elena Kagan, Sonia Sotomayor and Stephen G. Breyer, were scathing about the healthcare context of abortion bans in Mississippi and elsewhere. “States with the most restrictive abortion policies also continue to invest the least in women’s and children’s health,” they wrote.

More specifically, they draped Mississippi with the mantle of shame. “Perhaps unsurprisingly,” they noted, “health outcomes in Mississippi are abysmal for both women and children.” The state’s infant mortality rate is the highest in the country, they wrote, amid “some of the highest rates for preterm birth, low birthweight, cesarean section, and maternal death.”

Mississippi (designated “MS” in this chart) has the worst healthcare system of any state in the nation. Should we trust its statistics on reading scores?

(Commonwealth Fund)

After Dobbs, Mississippi’s Republican governor, Tate Reeves, promised to do right by his state’s mothers and children. “The next phase of the pro-life movement is focusing on helping those moms that maybe have an unexpected and unwanted pregnancy,” he said, “making sure that those babies, once born, have a productive life.”

He was plainly blowing smoke. As the Dobbs dissent observed, Mississippi “has strict eligibility requirements for Medicaid and nutrition assistance, leaving many women and families without basic medical care or enough food.”

In fact, Mississippi’s rules are among the strictest in the country: An amicus brief filed in the Dobbs case pointed out, families of three lost their eligibility for Mississippi Medicaid in 2021 if their annual income exceeded $5,490.

Mississippi is one of 10 states that never implemented Medicaid expansion under the Affordable Care Act (all under the thumb of Republican legislatures or governors, or both). In his State of the State speech in February, Reeves ascribed pressure to expand Medicaid to the “liberal media.” Actually, the pressure was coming from the state’s hospitals, which have been warning of “catastrophic” hospital closings, especially in rural areas, under Reeves’ policy.

Last year, Reeves returned $130 million to the federal government that had been earmarked for temporary rental assistance, calling the program a “socialist experiment.”

Returning to the reading issue, Mississippi implemented what appeared to be an aggressive attack on its literacy shortcomings in 2013. Its Literacy Based Promotion Act (LBPA) required that pupils who failed to pass a reading test at the end of their third-grade year be held back.

While repeating third grade, they were to receive intensified instruction. Funds were appropriated for “summer reading camps” for poorly performing kids. New teachers were required to show proficiency in literacy education before receiving their certifications. The educational system was reoriented toward the “science of reading.” That includes phonics, a method that teaches reading by breaking words up into sound bites of one letter or more, and showing students how to link them together into words.

As Somerby correctly noted on June 22, the raw NAEP figures showed Mississippi’s fourth-graders narrowly besting the national figures overall. Even when broken down by subgrouping the state’s kids outscored their national peers — white Mississippians beat the national average for white kids, Black kids beat the national average for Black kids, Hispanic kids beat all Hispanic kids, and lower-income kids of all races outscored lower-income kids nationwide.

Truly a miracle.

However…

There were some statistical readings that should have raised questions among the miracle claimants. One is that Mississippi’s Black and Hispanic fourth-graders still scored much lower than white children — and the gap had not changed over the 10 years of the new regime.

In fact, it widened to 28 percentage points in 2022 from 25 points in 2013. The same thing happened with Hispanic pupils, who fell short of white kids’ achievement by 22 points in 2022 from 21 in 2013.

Fourth-grade reading scores in Mississippi seemed to catch up with national averages in 2022. Or did they?

(National Assessment of Educational Progress)

Moreover, whatever gains had shown up in Mississippi’s fourth-grade scores had vanished by the eighth grade, when all students notched exactly the same scores in 2022 as they had in 2013. A teaching program whose gains evaporate over a four-year span doesn’t much warrant the label “miracle.”

What’s the real story? Drum and Somerby focused on the so-called “third-grade gate” implemented by the literacy program — the requirement that third-grade underachievers repeat third grade. In Mississippi, almost 10% of third-graders have been getting held back, a higher proportion than any other state. (Some may have been held back more than once.)

The statistical result of this policy should be obvious. If you throw the lowest-ranking 10% out of a statistical pool, the remaining pool inevitably looks better. Drum went so far as to add those dropped pupils back into the calculation. He found that the gains from 2013 to 2022 completely disappeared. “In other words,” he remarked, “the 2013 reforms had all but no effect.”

It’s proper to note that educators have been casting a questioning eye on Mississippi’s results ever since the state implemented its literacy program. Some noticed the potential for skewing the statistics through the third-grade gate early on.

Black pupils (light blue line) and Hispanic pupils (medium blue) seemed to be catching up to whites (dark blue) in fourth-grade reading by 2019, but they’ve fallen further behind since then.

(NAEP)

It’s also the case that in education, as in many other professional fields, theoretical fashions come and go. It wasn’t very long ago that studies were showing that whole-language instruction — the antithesis of phonics — was associated with improvements in reading scores.

The debate about how to teach reading has been going on in this country for at least seven decades, since the publication of Rudolph Flesch’s 1955 bestseller “Why Johnny Can’t Read.” (Flesch was a phonics advocate.)

No one has come up with a foolproof solution, but the problem may be the focus on instructional technique. The factors that seem to correlate most closely with reading skills are all found elsewhere. Education expert Paul L. Thomas has identified the key factors as “food and work security, healthcare, [and] class size.” Those get largely ignored in the literacy debate, he observed, “even though these conditions combined would dwarf any measurable impact of teacher quality or program/standards quality.” (Emphasis his.)

Those factors are utterly ignored by Mississippi. Its minimum wage of $7.25 an hour, shared with 19 other states, is the lowest in the nation. It’s a “right to work” state, which discourages union organizing that might enable workers to bargain for better pay and job security. Its healthcare system is the worst in the country.

A state with the the nation’s “highest rates for preterm birth, low birthweight, caesarean section, and maternal death,” as the Dobbs dissenters outlined, imposes almost insurmountable obstacles in the way of educational attainment for its children.

As long as Mississippi’s social and economic conditions, especially for Black and Hispanic residents, remain mired just this side of medieval, then its isolated effort to improve reading scores will be doomed to failure, and its statistical gains will look not much better than window-dressing. Journalists, take note.