Columbia Loses No. 2 Rankings Spot. Are Rankings Harming Higher Ed?

Colleges and Universities love rankings. “Rising to its highest position ever, Columbia has placed second in U.S. News & World Report’s annual rankings of the nation’s best universities,” crows a press release on Columbia University’s website.

But Columbia was one of 10 institutions that have been pulled from the U.S. News list in the past couple of weeks due to questions about the data they supplied that was used to calculate the rankings. The press release has since been removed from the school’s site and can only be accessed using Wayback Machine to retrieve an archived copy.

Schools all want to talk about how highly they rank until questions arise about whether they have been gaming those rankings. Students and their families rely on rankings to help decide which colleges to apply to and attend. What if rankings cannot be trusted? There are two reasons to look at them with suspicion.

First, rankings push institutions to focus on the things they measure. U.S. News uses multiple criteria when calculating its rankings. These criteria include graduation and retention rates, average SAT or ACT scores of newly admitted students, and how selective the institution is. The rankings also consider institutional reputation, faculty resources, average class size, and the average debt recent graduates leave with.

The incentive structure created by the rankings pushes institutions to focus on metrics that do not play a core role in the education provided to students. Since U.S. News rankings emphasize how selective an institution is compared to how good a job it does of graduating students from low-income backgrounds, institutions are more likely to focus on admissions selectivity over assisting students who often need the most help getting to graduation. The same can be said of prioritizing average SAT scores of incoming students and alumni giving over average class size. For example, the reputation survey that U.S. News sends to high-level administrators, which seems like a back-patting exercise, is given a 20 percent weight in the rankings, compared to the 2.5 percent weighting given to the graduation rate of Pell grant-eligible students.

In addition to focusing colleges on the wrong things, rankings also incentivize gaming the system, which is the second problem. It feels like every few months there is a new rankings scandal. This year it is Columbia, which U.S. News has now moved to “unranked” due to multiple errors in reported data from the school. The errors were uncovered by Michael Thaddeus, a Columbia math professor who found significant discrepancies in how the institution reported the data used by U.S. News to calculate the rankings.

Columbia was moved to “unranked” because it has yet to respond to inquiries from U.S. News about its data. Columbia has said it is investigating the issue and will not submit data to the rankings until it has created a more robust process for ensuring the data it supplies are accurate. Columbia is far from alone in playing fast and loose with data that could boost its rankings. In the past five years, multiple institutions have retracted or revised data. These are not minor errors; a few examples show they appear to be more than simple oversights.

Villanova University reported that its average need-based grant was just over $51,000, a figure that helped it get listed as a best value institution. The correct number was approximately $10,000 less.

Whitman College reported that the average federal debt for its 2020 graduates was $4,854 and that a mere 25 percent of graduates that year had loan debt. The correct amount was $17,294, and 37 percent of students. Given the looming student loan debt crisis, it is more telling that total debt and percentage of graduates who borrow only have a 3 and 4 percent weight, respectively, in the rankings calculations.

Just last year, Temple University’s business school dean was sentenced to 14 months in prison for his role in deliberately falsifying data that helped push the university’s business school up the rankings.

Rutgers is currently being sued for allegedly hiring students from its business school to make it look like it had better employment outcomes than reality in an effort to boost rankings.

U.S. News says that a tiny fraction of the institutions it provides rankings for each year—less than 1 percent—say that they have supplied incorrect data. But this is all self-reported data.

Other rankings, such as those from the Washington Monthly magazine,¹ give more weight to things like the economic mobility institutions provide their graduates and how much the institution promotes public service. These rankings look very different from the U.S. News rankings but also get much less attention in the media. It is possible that this will change with the seemingly never-ending drop of scandals related to other rankings, but so far U.S. News has maintained its hold on the rankings world in spite of repeated questions about the perverse incentives it has created.

What we measure matters a great deal, and the metrics U.S. News uses push institutions to care about some things more than others and arguably the wrong things. These decisions are not neutral, they distract institutions from their core missions and potentially harm the students who use the rankings to make decisions about their future.