U.S. News' college rankings put to the test

WASHINGTON -- Here is the early September must-do list for middle-class America: Store the charcoal briquettes. Buy school supplies. And check out the new U.S. News & World Report college rankings.

Since 1983, the newsmagazine's annual college guide has proved irresistible to ambitious parents and their children, pushing sales up 40 percent and drawing 8 million visitors to the magazine's Web site (at www.usnews.com), as it is expected to do again beginning at 7 a.m. Sept. 1.

It has also been a source of gastrointestinal distress for college presidents, alumni fundraisers and professional statisticians. They have decried what they see as the list's narrow and uneven treatment of a complex subject (while tweaking their own numbers to try to beat Crosstown U.).

Now, in an internal report acquired by Washington Monthly, there is evidence that U.S. News' own consultants have been telling the magazine pretty much the same thing.

"The principal weakness of the current approach is that the weights used to combine the various measures into an overall rating lack any defensible empirical or theoretical basis," said the 1997 report by the Chicago-based National Opinion Research Center. Not only do frequent changes in how factors are weighted affect the rankings, the report said, but some factors such as graduation rates are counted more than once and vital college characteristics such as academic rigor and student experiences are not measured at all.

U.S. News executives said they haven't changed the weighting system in five years. They said they followed most of the consultants' recommendations and continue to welcome suggestions from universities and interested experts.

"There is one standing body of admissions deans that we meet with regularly, and we have 50 to 100 meetings a year with people from colleges," said Peter Cary, the magazine's special-projects editor.

Washington Monthly writer Nicholas Thompson, who graduated in 1997 from Stanford (ranked sixth in last year's issue), provided The Washington Post with a copy of the research center's report, which he obtained while researching his article on the rankings. Thompson's article, which appears in Washington Monthly's September issue, says the ratings and accompanying articles are helpful to high school students who lack good counselors. He compliments the magazine for its willingness to respond to complaints.

But he bemoans the failure to address the quality of the academic experience at each school: "Do the big-shot professors actually teach? How many hours? Are they good teachers?" And he suggests that U.S. News has tailored its methodology to favor the biggest brand-name colleges -- Harvard, Princeton, Yale, etc. -- because of editors' preferences and their fear that unconventional winners would hurt confidence in the list.

Caltech's unexpected appearance at the top of the national university list last year, Thompson said, was the result of a newly hired statistical expert standardizing the scores for per-student spending. Suddenly Caltech, which spends $192,000 per student compared with No. 2 Harvard's $81,000, jumped to the top. Thompson, citing "sources close to the magazine," said this produced a "bitter internal struggle" over whether to let that result stand.

Cary vehemently denied that there was any such struggle or that magazine editors adjusted the methodology for any reason other than to produce the best list possible. Although editors were surprised to see Caltech ranked so high, Cary said, they instantly agreed "to stand behind the list and our methodology." Since then, he said, they have adjusted the methodology to ensure that schools with unusually high research budgets do not have an unfair advantage.

As the consultants' report notes, the U.S. News rankings use 16 measures, including graduation rate, class size, faculty salaries, student test scores, high school class standing, educational expenditures and alumni giving rate. The greatest weight -- 25 percent of the total -- is given to academic reputation as measured by a survey of university presidents and deans.

The consultants' report joined other experts in complimenting the magazine for forcing universities to calculate these factors in the same way, reducing confusion for students and their families who try to make comparisons. Robert Morse, U.S. News' director of data research, said he joined the magazine's editors in rejecting the recommendation to stop adjusting the methodology except for a review by experts every five to seven years. "That just wouldn't be the way to run an analytical exercise," he said.

Cary said he and Morse agree with the report that it would be good to include survey data on "student experiences outside the classroom and their degree of involvement in the life of the college" -- a change strongly endorsed by Thompson.

But they note that after the report suggested this, it went on to say that "we can think of no way in which such data could be collected at a scale that would be statistically defensible but not at the same time bankrupt U.S. News."