e careful if you ask Leon Botstein, the president of Bard College in Annandale-on- Hudson, N.Y., to discuss the U.S. News & World Report annual "America's Best Colleges and Universities" issue.
Suddenly, a genteel conversation with a bow-tie wearing college president can turn into something like a wild-eyed rant.
"The criteria are ludicrous," Mr. Botstein said last week. "It is the most successful journalistic scam I have seen in my entire adult lifetime. A catastrophic fraud. Corrupt, intellectually bankrupt and revolting."
Call it revolting or call it masterful, the best-colleges issue of U.S. News will hit newsstands on Sept. 10. and will burnish or dull the reputations of colleges and universities across the country, stir already addled adolescent hormones and cause status-conscious parents of high school students to run for the Xanax.
Since its debut in 1983, the annual list has achieved a totemic respect, especially for well-to-do parents willing to spend any amount to get Junior into Harvard (No. 2 on last year's list, after Princeton).
The issue's publication each year causes bitter rivalries among colleges and universities, which feel an effect in the number of applications each time the institutions rise or fall on the list. And students often depend on the annual issue, if not the numeric rankings, at least for reliable basic information about colleges and universities. And, of course, it sells magazines.
But many educators have long questioned its value. And now a former director of data research at U.S. News is publicly debunking the list's fundamental criteria.
In the September issue of The Washington Monthly, Amy Graham - who oversaw the list at U.S. News for two years until her resignation in 1999 - argues that the manner in which the magazine gathers data and ranks colleges "defies common sense" and produces misleading results.
The article, "Broken Ranks," whose co- author is Nicholas Thompson, an editor at The Washington Monthly, says that "the highly influential U.S. News & World Report annual guide to `America's Best Colleges' pays scant attention to measures of learning or good educational practices."
The rankings, the article says, "primarily register a school's wealth, reputation and the achievement of the high school students it admits," and give too much weight to data like an institution's wealth and the College Board scores of the incoming freshman class.
"That's like measuring the quality of a restaurant by calculating how much it paid for silverware and food: not completely useless, but pretty far from ideal," the article says.
Other information the article says is given too much weight includes the opinions of college and university presidents, who rank peer institutions - a factor that makes up 25 percent of an institution's score - and the generosity of alumni giving. Not enough attention, the article argues, is paid to the satisfaction of the students.
Peter Cary, the U.S. News director of special projects, who oversees the annual issue, said he disagreed with Ms. Graham and other critics. For example, Scholastic Assessment Tests "are a measure of the braininess of the entering freshman class, and by extension the braininess of the student body," he said.
Even in the drowsy days of August, within academia the article by Ms. Graham and Mr. Thompson has sounded a call to rhetorical arms. Judith Shapiro, the president of Barnard College (No. 29 on the 2000 list of best liberal arts colleges), said that she had always questioned the basic data upon which U.S. News relies.
"It's the old story about the drunk looking for his car keys under the lamp post - because the light is good there," she said.
Ms. Shapiro said that while she was happy to see information about colleges made available to the public by U.S. News, she disagreed with the idea of rating colleges. The things that make up a college's appeal are not factors that can be numerically measured, she says. "But we know the answer to that," she said. "It sells magazines."
The list has indeed spawned a profitable side business for U.S. News & World Report, at a time when newsstand sales are sinking for all magazines and advertising revenue for U.S. News is down 20.5 percent for the year, through July.
Last year's best-colleges issue of U.S. News, dated Sept. 11, sold 56,444 copies on the newsstand, according to the Audit Bureau of Circulations. That was about 40 percent higher than the magazine's typical newsstand sales.
Time, Newsweek and Business Week also publish guidebooks or special issues that scrutinize almost every educational institution, from private boarding schools to business graduate programs to graduate classes in entomology.
Kenneth Auchincloss is the editor of Kaplan/ Newsweek's "How to Get Into College," which was released last week. It sells about 100,000 copies a year. But it does not rank colleges, he said. "We have never been comfortable trying to quantify in numeric terms the various criteria that go into making a college good or less good," Mr. Auchincloss said. "And we don't want to devote the resources to doing an elaborate statistical analysis that frankly we don't think is valid."
Ms. Graham, who has a doctorate in economics from the University of Virginia (No. 20 on the best universities list), is now working as a statistical researcher for the state of Virginia. She said she left the magazine because the editor who hired her, James Fallows, had left.
Mr. Fallows, as it happens, has an article in the current Atlantic Monthly about college admissions, "The Great College Hustle," which addresses in part how colleges battle each other for the U.S. News rankings.
Ms. Graham said she wrote her article because she wanted to see the list makers conduct additional research among the students themselves before ranking the institutions.
She also said that using alumni donations as a gauge of student satisfaction is a "vastly unreliable" measure, which, she wrote, "depends in large part on alumni wealth and how many pesky sophomores man phone banks."
Mr. Botstein of Bard (No. 39 on the liberal arts list) agreed that alumni giving was not a useful criterion. "What does that mean?" he said. "That you created some passive set of people who sit back musing and consider their 20's were the best part of their lives?"
And Ms. Shapiro of Barnard said the attention to wealth "feeds a serious academic arms race" that is unhealthy for colleges.
But Mr. Cary of U.S. News said the wealth of an institution indicated its health. "How many labs they can set up?" he asked. "How much can they pay their faculty, and what kind of faculty they can attract?"
The magazine works hard to make the list as accurate and fair as possible each year, Mr. Cary added.
"We are not strangers to criticism," he said. He meets with college presidents, deans of admissions, high school guidance counselors and other educators at lengthy meetings throughout the year. "These are not placid discussions."
One of Ms. Graham's specific recommendations is that the U.S. News ranking should integrate results from the National Survey of Student Engagement, supported by the Pew Foundation. Mr. Cary said that he was eager to include more information from students and that he had already considered the Pew research.
"But they only had data from 276 colleges," he said. And the participating institutions had insisted that the student information not be made public, Mr. Cary said.
Stephen R. Lewis Jr., the president of Carleton College in Northfield, Minn., (No. 6 on the liberal arts list), said that the ranking was not necessarily bad and not necessarily good. "The question should not be, what are the best colleges?" Mr. Lewis said. "The real question should be, best for whom?"
