The Princeton Review recently released its latest college rankings. They are meant to be a more holistic, student-response driven survey of col-leges. Unfortunately, for students interesting in credible information, the rankings are entirely unscientific.

The Princeton Review’s Web site states that their “lists give no-nonsense guidance to the academic caliber and cushiness of 345 schools.” At another portion of the site, the Princeton Review claims: “We give you the facts and figures about diversity, sexual orientation, religion.” But the Princeton Review’s findings are very much nonsense guidance and facts and figures not worth putting much stock in.

Over the course of three years, the Princeton Review collected survey information. They gathered 100,158 respondents. Split over the 345 colleges sampled, approximately 300 students per campus on average participated. That’s not a large enough representation from each school from which to draw valid inferences.

Students receiving the survey were able to respond on paper or on the Web. But the key aspect is that students elected to respond or to refrain from responding. This is known as a selection bias.

Students that were sampled weren’t necessarily statistically average students. They very well could be the 300 students with strong attitudes toward a particular matter, which makes them more likely than not to reply, but which overrepresents them in the sample space.

For example, suppose at U.Va., students who fill out online surveys are less likely to be party-goers. Then, by allowing people who are invited to fill out the survey to choose themselves — that is, to fail to have any control over sample characteristics — researchers would find (illegitimately) that U.Va. students party less than they actually do.

As it turns out, students are still able to fill out these surveys online. The Princeton Review, therefore, remains fraught with serious problems if it is at all concerned with doing serious science.

It may very well be the case that the Princeton Review doesn’t consider its work good science. Perhaps the Princeton Review’s project is just to provide qualitative information (in a non-scientific way).

But that seems to be at odds with the Princeton Review’s overblown language which effectively touts its findings as authoritative.

Which they are not.

A bunch of students wanting to make their school known as the biggest party school can get their friends and acquaintances and muster up a few hundred surveys, thereby sending their school to the top of the list.

This survey tells us no more than the People Magazine online poll that elected Hank the Angry Drunken Dwarf the most beautiful man alive. (Note: This survey was plagued by the same self-selection bias that political scientists regard to be so damning).

Just remember to read Princeton Review’s rankings with a measure of caution. It does not convey any statistically meaningful information. So if it’s a fun read you’re looking for, fine, but if it’s the truth you seek, keep looking.