A Better Way to Rank Colleges?

Amidst another scandal surrounding U.S. News and World Report’s college rankings, economists Christopher N. Avery, Mark E. Glickman, Caroline M. Hoxby, and Andrew Metrick have proposed another option: rankings based on students’ revealed preferences. Here’s the abstract:

We present a method of ranking U.S. undergraduate programs based on students’ revealed preferences. When a student chooses a college among those that have admitted him, that college “wins” his “tournament.” Our method efficiently integrates the information from thousands of such tournaments. We implement the method using data from a national sample of high-achieving students. We demonstrate that this ranking method has strong theoretical properties, eliminating incentives for colleges to adopt strategic, inefficient admissions policies to improve their rankings. We also show empirically that our ranking is (1) not vulnerable to strategic manipulation; (2) similar regardless of whether we control for variables, such as net cost, that vary among a college’s admits; (3) similar regardless of whether we account for students selecting where to apply, including Early Decision. We exemplify multiple rankings for different types of students who have preferences that vary systematically.

Eric Hoover of the The Chronicle of Higher Education comments on another interesting finding in the paper: 

The authors also challenge the assumption that an admission rate is an indicator of desirability. Half of the top 20 colleges in the revealed-preferences list, they found, would fall outside the top 20 if one ranked them only according to their admission rates (the lower the rate, the better, conventional wisdom holds). Notre Dame, for instance, placed 13th on the desirability list, but its admission rate was only the 58th lowest. The University of Virginia placed 20th on the desirability list, but it had only the 76th lowest admission rate.

(HT: Seth Matthew Fishman)

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 19

View All Comments »
  1. SKA says:

    Hidden due to low comment rating. Click here to see.

    Disliked! Like or Dislike: Thumb up 2 Thumb down 9
  2. Gin Song says:

    Wouldn’t this suffer the same under-coverage as the official employment rate?

    I would wager with many fine folk that a significant amount of prospective students opt not to apply to schools for the following reasons: chance of acceptance, chance of financial aid/scholarship, distance from home. (The latter would be an interesting study to see what the ideal distance one’s home should be from the university of their choice. [~200 miles would be my guess])

    The key issue is the assumption that the available universities to accept from includes the most desirable school for the student.

    Thumb up 4 Thumb down 3
  3. Matt says:

    That seems like poor logic to me: I grew up near a state school with a good football team and a lively night scene. Everybody wanted to go there, but not because it was a good school, simply because it was a good time.

    Does a rampant drinking and football culture make a school or just ?

    Well-loved. Like or Dislike: Thumb up 7 Thumb down 2
    • Enter your name... says:

      They restricted their study to high-achieving students, who are more likely to care about the potential for academic achievement than the football team.

      Thumb up 4 Thumb down 1
      • Pdubble says:

        Yes, because 17-18 year old boys and girls who get good grades hate to have a good time, or even really know anything about college other than what they hear about from random people they meet.

        Thumb up 6 Thumb down 3
    • Voice of Reason says:

      Believe it or not, the reality is that almost every school that has a high achieving football and/or basketball team also has a great academic program. Who knows why, probably because it led to more applications, which allowed the admissions officers to be more selective, which led to more successful and wealthy alumni donors.

      The reality is that there are few educational instituations yet that are top of the line that aren’t a sports powerhouse in something. Exceptions being Ivy League schools (although they might not be much of an exception because their students take league games very seriously, and most excel in the more obscure sports), MIT/Cal Tech, historically strong small liberal arts schools, and performing arts schools.

      These days, it’s hard to find high paying employment from small liberal arts schools, and you’re still stuck with $300,000 in student debt. The sports schools offer great eduction, real job training, and strong alumni connections.

      Thumb up 0 Thumb down 0
  4. Seminymous Coward says:

    This would produce a weird feedback loop where students choose to go to the places ranked higher because those are the places students last year chose to go.

    Well-loved. Like or Dislike: Thumb up 16 Thumb down 1
    • Enter your name... says:

      That’s no different from any other good in the market. People buy clothes, houses, books, and food by considering what’s popular with people who are believed to be knowledgeable about these things.

      Thumb up 3 Thumb down 0
      • SKA says:

        With one minor exception… the “people who are supposedly knowledgeable” in our case is “a cohort of high-achieving high school students who have never been to college in the first place!”

        Well-loved. Like or Dislike: Thumb up 11 Thumb down 0
      • Seminymous Coward says:

        As SKA points out, most of those involve popularity amongst those who have used the good in question. This involves the opinion of those entering college, not leaving it. Also, most college attendees only ever attend one college, so there’s a problem with their frame of reference even if it were to be based on data from the exit side.

        Thumb up 2 Thumb down 0
      • Steve says:

        Might it be better to rank schools by people who are seeking to go for a graduate degree? If a person likes the program at their school enough, they’ll put that school at the top of their list, thus boosting its status. If not, they have more knowledge about which programs and schools are better to go to than they did as high-schoolers. They might fit better into the “knowledgable people” category, while still avoiding people just looking for a party (since grad school ain’t no party).

        Thumb up 0 Thumb down 1
      • tung bo says:

        @Steve
        Looking at grad student choices don’t help that much. They are going after a degree in a specific major while college freshmen rarely have such a focused plan. They probably only have general idea of what they are intersted in and could end up majoring in a range of majors. While these majors are likely to be related, it is also valuable for the freshman to have choices among them within one university. Thus, single major focused data points won’t reveal the value of this ‘option’.

        Also, there are good undergraduate schools which do not have a graduate program. They would be left out of your data completely.

        Thumb up 2 Thumb down 0
    • Clancy says:

      That was my thought too. The students’ choices are based of their perceptions and limited information. If you are looking at all the aspects of a college and trying to decide if it is a good fit for you, how does it help to know what thousands of other people chose using the same information, and not knowing whether or not they made the right choice?

      Thumb up 1 Thumb down 0
  5. Ian says:

    Isn’t this just begging the question? Won’t many ‘high achieving’ students preference for a college be based on the exact rankings that these ones are trying to replace?

    Thumb up 2 Thumb down 0
    • tung bo says:

      I think it’s pretty clear that this study is showing the ‘desirability’ of the colleges and NOT the ‘quality’ of the colleges. Thus, external opinions such as magazine rankings would affect people’s perception and the desirability.

      I think this part is much more important: ” We exemplify multiple rankings for different types of students who have preferences that vary systematically.”

      Trying to have 1 single list of ranking is ridiculous given the variation in college applicants.
      There should be separate lists for: “math/science”, “health care”, “liberal arts”, “sports”, “business”, “politics”, etc.

      Thumb up 4 Thumb down 0
  6. Pdubble says:

    This is just as stupid as ranking based on admissions denied. Who is ranking on the quality of education? The reality is, outside of some dedicated head hunters, the real ranking that matters is your school’s name recognition. These rankings are apparently so bogus that all anyone in the real world is concerned with is that the school actually exists.

    Basing things on applicant preference is kind of stupid really. This assumes people who apply (17 year olds) know going in which is the best school ( “best” is highly subjective anyway) and use the complete information to pick the best school. Look at the real problem. The rankings the students have been subjected to this far are being calculated wrongly, so how are we to believe that a mass of high school students will, en masse, be able to weed through the losers and go after that school that offers the best educational experience?

    Why not see how many schools will submit to some kind of test on incoming students, and then test outgoing students? You could use some kind of algorithm to factor in drop out/transfer rate/night lie/football team and roll from there. I know I got a 98% on my MFAF leaving my college, I’d be interested to see where I was when I went in. I could tell you if the money was worth it then.

    Thumb up 0 Thumb down 0