I'm a bit late to the party on this, but I couldn't resist
A rather obnoxious twit by the name of Richard Vedder has set up a
front-group called "The Center for College Affordability and Productivity". The goal of this group is purportedly to apply market-based mechanisms to the problems of higher education
in America. When you take a look at their "research", you'll quickly recognize that this is astroturf, plain and simple.
A typical example of this is described in an article Dr. Vedder recently wrote for Forbes magazine about a supposed research study done by his organization on college rankings. According to Dr. Vedder, the popular "US News and World Report" college rankings are no good, and that market-based principles can produce a better, more meaningful ranking. The rationale for this new ranking system is that the standard rankings are based
on the input to the schools: schools are ranked based on the quality of students admitted. Dr. Vedder wants to rank schools based on outcomes: how well the school achieves the goal of
turnings its students into educated, successful people after they
graduate. According to Dr. Vedder, his ranking system tries to rank
schools based on several "output" measures: "How do students like their courses?", and "What percentage of students graduate?", "How many awards do the students recieve?", and "How successful are students after they graduate?"
Ranking schools based on their graduates rather than their enrollees
is an interesting idea - and it's a worthwhile goal. But it's pretty hard to
do well. Just to point out one obvious problem, how do you define success in a quantifiably way?
Just to drive that point home, let's compare a few basic graduates, each
of which is arguably highly successful.
- Me. I'm a software engineer at Google. It's a great job, at an amazing
company which is incredibly selective in its hiring process. I make
a good living, but I'm not rich, and I probably never will be.
- A friend of mine who graduated from college the same time I did. He
went and took a job with Salomon Brothers on Wall Street. He became
extremely rich. He also became an alchoholic and totally burned out.
Last I heard, he was still an alchoholic, but had quit working, and was
living on the money he made in his 10 years on Wall Street.
- The husband of my PhD advisor. Terrific guy. Graduated from college,
and got a job working for an oil company. Hated it. So after a few years,
he quit, and became a high school science teacher. Now, last I heard,
he was incredibly happy with what he was doing, and his students loved him.
- Larry Page. Dropped out of grad school at Stanford to start a business
with his friend Sergei. Now he's one of the richest people in the world. I haven't met him personally, but from everything I've heard, he's a pretty happy guy, and he helped start the amazing company that I love to work for.
- George W. Bush. Graduated from great schools, which he got into through parental connections. Before he got into politics, he ran several terribly
unsuccessful businesses, and lost a huge amount of money. He also became
an alchoholic. Then he straightened himself out, and got into politics, and got elected first governor of Texas, and then president of the US.
Which of those people are successful? Am I? I'll never be rich. I'll never own my own business. I'll never be famous outside of a very small community of my peers. Is my college friend? He's filthy rich. Even if he never works
another day in his life, he'll be able to live comfortably, as long as he's a little bit smart about where he puts all that money. But he's an alchoholic. Larry? I think everyone would agree that Larry Page is an extremely successful guy. The science teacher? He's doing something valuable, which he loves doing, but the pay is garbage. Someone with a master degree in geology can do a hell of a lot better, money wise. How about our president? A man known for his utter lack of intellectual curiosity, who before getting into politics was
an unquestionable failure at business; who was, by almost any standard, an utter failure before getting into politics?
Which ones are successful, and which aren't? Which is most successful? How can you quantify it?
Of course, Dr. Vedder doesn't worry about questions like that. To people like him, people who believe in what he means by "market based approaches", money is all that matters. Bush is a success - because no matter how much money he lost, he always had the ability to raise more through his family connections - so he'd be rich, no matter what, and wealth is success. Likewise, the alchoholic is a success - he's rich. I'm moderately successful, but not very. Larry is obviously successful. And the science teacher is clearly not a success, because the outcome of his education was getting a job that he hated, and ending up becoming a teacher.
I'd say that just ranking on wealth obviously doesn't work. But how can
you quantify real success? You could probably find a way to do it - but it
would be a significant amount of work. And people like Dr. Vedder aren't
interested in doing real work. They're interested in getting lots of publicity
in order to (A) get really big paychecks, and (B) advance their political
agenda. Doing real research doesn't qualify under either of those.
So how does Dr. Vedder generate his outcome based college rankings? He
uses two inputs:
- Course quality: The average ranking of the school's courses on "ratemyprofessors.com".
- The percentage of students who having listings in "Who's Who in America".
He also claims that they use the number of students who win awards like Rhodes scholarships and the graduation rate, but in the fine print, you'll find that the rankings they've released don't include that data yet.
So... We've got two inputs. One is an anonymous teacher ranking system. It includes 5 basic scores for each professor: easiness, helpfulness,
clarity, hotness, and overall quality. Yes, you read that right: one of the
major things that they ask you to rate a professor on is "hotness". That's the best thing that Dr. Vedder could find for evaluating course quality.
The "graduate success" ranking is no better. For those of you who haven't heard of "Who's Who", it's an elaborate vanity scam. Basically, this publisher sends you very fancy, formal, personalized letter explaining how "Who's Who" publishes an extremely selective list of the most successful people in America - and they'd like to include you. All you need to do is write up
a your biography for them, summarizing your (doubtless) highly impressive
career, and they'll include you in the next edition of the book. Of course,
as a member of the Who's Who directory, it's absolutely essential for you to have a copy! So just send in your biography, along with a nice hefty check,
and presto! You're in! (The last time I got a Who's Who offer was about 10 years ago, and at the time, they wanted $150 for the "prestigious leather bound volume".)
So... According to Dr. Vedder, you can evaluate colleges and universities
better that US News and World Report, using "market based principles"; and that market-based principles dictate that successful people are likely to be taken it by vanity scams.
This is what I call obfuscatory mathematics. The idea is that you have a predetermined outcome, and you want to find some way of using numbers to
produce that outcome. It doesn't matter whether it makes any sense. In fact, the point of it isn't to make sense - it's to appear to make sense. If you look at the press that Dr. Vedder has gotten, they uniformly talk about how his rankings are "outcome based". They almost never go into any detail on what that means, and they never go far enough to explore the validity of his data sources. The data is garbage, utterly worthless for drawing any meaningful conclusion. But Dr. Vedder doesn't want a meaningful conclusion. What he wants is to make an argument that higher education in America is hopelessly screwed up, and that only ultraconservative "market based principles" can possibly save it. In order to support that argument, he has to show that what we consider to be the top elite colleges aren't as good as we thought. Using college reviews that include inspector "hotness" will produce that result - and so he uses it.