Netflix $1 million prize

Netflix is offering a $1 million prize. This sounds like something that a Freakonomics blog reader should try to win:

Netflix is all about connecting people to the movies they love. To help customers find those movies, we’ve developed our world-class movie recommendation system: CinematchSM. Its job is to predict whether someone will enjoy a movie based on how much they liked or disliked other movies. We use those predictions to make personal movie recommendations based on each customer’s unique tastes. And while Cinematch is doing pretty well, it can always be made better.

Now there are a lot of interesting alternative approaches to how Cinematch works that we haven’t tried. Some are described in the literature, some aren’t. We’re curious whether any of these can beat Cinematch by making better predictions. Because, frankly, if there is a much better approach it could make a big difference to our customers and our business.

So, we thought we’d make a contest out of finding the answer. It’s “easy” really. We provide you with a lot of anonymous rating data, and a prediction accuracy bar that is 10% better than what Cinematch can do on the same training data set. (Accuracy is a measurement of how closely predicted ratings of movies match subsequent actual ratings.) If you develop a system that we judge most beats that bar on the qualifying test set we provide, you get serious money and the bragging rights. But (and you knew there would be a catch, right?) only if you share your method with us and describe to the world how you did it and why it works.

Serious money demands a serious bar. We suspect the 10% improvement is pretty tough, but we also think there is a good chance it can be achieved. It may take months; it might take years. So to keep things interesting, in addition to the Grand Prize, we’re also offering a $50,000 Progress Prize each year the contest runs. It goes to the team whose system we judge shows the most improvement over the previous year’s best accuracy bar on the same qualifying test set. No improvement, no prize. And like the Grand Prize, to win you’ll need to share your method with us and describe it for the world.

I love the Netflix approach to the problem. They could easily spend $1 million internally hiring some programmers or Ph.D’s to try to improve their algorithm, with uncertain results. Instead, by making it a contest and offering up data to outsiders, they will probably succeed in having 100 times as many person-hours devoted to the problem for the same price — or cheaper because they only pay out the million if someone really improves on what they are doing now. In addition they gets lots of free publicity. Truly a brilliant strategy.


Crosbie

Watch out for the '42' syndrome though.

I note they aren't also challenging programmers to demonstrate the correctness or wisdom of the problem's specification.

They may well get a solution to the problem they've stated, but they may not realise that they've failed to state the problem they actually have.

After they've paid out $1m, they may discover they need to propose a far trickier challenge, e.g. "$10m to whoever tells us how our challenge should have best been specified given our actual problems and requirements".

;-)

Craig

Too bad netflix is a scam company and they're going out of business soon.

711buddha

The idea is brilliant. While they will undoubtedly get more hours of work this way, much of it will be redundant. The real question is, how much original work will they get.

benmiller

1 percentage point up; nine to go.
http://www.netflixprize.com/leaderboard
Not bad for a week.

cbarna

This is a variation on "crowdsourcing". See the Wired article "The Rise of Crowdsourcing" http://www.wired.com/wired/archive/14.06/crowds.html

Glo

Ha! The goal of a 10% improvement has already been achieved. According to this story on Slashdot, an academic type has reached the goal.
http://slashdot.org/article.pl?sid=06/10/09/1344235

kelan

The goal has not been met. The Slashdot story points out that a competitor has already beaten Netflix's system, but not by the 10% needed. They only have about 1% at the moment.

TheDumbSmartGuy

Thanks kelan, I was just about to point this out. Still, doing as well as NetFlix in just a week is quite a feat.

Doug Nelson

Pardon my math ignorance, but don't they have programs that can retroactively determine patterns based on endpoint (the most recent movie rated), startpoint (the first movie rated the same), and midpoints along the way (movies rated the same in the interim period).

Or, more likely, ignore date and lump all movies rated the same by one viewer into one pile and datamine it for patterns. Then dump the patterns together and datamine them for commonalities?

There are only so many possible variables (though that finite list will still be quite large). Genre (likes westerns), lead actors (James Woods fan), director (fool for Miyazaki), etc. I suspect time will be important in one factor, which is date of rental from release date (slave to fads).

I also believe Netflix currently ignores queue entry date (anyone that already has Scorsese's The Departed in their queue is probably going to tend to be more favorable to it than if they add it several months after the DVD release).

So many movies, so little math ability :)

Read more...

jeffstier

Smoking is a deadly habit, and fighting the addictive properties of nicotine is hard enough without fighting taste addictions too. According to Tavis Smiley in the New York Times, the taste of menthol cigarettes may increase the difficulty people have when trying to quit smoking. The leading smoking cessations tools have unacceptably low success rates, in part because they don't meet the nicotine needs of addicted smokers and also do nothing to help the smokers deal with taste addictions. However, it's not the nicotine, but the inhalation of the smoke that is most harmful to human health.

In order to avoid the lung damage and other health problems associated with cigarette use, smokeless tobacco can be used as a smoking cessation tool for those who have failed with other methods http://www.acsh.org/publications/pubID.1403/pub_detail.asp . Even the risk of oral cancer is less with oral tobacco such as snus, than from smoking cigarettes. Like the nicotine gum or patch, smokeless tobacco can satisfy the nicotine cravings experienced during smoking cessation, and new products like Taboca Green, a menthol flavored “spitless” tobacco, could help people deal with the taste cravings as well.

The American Council on Science and Health is careful to mention that smokeless tobacco is harmful and should not be considered a safe alternative to smoking, however the evidence certainly makes it clear that it is less harmful than smoking cigarettes, and as such, should be considered another valid option to ease a smoker's transition from cigarette use to a tobacco free lifestyle.

Read more...

jeffstier

Sorry- that was supposed to be posted on the menthol blog!

meomaxy

One should not conclude that Netflix's programmers are a bunch of idiots just because a group has already beaten the performance of Netflix's own prediction engine.

In the real world, Netflix's prediction engine operates under constraints that are not part of the contest. The real system deals with many more customers and must generate predictions fast enough to show on their web site. The competitors are free to use as much computer power for as long as they want to without worrying about whether the system is practical for real time results or can handle orders of magnitude more data.

Netflix might be very aware of the shortcomings of their system. They may even have some ideas for ways to improve the system. Maybe they have 100 worthy ideas to try out but only a small fraction would actually make a real improvement. Rather than toiling on the 100 ideas, let outside teams of competitors hash it out.

Netflix can watch one incremental change build on another as the contest progresses, and only after there is a 10% improvement do they actually have to get their engineers seriously working on the issue of making somebody's contest entry into a real-world scalable implementation.

Read more...

meomaxy

From here:
http://blog.recommenders06.com/wp-content/uploads/2006/09/bennett.pdf#search=%22cinematch%22

2 million new movie ratings per day
2 days to retrain on new ratings
1 billion predictions made per day

Netflix software engineers have lots of things to focus in addition to improving prediction accuracy when they also must ensure that they can deliver tens of thousands of predictions every second.

zbicyclist

Netfix is certainly getting PR out of this. I've had two people walk into my office and seriously suggest working on this. (1) With me, of course, doing the math part and they splitting the prize or (2) hiring math types from the nearby university for $little to work on it, with me managing them.

I guess I appreciate the compliment, but $1mm * odds of winning doesn't compute for me personally.

I predict the winner will be from the former Soviet bloc.
(1) A lot of underemployed math-saavy people over there.
(2) More limited economic alternatives locally. I wonder what the earnings of an academic statistician in Ukraine might be these days? [dare we consider the economic position of grad students there?]
(3) More ways to win -- even if they can't get 10%, they might get enough publicity with a smaller improvement to land some consulting gig or a job in the West.
(4) Yes, they won't have the hardware available here. But that might not matter as much as you might think -- especially in terms of my "more ways to win" point.
Meomaxy (#12,#13) notes there are substantial production constrants in the "real problem". So, if they are constrained by slow hardware this might make them more likely to come up with something that can fit the actual time constraints on state-of-the-art hardware.

Read more...

sophistry

I think that there is a good chance that someone from the hedge fund business will win. Of course $1million may not be a lot of money to those guys, but they do already have a lot of the infrastructure in place to analyze large amounts of data, forecast prices under high frequency conditions and be able to do it under very real time constraints with real dollars on the line.

sophistry

Of course the PR from the win will certainly help those hedge funds in attracting more capital... in fact if they get just $50 million more capital, that's an additional $1 million in management fees alone.

meomaxy

I'm guessing that the contest will not last as long as Netflix thinks it will. I think that Netflix's current algorithm is not optimized to compete in this competition, and so improved algorithms will be found quickly. I think that more than one approach will in fact be found to cross the finish line. If Netflix then decides to announce a second contest for an additional 10% improvement over this contest's winner, now *that* will be hard.

hesky

Who will own the Intellectual Property around this development? It seems like they want to have the algorithm in the public domain. Their own development remains their property - but this seems to be able to be reused by any other organisation. How important is this IP to their business model?

Sudden Disruption

Netflix Contest Winning Entry : Link from IMDB

Netflix,

I'd like to commend your efforts to improve Cinematch.

But haven't you overlook the obvious?

While you're waiting for some geek to improve Cinematch by 10%, there's a solution that will MORE than DOUBLE the effectiveness of movie search for Netflix.

Let's take this design from the top...

What's the most powerful tool in any search? That's right, the human mind. And when the object of the search is as personalized and subjective as a movie, even when the human mind is wrong, it's right! Take care to keep the customer in the loop.

What's the next most powerful tool in any movie search? That's right, it's IMDB (Internet Movie Data Base). A simple Netflix button on each IMDB movie page would dramatically improve search and selection for Netflix.

Face it guys - Netflix may have a nice site, but it doesn't even get close to what IMDB will do. It's time to mash something up.

IMDB is not only good with movie data, it's actually one of the best (and fastest) examples of TRUE hyperlink searching on the Internet. It's made the Kevin Bacon game kid's play.

Even if you only consider the simple searches by director, cast, crew, or title, you can get around quicker than any other comparable collection of data in the world. EVERY industry should have such text-based search tools.

IMDB is so quick and easy, I've even used it as a spell checker. When I can't immediately find a word in the dictionary, I do some free association with a movie title containing the word. If I can't remember the actual title, I simply cross-reference to one of it's actors. IMDB is THAT fast. IMDB is THAT effective.

And when you consider IMDB's Keywords, user ratings and compound searches, things improve by another order of magnitude. IMDB's Keywords are what Netflix's Genres SHOULD be. It's where you can find thousands of user-defined topics with the movie selections listed by user-defined popularity. Keywords are worth more than 10% all by themselves.

And don't forget the Power Search for when you want to get technical with complex searches. If you can't find movie data on IMDB, you're not likely to find it ANYWHERE on the Internet.

OK. So lots of IMDB movies won't be on the Netflix list. Button data would be a powerful indicator of what to stock next. And it would still be worth it for the ones that are. It would be so nice to click a Netflix button then go on searching at IMDB with no Cinematch splash screen interruption (hint).

Yes, I already copy and paste the IMDB movie title to Netflix for queue additions, and it works fine. But why not make that step automatic? This feature would be FAR more useful than a 10% improvement in Cinematch any day.

And no, I don't work for IMDB. It's just such an obvious solution, it deserves a blog post.

So, get over to IMDB with a busload of lawyers and geeks. With Netflix's volume, there MUST be a reasonable link fee solution. The rest is just standard technology. And don't let NIH (Not Invented Here) get in your way. Do it before one of your competitors does.

And if you still want to give away some money...

Make the check out to Sudden Consulting.

Thanks a million.

Sudden Disruption
--
Sudden View...
the radical option for editing text
http://www.sudden.net/
http://suddendisruption.blogspot.com
http://suddenart.blogspot.com
http://suddenlifestyle.blogspot.com
http://suddentechnology.blogspot.com

Read more...

owinok

Netflix approach is very sound. I hope that Sir Richard Branson would adopt the same approach with the search for alternative fuels as opposed to spending US$ 3 billion on the search. I can guess that substantial improvements to the algorithm will emerge from this contest.