Yesterday, Sept 21st, Netflix announced that the winner of the $1 million Netflix Prize is "BellKor's Pragmatic Chaos(BPC)," and we congratulate them on their victory. The contest was remarkably close, and now we know precisely just how close it was.
In accordance with Netflix's rules, the RMSE score for the judged half of the submission (the test subset) is rounded to 4 decimal places. The two teams actually tied on this measurement. In this situation, the tie breaker becomes the order of the submissions. Bellkor's Pragmatic Chaos defeated us by submitting just 20 minutes earlier! Talk about close.
Netflix has graciously donated the entire data set for public consumption to: http://archive.ics.uci.edu/ml/datasets/Netflix+Prize. Included in this dataset is the quiz and test set answers, as well as the winning submission from BPC. This allowed us to determine both the precise test RMSE values and to get a little insight into the question: What would have been the result of The Ensemble and BPC working together?
Here are those values:
Test RMSE for The Ensemble: 0.856714 (10.06%)
Test RMSE for Bellkor's Pragmatic Chaos: 0.856704 (10.06%)
Test RMSE for a 50/50 blend: 0.855476 (10.19%)
The percentages are the improvement over the Test RMSE of Cinematch (.9525).
We would once again like to thank Netflix for it's pioneering contest and deft skill in managing it. It is easy to predict that Netflix Prize 2 will be just as well administered. We would also like to thank all of the numerous participants in the contest. The vast interest kept it healthy and marching forward. This little corner of science we all studied was lit up quite brightly for a while.
Contributed by Lester Mackey
Only sixteen minutes remained in the $1 million Netflix Prize competition when I handed over the final set of predictions to the Ensemble team captain. The members of our newly minted team had been working furiously through the night, hoping to improve upon our previous day’s score of .8554. It was hard to believe that just twenty-four hours ago we had passed the four-team coalition that had occupied the first place spot for the last 29 days. There was little time to celebrate; the previous leaders would not go down without a fight, so we had to be ready with something better. A call had been issued for any remaining valid predictors, anything that could tip the scale in this final day, and our members around the globe had answered the call: nearly 200 new predictor sets, some previously passed over for their poor performance and others newly conceived only moments prior, had flooded in from all corners of the team. It was now up to our blenders to work some last-minute magic.
me: gabor -- here's my new best i don't know how well it will do but it's possible it will give .8553
Gabor: what does the blender say?
me: 0.855345926622 so, cutting it close
Our submission window opened just 9 minutes before the contest closed. Gabor, our captain, had asked me to give him a countdown to our submission time via Google Chat.
me: 5 minutes
There was nothing left to do but wait.
me: 4 minutes bpc tied us booo
Indeed, the previous leaders had issued their final submission and risen to tie our score. I decided to check the Ensemble mailing list to gauge our members’ reactions. Some expressed delight, others, alarm. David W. repeatedly expressed confusion about why we hadn’t submitted, as only less than 1 minute remained in the contest. (His clock was 13 minutes fast.)
me: 3 minutes
me: 2 minutes
me: wait peng says he has .8552 !!
That’s right: one minute before our scheduled submission time, I spied a post by Peng revealing his latest blending effort. With an estimated score of 0.85525, it soundly trumped our intended submission. I logged onto the Vandelay Industries server to retrieve the predictions file but didn’t know which of the several hundred files was the right one to download.
me: Peng -- what file is it quick!
No response. Peng, you see, unbeknownst to me or anyone else had lost his internet connection and had texted the first post by Blackberry.
me: WHAT FILE IS IT???
Jacob: This is the best RMSE we’ve seen so far, right? I vote we swing for the fences and submit this one!
Gabor: 1 minute and i have to submit!!!!!
me: WE DON'[T KNOW WHICH FILE--HURRRRRRRRY
Ces: SHouldn't we submit NOW?
You can imagine my frustration.
It was now two minutes after our intended submit time and seven minutes until the contest closed. I searched frantically through the folder for files with .85525 in the name but couldn’t seem to find any.
me: i don;'t know!! i can't find it!
Gabor: ok i submit yours
And then, I saw it.
me: wait i found it opera_est_0.855259.txt.gz
Gabor: i'm uploading
me: ok, awesome
Gabor: i'm afraid it was late
me: uh oh it's not 11:42 yet
Gabor: but uploading is slow
me: 4 minutes to spare it registered
Gabor: ok submission successful :)
Yes, submission successul.
me: That's what this team's all about -- living on the edge :)
The contest is now officially over. Bellkor's Pragmatic Chaos and The Ensemble finished within .01% of each other. What great work on both sides! It was a very dramatic march for The Ensemble these last 30 days, and there was an even more nail-biting conclusion as we raced to get the best submission in place.We salute Netflix, BellKor's Pragmatic Chaos, and everyone who participated in the competition!
We are happy today to have made a submission which surpasses Netflix's 10% Grand Prize target. The Ensemble is the second team to do this in less than a month. We are very proud of our achievements, and those of our top competitor, Bellkor's Pragmatic Chaos.
The Ensemble was created by a merger between the two teams that appear on the leaderboard as "Grand Prize Team" and "Opera Solutions and Vandelay United".
The crowd is indeed wiser than the individual.
The 10% barrier once seemed distant and insurmountable. But when the contest's "last call" heralded the heroic achievements of Bellkor's Pragmatic Theory, the rest of the crowd pondered, and asked why the barrier couldn't be broken twice.
And lo, as if powered by gravity, Grand Prize Team and Vandelay Industries ! began to draw in more and more members. And Vandelay went on to join forces with Opera Solutions, and then Vandelay and Opera united with Grand Prize Team, and then ... and then ... well, things got so complex we decided just to call ourselves The Ensemble.
Chris Hefele's dramatic visualizations of the relationships between movies in the Netflix database are featured in this article from the August issue of Communications of the ACM.
For much more, check out our visualization page, where there's a large "map" of movie similarities that you can explore interactively, as well as many newer, larger high-resolution images you can download.
For the most up-to-date images for publication, please contact us.
The members of The Ensemble want to thank Riley Cordova ( http://rileycordova.com ) for the great job that he did designing this website.
We also want to recognize the contributions of Clive Gifford ( http://clive-gifford.blogspot.com ). Clive did everything that he could to help the team.
Very special thanks to Susie Murphy for writing, working on the website, coordinating interviews, and everything else that she has done.
And of course, thanks to Netflix for running such a great contest!