So another major component was understanding the data,
what we call the global effects, residing in the data.
Global effects are like the average score that the movie receives.
Some movies are just better than others.
And you can say with confidence that the good movie will get the high rating
without knowing anything about the writer.
[CROSSTALK] >> Some people just like movies more than
others.
>> Yes, exactly.
That's the other side of the coin.
And some people also are less critical than others and
tend to give higher ratings.
So if it was unknowing nothing about the movie, just say for
this writer, then anything is going to be high.
So this is just that, I call it data cleaning because you say,
okay, let's remove these effects, and try now in a model of a cleaner data.
So this was the big part of what we did in the beginning.
And by the way, later on, we stop doing this,
because it was better to have these effects baked in into the model.
So the model is part of learning the Megawood model or
the methods factorization model also land most of these passes.
But in the beginning, we didn't realize this and we were cleaning the data.
And cleaning the data is important, in general, I always advise it.
>> Great. >> So these were small steps.
>> Let's jump ahead for a minute.
>> Yeah. >> Because we could take you through all
the steps, but over time, a bunch of different teams that were
competing seemed to come to the same realization.
That the techniques they were using were making progress but
they weren't going to get them to this magic 10%.
And they seemed to figure out that the key to solving this
challenge was to find a way to merge together the different algorithms that
each had different strengths and weaknesses in to some composite algorithm.
How did that insight come about and
what was sort of the general high level approach that pulled everything together?
>> Yes, I said this is probably the one thing that people decides matrix
factorization.
I think, yes, people remember phonetics competition,
the insane number of predictors that we have used which is a shame.
[LAUGH] No project assistant should use so
many predictors, and so this happened gradually.