Because if doing well on the test set according to some cost function
doesn't correspond to your algorithm doing what you need it to do in the real world,
then it means that either your dev test set distribution isn't set correctly,
or your cost function isn't measuring the right thing.
I know I'm going over these examples quite quickly, but we'll go much more
into detail on these specific knobs later this week and next week.
So if you aren't following all the details right now, don't worry about it.
But I want to give you a sense of this orthogonalization process,
that you want to be very clear about which of these maybe four issues,
the different things you could tune, are trying to address.
And when I train a neural network, I tend not to use early stopping.
It's not a bad technique, quite a lot of people do it.
But I personally find early stopping difficult to think about.
Because this is an op that simultaneously affects how well you fit the training set,
because if you stop early, you fit the training set less well.
It also simultaneously is often done to improve your dev set performance.
So this is one knob that is less orthogonalized,
because it simultaneously affects two things.
It's like a knob that simultaneously affects both the width and
the height of your TV image.
And it doesn't mean that it's bad, not to use, you can use it if you want.
But when you have more orthogonalized controls,
such as these other ones that I'm writing down here,
then it just makes the process of tuning your network much easier.
So I hope that gives you a sense of what orthogonalization means.
Just like when you look at the TV image, it's nice if you can say, my TV image
is too wide, so I'm going to tune this knob, or it's too tall, so I'm going to
tune that knob, or it's too trapezoidal, so I'm going to have to tune that knob.
In machine learning, it's nice if you can look at your system and
say, this piece of it is wrong.
It does not do well on the training set, it does not do well on the dev set,
it does not do well on the test set, or it's doing well on the test set but
just not in the real world.
But figure out exactly what's wrong, and then have exactly one knob, or
a specific set of knobs that helps to just solve that problem
that is limiting the performance of machine learning system.
So what we're going to do this week and next week is go through how to diagnose
what exactly is the bottleneck to your system's performance.
As well as identify the specific set of knobs you could use to tune your system to
improve that aspect of its performance.
So let's start going more into the details of this process.