Of the bias between your predicted values, and your true values.

But it'll be highly variable.

In other words, it'll depend a lot on which random subsets that you take.

For smaller ks, we won't necessarily get as good

an estimate of the out of sample error rate.

And that's because, you're only leaving one sample out.

And so you're using most of your data to train your model.

But there'll be less variance.

In other words, if you do in the extreme case.

If you have for exampleonly two cross, two-fold

cross validation, there are only a very small.

Number of subsets that can make up a two-fold cross validation.

And so you get less variance.

Here, the randomless sampling must be done without replacement.

In other words, we're subsampling our data sets.

That's, of course a disadvantage, because, it means that we

have to break our training setup even further to smaller samples.

If you do random sampling with replacement, this is called the Bootstrap.

That's something that you've learned about in your early.

Inference classes, if you've taken those.

The bootstrap, in this particular example,

will in general, underestimate the error rate.

And the reason why is because if you do the bootstraps, you sample

with replacement from some of your samples,

some samples will appear more than once.

And so, in more, samples appear more than once, that means

that if you get one right, you'll definitely get the other right.

And so, you actually get underestimates of the

error rate, and this can be corrected but

its rather complicated, the way to do that,

is with something called the 0.632 Bootstap, which

is not exactly a great name for a method, but it explains, it sort of explains

how you can account for the fact that you have this underestimate in the error rate.

You can do any of these approaches when you go models

with the care package like we'll be learning in this class.

If you cross validate to pick predictors, you again must estimate the errors on

an independent data set, in order to get a true out of sample value.

So in other words, if you do cross validation to predict

your model, or to pick your model, the cross validated error rates.

Since you always picked the best model, will not necessarily be a

good representation of what the real out of sample error rate is.

And the best way to do that, is again, by applying

your prediction function, just one time, to an independent test set.

[SOUND]