Cross Validation and Confusion Matrix

Loading...

来自 University of Colorado Boulder 的课程

Advanced Business Analytics Capstone

12 个评分

University of Colorado Boulder

Advanced Business Analytics Capstone

12 个评分

课程 5（共 5 门，Specialization Advanced Business Analytics）

The analytics process is a collection of interrelated activities that lead to better decisions and to a higher business performance. The capstone of this specialization is designed with the goal of allowing you to experience this process. The capstone project will take you from data to analysis and models, and ultimately to presentation of insights. In this capstone project, you will analyze the data on financial loans to help with the investment decisions of an investment company. You will go through all typical steps of a data analytics project, including data understanding and cleanup, data analysis, and presentation of analytical results. For the first week, the goal is to understand the data and prepare the data for analysis. As we discussed in this specialization, data preprocessing and cleanup is often the first step in data analytics projects. Needless to say, this step is crucial for the success of this project. In the second week, you will perform some predictive analytics tasks, including classifying loans and predicting losses from defaulted loans. You will try a variety of tools and techniques this week, as the predictive accuracy of different tools can vary quite a bit. It is rarely the case that the default model produced by ASP is the best model possible. Therefore, it is important for you to tune the different models in order to improve the performance. Beginning in the third week, we turn our attention to prescriptive analytics, where you will provide some concrete suggestions on how to allocate investment funds using analytics tools, including clustering and simulation based optimization. You will see that allocating funds wisely is crucial for the financial return of the investment portfolio. In the last week, you are expected to present your analytics results to your clients. Since you will obtain many results in your project, it is important for you to judiciously choose what to include in your presentation. You are also expected to follow the principles we covered in the courses in preparing your presentation.

从本节课中

Module 2 - Perform predictive analytics tasks

This week you will perform some predictive analytics tasks, including classifying loans and predicting losses from defaulted loans. You will try a variety of tools and techniques this week, as the predictive accuracy of different tools can vary quite a bit. It is rarely the case that the default model produced by ASP is the best model possible. Therefore, it is important for you to tune the different models in order to improve the performance.This week’s assignments require you to build predictive models for both classification and regression tasks. <p> Before working on the assignments, you may review a few videos to remind yourself several important concepts, such as cross validation. These concepts are discussed in the videos Cross Validation and Confusion Matrix and Assessing Predictive Accuracy Using Cross-Validation. You may also find a refresher on XLMiner useful. The videos Building Logistic Regression Models using XLMiner and How to Build a Model using XLMiner discuss how to build logistic regression and linear regression models. Depending on your needs, you may also go back to the videos that discuss how to build trees and neural networks. </p>

Cross Validation and Confusion Matrix5:03

Assessing Predictive Accuracy Using Cross-Validation5:18

Building Logistic Regression Models using XLMiner6:45

How to Build a Model using XLMiner8:02

与讲师见面

Manuel Laguna
Professor
Leeds School of Business
Dan Zhang
Professor
Leeds School of Business
David Torgerson
Instructor

In this video, we discuss how to perform cross-validation for classification problems.

In particular, we use a two by two table called a confusion matrix

to assess classification performance.

As for a regression problem, the first step of cross-validation is data partitioning,

where we randomly split the entire dataset by row into two sets.

The training set is used to fit the models; other set is called the validation set,

and is used to choose among different models.

We use different measures to assess prediction accuracy for regression and classification.

For regression, we want error or residual to be small in the validation set.

For that purpose, we use sum of squared errors or, equivalently,

the root mean square on validation data.

For the classification, those measures cannot be used.

Instead, we look at a confusion matrix, which we will explain using example.

Let's go back to the appointment data.

We will randomly divide the data by row into training

and validation sets using a 60/40 split.

4,478 rows are in the training set, and the 2,985 rows are in the validation set.

Now, to assess the prediction accuracy of our model,

we make a prediction of whether each appointment will be canceled in the validation data.

Note that the outcome of the logistic regression model is a predicted probability.

How do we make a binary prediction using predictive probabilities?

One way is to set a threshold value t between zero and one.

When the predicted probability of a cancellation is above this threshold,

then we predict cancellation.

Otherwise, we predict no cancellation.

In this manner, we can arrive at a binary prediction from the predicted probabilities.

This immediately leads to the question: What value should we pick for t?

The threshold value is an important parameter we have to choose.

Choosing different values of t leads

to changes in different encountered prediction errors.

If we choose a large t, then we rarely predict cancellation.

Therefore, we will only detect the cancellation when the chance is very high.

This will lead to more errors when we predict no cancellation,

but the appointment is actually canceled.

We choose a small t, we will predict lots of cancellations.

This allows us to detect more appointments that may be canceled.

However, we will make more mistakes where we predict cancellation,

but the appointment is actually not canceled.

By default, we usually choose the threshold value of 0.5,

where we predict the more likely outcome.

Here is a confusion matrices for two different threshold values.

Note that cancellation is denoted by one.

The matrix shows the observed or actual class and predicted class.

Actual=0 means the observed status in the data is arrival.

And actually, you could do one means the observes status is cancellation.

Similarly, Predicted=0 means we predict the appointment will not be canceled,

and Predicted=1 means that the appointment is predicted to cancel.

When the threshold is 0.5,

2,303 appointments that were not canceled match our prediction.

However, 17 appointments that were not to canceled are predicted to cancel.

649 appointments that were canceled are predicted not to cancel,

where 16 canceled appointments are correctly predicted.

Note that very few appointments are predicted to cancel even though

there are more than 600 canceled appointments in our validation data.

The confusion matrix changes considerably when we set the threshold to 0.3.

In particular, a lot more appointments are predicted to cancel.

All of the appointments that were not canceled, 216, were predicted to cancel.

122 canceled appointment were also correctly predicted.

Therefore, with a smaller threshold, we'd correctly predict more canceled appointments,

but also makes more mistakes of predicting non-canceled appointments to cancel.

This shows the tradeoff in picking the threshold.

The confusion matrix essentially shows the possible outcomes

when we make binary predictions on the validation data.

We would like to detect the class with value one.

In the example we discussed, we would like to detect the appointment cancellations.

In this case, we usually call the event that we would like to detect positives.

The complement event, in this case non-cancellations or arrival, is called negatives.

A correct prediction can be either a true positive or true negative.

A canceled appointment that is correctly predicted to cancel is called a true positive.

Similarly, a non-canceled appointment

that is correctly predicted not to cancel is called a true negative.

An incorrect prediction can be a false positive or a false negative.

A canceled appointment that is incorrectly predicted

not to cancel is called a false negative.

Similarly, a non-canceled appointment that is incorrectly predicted

to cancel is called a false positive.

This terminology has their roots in the medical literature

but are commonly used in data analytics.

探索我们的目录

免费加入并获得个性化推荐、更新和优惠。

Coursera 致力于普及全世界最好的教育，它与全球一流大学和机构合作提供在线课程。

© 2018 Coursera Inc. 保留所有权利。

通过 App Store 下载

通过 Google Play 获取