The area under the receiver operating characteristic or ROC curve, also called the AUC, is the most widely used performance metric for binary classification models. Its two great strengths are, first, that AUC results do not change with changes in the incidents of the actual condition, nor is AUC affected by changes in the relative cost of the two different types of binary classification errors, false positives and false negatives. Therefore, when either future incidents or the cost of classification errors or both are unstable or cannot be known, the AUC is generally the best possible performance metric available. The AUC metric ranges from a minimum of one-half to a maximum of one. An AUC of one-half indicates total uncertainty about classification. An AUC of one-half is a result when guessing outcome are random. An AUC of one would be classification with zero errors, a theoretical ideal that no one expects you to meet. A normal good to very good area under the curve is typically in the .65 to .85 range. A common technique in data analysis is to develop a binary classification model by varying certain parameters so as to maximize the AUC on a sample set of data with known outcomes, often called a training set. You will calculate your own area under the curve on a training set and apply this technique for the final course project. This video is a quick tutorial on how to calculate for a given classification model and collection of events with known outcomes, the resulting area under the curve. You should follow along using the review of AUC curve Excel spreadsheet, which I will now turn to. To make an ROC curve, we start with a collection of events. In the example spreadsheet, there are seven events numbered one through seven. In column B, rows 20 through 26. Each event must be assigned a score and an outcome if we are to generate an ROC curve. Our model is a procedure or algorithm that inputs known data about an event and outputs a single score. Here we focus on turning scores, and outcomes, into an ROC curve, so we aren't concerned with how the scores are generated. Our binary outcomes are represented by one, for positive, and zero, for negative. To generate an ROC curve, we need to know what the true outcomes are. And then we need to know how effective our model is at each possible threshold for positive classification. The way that we find this out is that we rank all of our events by score. And you'll notice this has already been done for you, but this procedure only works if you have a strict rank ordering by score of all of your events. And then you're going to proceed down your list, performing classification at each distinct possible threshold. The threshold that we're using, we will call the active threshold. The first threshold that we use is actually a number larger than or higher than the top score. At this threshold, everything is classified as negative. Therefore, there are no false positives, because nothing is positive. And there are no true positives, because nothing is classified positive. Okay, we're going to proceed through our list, and at the very bottom threshold, at a threshold of greater than or equal to -3, every event is classified as positive, and so we will have a total number of true positives, here three, equal to the actual number of positives in our collection. But all of the negative outcomes are classified incorrectly as false positives. What an ROC curve is is a measure of the effectiveness of the model at each possible threshold. In this case, there are eight thresholds from above the highest score to below or equal to the lowest score. We need to convert our total number of false positives and total number of true positives into a false positive rate and a true positive rate. To generate the false positive rate, we're going to divide the number of false positive's at the each particular threshold by the total true number of negatives or four. So let's just pick one particular example. Let's just pick a threshold. That is greater than or equal to one. There are three outcomes, two positive outcomes and a negative outcome. The negative outcome is incorrectly classified as positive, therefore there is one false positive, and the two actually positive outcomes are correctly classified, therefore there are two true positives. The false positive rate is the number of false positives, here one, divided by four, so one divided by four is .25. And the true positive rate is a number of true positive, here two, divided by the total number of positive outcomes, which is three, or two-thirds. This ordered pair one-quarter two-thirds is represented on our ROC curve by the point, .25, .67, here. And, what we're doing when we work our way down from the top, down to the bottom, is we're starting with the point where everything is negative, and we have a false positive rate of 0 and a true positive rate of 0. Then we are proceeding up and to the right, up and to the right. As we work our way down this list, until we arrive at the point 1,1 where everything is classified positive. So we have three over three equals one and four over four equals one. We next will sum the area under these rectangles. We have four rectangles 1, 2, 3, 4. Notice that only the x-axis value or the y-axis value can change between any two points. We never have both changing at the same time, because we are using narrow changes in threshold. So, there only one outcome at a time can be switch from a negative to positive, positive classification. So, only the false positive rate or true positive rate can changes outlooks. So, we sum the area of these rectangles to get an idea of the overall strength of our classification model. So we're going to take the width of each times its height, so the first rectangle has a width of .25 minus 0 and has a height of .67. The second rectangle has a width of .5 minus .25 and it has a height of one. Third rectangle has a width of .75 minus .5, and it has a width of one, and the last rectangle has a width of 1 minus .75, and it has a height of one. And these areas are equal to 0.917, which appears here. And that is how you calculate the area under the ROC curve. I'd like to point out that each threshold has its own confusion matrix, and these confusion matrixes are provided in your spreadsheet. So we have outcomes that do not change. Where we have three true positive outcomes and four actually negative outcomes. But our classifications are shifting. So each time we move down our list, we are shifting one item, Over two from a negative to a positive classification. So next we're going to take the item two, and we're shift it from a negative to a positive classification. And you see that one of our items here will change. Only one can change at a time. Okay? And then we look at the false positive rate and true positive rate. And that is what gives us our points on our ROC Curve. I hope this has been instructive. Good luck in generating your own ROC Curves. I think you'll find that this is one of the most useful tools provided in this course.