So this is the last lecture in this course. What have we learned up until this point? We've learned what supervised machine learning is for text classification. We've learned the considerations, we've learned the evaluation metrics. We've learned how to do supervised machine learning using K train which is a wrapper of tensor flow. And we've used a very simple elastic net model that uses linear regression and a bag of words approach. They're both pretty similar. The K train model tends to be slightly better in terms of performance. The elastic net model is much quicker in terms of speed and being able to classify and it was easier to train. It took just seconds versus an hour or so. So there's real tradeoffs here. Whatever we do is probably up to us. I mean at this point I think it's really important that we compare our metrics dollar for dollar, doughnut for doughnut. If we do this, we can be sure that we're comparing models in the same way, most performance metrics that you get from K train or that you get from elastic net are going to be accurate. But if you really want to make a final decision on which one is better, it's best to take the test predictions and the test actuals and calculate performance metrics outside of those packages using an independent metrics tool. So that we feel really good that we've compared the two side by side and to make sure that they are the same. So I'm going to use SK learns metrics package here to wrap up this class because it gives us all these basic metrics that we want. All we need to do is give it our predictions and our actuals for a data set and it's going to calculate all of those metrics that we need for us automatically. So if you'll remember we went ahead and saved all of our predictions and actuals for our test set in a folder in our google drive under our evaluation folder which was just slash evaluation. We did that so that I could easily then load them back in to perform some basic metrics. So the SGDC, was technically the second best performing algorithm. But we'll go ahead and look at the performance metrics for it. So if we load it in we really all we need to do is find the files. So I went through the evaluation folder and I'm using a little tool called glob. Glob is great because it will take a folder and search for a pattern. And here I'm saying anything can start before it and anything can go after it. But if in the middle of the string for the foul it has the term SGDC I want it to be returned in a list. So if I return this list or if I print this list I get out the two pickle files that correspond with the test set answers and what we predicted for that test set. So I can load those in very simply using pickles and I have to convert this one to an array because this one for whatever reason was technically a list. So we need to make sure it's an array. What we need to do metrics in SK learn is simply to arrays one with the test actuals, one with the test predictions and that's it. And so we have those were loading them in and now all we need to do to get an accuracy score it's just called SK learn metrics. The accuracy score, give it those two variables and we get the score, if we want to know the mean or the percentage of ones in the test set, we can see that by doing that. If we want the mean for the actual predictions, we can do this and you can see they're pretty darn close. I love that these percentages are both close to 50. That means that it's generally understanding the distribution of positives and negatives. So that's good. So this is a confusion matrix. And we really have not talked much yet about confusion matrices, but really what they tell you are the true positives, which is the number of documents we said we're healthy living that were actually healthy living. The number of true negatives, which is the number of articles that we said, we weren't healthy living, that we're actually not healthy living. The number of false positives, those are times where we labeled as one, but the reality was zero and false negatives, which is the reality of that. It was a one and we said it was zero. You can pull these things out of the confusion matrix by doing this and that's really pretty easy to do and you can inspect them. But when you see your confusion matrix, you may get confused whether this is a false positive or false negative or vice versa. But in reality what you're looking for in this is you're looking for balance in the era, you want to see the false positives and the false negatives. You want to see these two values be approximately the same. You don't want to miss classify documents more one way than another. If you're doing that then you may have a bias in your class. We're seeing a pretty close to even distribution here and of course the majority of documents are being classified correctly. So this confusion matrix looks okay to me. I've given you the ability to compute some additional metrics like accuracy, classification error. We did not cover sensitivity and specificity but if you need them for specific problems, all of these metrics are available for you in SK load. Finally we did precision here which is exactly what we would have expected. It's a little bit often, I wonder if that's due to random kind of rounding error or something. And then we can see here though that are precision and then our recall are pretty close to each other. So that's the beauty of SK learn is that you can calculate all of these metrics independently outside of the machine learning tools that you're using. And when you're using multiple approaches, I highly encourage you to use these metrics outside of whatever package you're using because you never know how exactly it's calculated. And you just want to be able to calculate things dollars for dollars, donuts for donuts across models. And if you save your predictions and your actuals for your test set, it's very easy to load it in to SK learn and actually get these results out so that you can report them and verify that the model you think is better is actually better. So that's it, in about five hours, we've covered all there is to know, I'm just kidding about supervised machine, there's a lot more out there that you can learn and do. But this is a great start for you to have a tool that you can use for classification. Now, we used a contextual advertising solution here, but you feel free to do whatever you want with supervised machine learning. It is a tool that can be used for automation purposes and it really can do a lot. Remember, this isn't a method that's just limited to text. You can classify photos, you can classify videos, anything that you imagine really can be classified in this way, as long as you have a training set and there's a way to transform the model into data that can be trained on. So it's a wonderful tool and I hope that you found this first class useful. Remember now is the time for you to go in there and finish that project, remember the notebooks that I've given you are what you need to get the projects done. Follow the guidelines on the project closely and you'll do well, I look forward to seeing you again in class to where we are going to do unsupervised machine learning, which is the exact opposite. We don't know the answers, we don't have a training set, we're ready to go forward with figuring out what those things are. We're going to explore datasets using unsupervised machine learning in our next course.
