This video examines the feature of Watson studio that helps to ensure fairness and explain-ability of machine learning pipelines, as well
as monitored their performance after deployment. IBM Watson Openscale is a product that includes several important features. It can test the model and its predictions for fairness and apply ways to overcome bias. It can also help to provide explanations for model predictions that are often hard to get but are necessary for compliance in some application areas. It monitors the model performance and can detect its deterioration or model drift over time. It can alert the users when drift is detected and explain which predictors are causing it. We can specify criteria under which the model gets automatically retrained on fresh data; it also helps to measure how the model helps the business. The attributes to monitor for bias are automatically recommended based on prior experience. They can be edited as needed. Openscale then keeps track of model predictions for the specified groups and checks the bias in the predictions. Users need to know that their AI models are fair but the date of their models were trained on and include unwanted biases\a which may unintentionally be included in the resulting models. IBM Watson Openscale can detect bias when a model is in production and not just when it's being built. In this demo of Watson Openscale we'll monitor a credit risk model which has been trained to determine whether or not someone is eligible for a loan, based on a variety of different features, such as their credit history age and their number of dependents. After launching Openscale we can see a few highlighted metrics for the monitored model, such as its quality and a fairness score. What Openscale does is measure a model's fairness by calculating the difference between the rates at which different groups, for example, women versus men, received the same outcome. A fairness value below 100% means that the monitored group receives an unfavorable outcome more often than the reference group. In this case, we see that women are receiving the no-risk outcome, or getting approved for loans, at a lower rate than men. Openscale enables the inspection of each model's training data and this reveals that there was more training data for men than women. This can give some insight as to why the model exhibits bias against women who apply for loans. Data scientists can use this information to approve the model. Now, detecting bias is one thing-- Openscale can also mitigate it by creating a D bias model that runs alongside the monitored one. In this case the D bias model is 12% more fair than the production model. The D bias model has been trained to detect when your production model will make a bias prediction so that you can isolate the specific transactions that result in the bias. For each of these transactions Watson Openscale will flip the monitored value in a record to the reference value, in this case from female to male, and leave all other data points in that record the same. If this changes the prediction from risk to no-risk then the D biased model will surface the no-risk outcome as the D biased result. This is just one of the ways that Watson open scale helps you ensure that your models are fair explainable and compliant wherever your model was built or is running. Insurance underwriters can use machine learning and Openscale to more consistently and accurately assess claims risk, ensure fair outcomes for customers, and explain AI recommendations for regulatory and business intelligence purposes. Why does an AI model arrive at a given recommendation or prediction? Users and customers want an explanation and with most models providing this information is not an easy task. IBM Watson Openscale explains predictions in business friendly language. This credit application, for instance, was predicted to be a risk. Openscale determines the features which contributed positively or negatively to that prediction and spells them out. The explanation is presented visually, as well as in a sentence-based text summary in order to ensure maximum clarity. Using proprietary IBM research technology, Openscale also generates a contrast of explanations. Here we see the minimum changes to this input record which would produce a different output, changing the prediction from risk to no-risk. The explanations provided by Watson Openscale can help organizations comply with regulations such as the Fair Credit Reporting Act and GDPR which give customers the right to ask for reasons why their applications were denied. Before an AI model is put into production it must prove it can make accurate predictions on test data, a subset of its training data; however, over time, production data can begin to look different than training data, causing the model to start making less accurate predictions. This is called drift. IBM Watson Openscale monitors a model's accuracy on production data and compares it to accuracy on its training data. When a difference in accuracy exceeds a chosen threshold Openscale generates an alert. Watson Openscale reveals which transactions caused drift and identifies the top transaction features responsible. For instance, 25% of a transactions causing drift in this loan approval model were problematic because of these features, which contained data crucially different from the training data. The transactions causing drift can be sent for manual labeling and use to retrain the model so that its predictive accuracy does not drop at run time. Watson Openscale not only helps identify drift but also highlights its root cause and provides transactions which can be turned into training data useful at fixing drift. It gives you the insight you need to ensure that your models will consistently deliver the results you want over time. For instance, the retrain version of the model, but based on the recommendations made by Watson Openscale, started making accurate recommendations alleviating the drift. This is just one of the ways that Watson Openscale helps you ensure your models are fair explainable and compliant wherever your model was built or is running. In this video you have learned how Openscale ensures fairness and explain ability of models and monitors for model drift in production. This completes the model on IBM products for data scientists. Good luck on the quizzes!
