7.2 Evaluate Models with Responsible AI Principles

Evaluate Models with Responsible AI Principles

Quick Answer: The Responsible AI (RAI) dashboard in Azure Machine Learning studio evaluates tabular classification and regression models that are registered as MLflow sklearn-flavor assets. Components cover data analysis, model overview and fairness (Fairlearn), error analysis, explanations (InterpretML, SHAP-style feature importance), counterfactuals (DiCE), and causal analysis (EconML). Generate it with a pipeline: constructor → tool components → gather, plus an optional PDF scorecard. RAI is evaluation of classic ML models. It is not Microsoft Foundry content safety for generative apps (Domain 4).

Exam AI-300 Domain 2 asks you to evaluate a model by using responsible AI principles. After you register the MLflow asset in section 7.1, you still have not shown that the model is fair, interpretable, or reliable on the cohorts that matter. Azure Machine Learning packages those checks in one dashboard attached to the registered model.

Why a single dashboard

Responsible AI in production is an engineering loop, not a slogan. Microsoft groups the work into identify (where does the model fail or treat groups differently?), diagnose (why?), and mitigate (what do you change in data, features, or the model?). Mitigation algorithms themselves live in standalone libraries such as Fairlearn; the dashboard's job is the identify and diagnose stages, plus decision-support (counterfactuals and causal effects).

Use the dashboard when you need to:

• Measure group fairness (difference or ratio of a metric across sensitive features).
• Find error-concentrated cohorts with the error tree or heat map.
• Produce global and local explanations (SHAP-style feature importance, ICE plots).
• Answer what-if and treatment questions with counterfactuals and causal analysis.
• Export a PDF scorecard for product, risk, and compliance readers.

The dashboard attaches to a registered model. Open Models, select the asset, then the Responsible AI tab. You can attach multiple dashboards with different component mixes. Studio's no-code flow (Create Responsible AI dashboard, currently preview) and SDK/CLI v2 both submit a pipeline job that writes the insights back onto that model.

Supported models and hard limits

The current RAI components are narrow. Memorize the limits; they are frequent distractors.

You must supply a model even for causal analysis of the data. A scikit-learn DummyClassifier or DummyRegressor is the documented stand-in when you have no real predictor. Multi-class classification does not support the studio real-life interventions (causal) profile.

Components you will be tested on

Fairness is not a separate Azure resource. On the Feature cohorts page you pick sensitive (and other) features and a metric. Classification metrics include accuracy, precision, recall, F1, false positive rate, false negative rate, and selection rate. Regression metrics include mean absolute error, mean squared error, R-squared, and mean prediction. Disparity is then max − min (difference) or max / min (ratio) across subgroups. The PDF scorecard uses the same idea: you set a target such as difference in accuracy <= 0.05.

Error analysis trains a surrogate tree on errors. Nodes show error coverage (share of all errors sitting in that node) and error rate. The heat map crosses up to two features. You can save a node as a cohort and reuse that cohort in every other pane.

Explanations split aggregate (which features drive the model overall) from individual (up to five rows, plus an ICE plot if a compute instance is connected). Counterfactuals need a desired class (classification; binary often uses opposite) or a desired range (regression). Generate at least 10 diverse counterfactuals per row if you want the bar chart of which features were perturbed most often. Causal analysis is a different question: not "what did the model use?" but "if we changed this treatment in the real world, what happens to the outcome?" Do not report SHAP importance as a causal effect.

Some panes need live compute: dynamic error-tree retraining, ICE plots, generating a new what-if counterfactual, and causal what-if on a single row. Connect a running compute instance at the top of the dashboard; the service starts a RAI endpoint in a terminal on that instance. Without compute, you can still read precomputed views.

How you generate the dashboard

The pipeline shape is fixed:

1. RAI Insights dashboard constructor — inputs are the MLflow model, train mltable, test mltable, task_type (classification / regression / forecasting), and target_column_name. Optional: categorical column names as a JSON string, class labels, maximum_rows_for_test_dataset (default 5,000).
2. One or more tool components: Add Explanation, Add Error Analysis, Add Counterfactuals, Add Causal. Each takes the constructor output.
3. Gather RAI Insights dashboard — constructor port plus up to four insight_n ports. At least one insight is required.
4. Optional Gather RAI Insights scorecard / rai_score_card — PDF for product, risk, and compliance readers. You pass a JSON config of metric thresholds, sensitive features, and fairness_evaluation_kind of difference or ratio.

Components are published in the azureml registry (names such as microsoft_azureml_rai_tabular_insight_constructor). Studio's no-code wizard is the same graph: pick train/test MLTable datasets, modeling task, then either the Model debugging profile (error analysis + counterfactuals + explanations) or Real-life interventions (causal). You still need a registered MLflow sklearn model before the button appears.

RAI versus generative safety

This Domain 2 skill is tabular model evaluation in Azure Machine Learning. Domain 4 covers risk and safety evaluations for generative systems in Microsoft Foundry (harmful content, jailbreak, groundedness). A stem that says "the chat endpoint is producing unsafe completions" is not asking you to open the RAI dashboard on an sklearn loan model, and a stem about a credit-risk classifier is not asking you to attach a Foundry content-safety evaluator. Use both in a real estate, but the exam maps them to different domains.

Exam scenario

A bank registers loan-approver version 4 as an MLflow sklearn pipeline (one-hot encoder plus gradient boosting, predict_proba exposed). MLOps submits a RAI pipeline: constructor on the train/test MLTables (downsampled to 4,000 rows, gender and age_band marked categorical), then explanation, error analysis, counterfactuals (total_CFs: 10, desired class opposite), and gather. The dashboard's fairness view shows a large difference in false negative rate across gender. Error analysis concentrates misses on young applicants with thin credit files. Aggregate explanations rank zip_code high — a likely proxy. Counterfactuals show that a modest income increase plus one fewer open revolving account would flip several denials. The team exports a PDF scorecard with accuracy >= 0.85 and difference in false negative rate <= 0.05 for the risk committee. They do not point this dashboard at the bank's Foundry GPT assistant; that assistant is evaluated with Domain 4 safety metrics.

Common trap

Pointing the RAI dashboard at a PyTorch vision model, a custom pickle, or an AutoML studio model and expecting the constructor to run. Sibling traps: skipping the 5,000-row downsample; passing NumPy arrays instead of mltable Parquet; treating SHAP importance as causality; using RAI as content safety for a generative endpoint; omitting categorical feature names so fairness bins collapse; generating fewer than 10 counterfactuals and wondering why the perturbation bar chart is missing; forgetting that causal still needs a registered model (even a dummy estimator).