Trust Metrics and Validation

Measuring Explanation Quality

An explanation is only valuable if it's trustworthy. But how do we measure the quality of an explanation? The WIA-AI-009 standard defines comprehensive trust metrics that quantify explanation reliability, consistency, and usefulness.

Core Trust Metrics

1. Fidelity

Fidelity measures how accurately the explanation represents the model's true behavior. A high-fidelity explanation means the explanation method correctly captures what the model is doing.

// Fidelity computation
function computeFidelity(explanation, model, testInstances) {
  let totalError = 0;

  for (const instance of testInstances) {
    const modelPrediction = model.predict(instance);
    const explanationPrediction = explanation.predictFromExplanation(instance);
    totalError += Math.abs(modelPrediction - explanationPrediction);
  }

  return 1 - (totalError / testInstances.length);
}

// Fidelity > 0.85 is considered good
// Fidelity > 0.95 is excellent

2. Consistency

Similar inputs should produce similar explanations. Consistency measures whether the explanation method behaves predictably across the input space.

WIA-AI-009 defines two consistency variants:

• Lipschitz consistency: Small input changes → small explanation changes
• Semantic consistency: Semantically similar inputs → similar explanations

3. Stability

For stochastic explanation methods (LIME, KernelSHAP), stability measures how much explanations vary across multiple runs with the same input. High stability means the randomness is well-controlled.

function computeStability(explainer, instance, numRuns = 50) {
  const explanations = [];

  for (let i = 0; i < numRuns; i++) {
    explanations.push(explainer.explain(instance));
  }

  // Compute variance across explanations
  return 1 - computeVariance(explanations);
}

// Stability > 0.80 recommended for production

4. Comprehensibility

Can humans actually understand the explanation? Comprehensibility is harder to quantify but crucial for practical utility.

Proxy metrics include:

• Explanation complexity: Number of features/rules presented (fewer is often better)
• Cognitive load: Estimated mental effort required (measured through user studies)
• Actionability: Can users act on the explanation to change outcomes?

5. Completeness

Do explanations account for the entire prediction? For additive methods like SHAP, this is exact: sum of attributions = prediction - baseline. For other methods, completeness is approximate.

Validating Explanations

Ablation Testing

Remove the features the explanation identifies as important. The prediction should change significantly. If it doesn't, the explanation is suspect.

Perturbation Analysis

Modify features in the direction the explanation suggests. If the explanation says "increasing X would decrease risk," actually increase X and verify the prediction changes as expected.

Cross-Method Validation

Compare explanations from multiple methods. If SHAP, LIME, and feature permutation all identify different "most important" features, investigate further—at least one is likely wrong.

Domain Expert Review

Have experts evaluate whether explanations align with domain knowledge. An explanation that violates known causal relationships is problematic, even if it has good metrics.

Fairness Metrics

Explanations should treat different demographic groups fairly.

Explanation Parity

Do explanations have similar quality (fidelity, stability) across demographic groups? Lower quality for protected groups could signal bias.

Feature Importance Parity

Do the same features drive predictions across groups? If the model relies heavily on feature X for Group A but feature Y for Group B, this might indicate disparate treatment.

Counterfactual Fairness

Do counterfactual explanations require changing protected attributes? If the counterfactual for a denied loan is "change your race," the model is clearly discriminating unfairly.

Fairness Metric	What It Measures	WIA-AI-009 Threshold
Fidelity Parity	Fidelity difference across groups	< 0.05
Stability Parity	Stability difference across groups	< 0.05
Feature Distribution	Top features similarity across groups	> 0.70
Protected Attribute Weight	Importance of protected attributes	< 0.05 (ideally 0)

Automated Validation Pipelines

The WIA-AI-009 standard recommends automated validation as part of the ML pipeline:

// Automated explanation validation
class ExplanationValidator {
  constructor(model, explainer) {
    this.model = model;
    this.explainer = explainer;
  }

  async validateExplanation(instance, explanation) {
    const results = {
      fidelity: await this.testFidelity(instance, explanation),
      stability: await this.testStability(instance),
      ablation: await this.testAblation(instance, explanation),
      consistency: await this.testConsistency(instance, explanation),
      passed: false
    };

    // All metrics must meet thresholds
    results.passed = (
      results.fidelity > 0.85 &&
      results.stability > 0.80 &&
      results.ablation.correlation > 0.70 &&
      results.consistency > 0.75
    );

    return results;
  }

  async testAblation(instance, explanation) {
    const topFeatures = explanation.getTopFeatures(5);
    const originalPrediction = this.model.predict(instance);

    let correlationSum = 0;

    for (const feature of topFeatures) {
      const ablatedInstance = {...instance};
      delete ablatedInstance[feature.name];

      const ablatedPrediction = this.model.predict(ablatedInstance);
      const predictionChange = Math.abs(originalPrediction - ablatedPrediction);
      const expectedChange = Math.abs(feature.importance);

      correlationSum += (predictionChange / expectedChange);
    }

    return { correlation: correlationSum / topFeatures.length };
  }
}

Human Evaluation Protocols

Automated metrics don't capture everything. Human evaluation remains essential.

Simulatability Testing

Can users predict the model's output given the explanation? Show users explanations without predictions, ask them to guess the prediction, measure accuracy.

Trust Calibration

Do explanations lead to appropriate trust? Users should trust the model more when it's correct and less when it's wrong. Explanations should help users distinguish these cases.

Decision Support Evaluation

Do explanations improve human decision-making? In human-AI collaborative tasks, measure whether explanations help humans catch model errors and make better choices.

Detecting Adversarial Explanations

Models can be designed to produce plausible but misleading explanations. Detection strategies:

Explanation-Prediction Decoupling Test

If you can change the explanation without changing the prediction (or vice versa), the explanation may be decoupled from actual reasoning.

Counterfactual Verification

Generate counterfactuals suggested by the explanation. If they don't actually flip the prediction, the explanation is suspect.

Cross-Architecture Comparison

Compare explanations from models with different architectures but similar performance. Drastically different explanations for the same input warrant investigation.

Explanation Quality Dashboards

The WIA-AI-009 standard includes specifications for monitoring explanation quality in production:

{
  "explanation_quality_report": {
    "model_id": "loan_classifier_v3",
    "reporting_period": "2025-01-01 to 2025-01-31",
    "metrics": {
      "average_fidelity": 0.92,
      "average_stability": 0.87,
      "average_consistency": 0.84,
      "completeness_score": 0.98
    },
    "fairness_metrics": {
      "fidelity_parity": 0.03,
      "stability_parity": 0.04,
      "protected_attribute_importance": 0.02
    },
    "validation_tests": {
      "ablation_correlation": 0.79,
      "perturbation_alignment": 0.81,
      "cross_method_agreement": 0.76
    },
    "alerts": [
      {
        "type": "consistency_drop",
        "severity": "medium",
        "description": "Consistency for age > 65 dropped to 0.68",
        "recommendation": "Investigate data distribution changes"
      }
    ],
    "human_evaluation": {
      "simulatability": 0.72,
      "trust_calibration": 0.81,
      "decision_support_improvement": 0.15
    }
  }
}

Continuous Monitoring

Explanation quality can degrade over time due to concept drift, data distribution changes, or model updates. Continuous monitoring ensures sustained quality:

• Weekly automated tests: Run validation suite on sample of production instances
• Monthly human evaluation: Have domain experts review sample explanations
• Quarterly comprehensive audit: Full validation across all metrics and subpopulations
• Alert on degradation: Automatic notifications when metrics drop below thresholds

Best Practices for Trust

1. Multi-metric evaluation: Never rely on a single metric; use a comprehensive suite
2. Subpopulation analysis: Check metrics for different demographic and feature-based subgroups
3. Longitudinal tracking: Monitor how metrics evolve over time
4. Cross-method validation: Verify important findings with multiple explanation methods
5. Domain expert involvement: Combine automated metrics with expert review
6. Transparency about limitations: Document known weaknesses and edge cases
7. User feedback loops: Collect feedback on explanation usefulness and adjust accordingly