Ch5: AI/ML Development|WIA-FIN-020

Machine Learning Pipeline

Building production-ready credit scoring models requires systematic approach combining data science best practices with regulatory requirements.

1. Data Preparation

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# Load historical applications with outcomes
df = pd.read_parquet('credit_applications_2020_2024.parquet')

# Target variable: 1=default, 0=paid as agreed
# Definition: 90+ days past due within 24 months
df['target'] = (df['worst_status_24m'] >= 90).astype(int)

# Train/Val/Test split (60/20/20) with time-based split
train = df[df['application_date'] < '2023-01-01']
val = df[(df['application_date'] >= '2023-01-01') & 
         (df['application_date'] < '2024-01-01')]
test = df[df['application_date'] >= '2024-01-01']

print(f"Train: {len(train):,} | Val: {len(val):,} | Test: {len(test):,}")
print(f"Default Rate - Train: {train.target.mean():.2%}")

2. Feature Engineering

Bureau Features (50+): Payment history, utilization, inquiries, derogatory marks
Alternative Data (100+): Bank transactions, rent, utilities
Derived Features (200+): Ratios, trends, aggregations
Interaction Features (150+): Combinations of important variables

3. Model Training

import xgboost as xgb
from sklearn.metrics import roc_auc_score

# XGBoost for credit scoring
model = xgb.XGBClassifier(
    max_depth=6,
    learning_rate=0.05,
    n_estimators=200,
    objective='binary:logistic',
    subsample=0.8,
    colsample_bytree=0.8,
    scale_pos_weight=10,  # Handle imbalance
    eval_metric='auc',
    early_stopping_rounds=20
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=10
)

# Evaluate
val_pred = model.predict_proba(X_val)[:, 1]
auc = roc_auc_score(y_val, val_pred)
print(f"Validation AUC: {auc:.4f}")

4. Model Calibration

Ensure predicted probabilities match observed default rates:

from sklearn.calibration import CalibratedClassifierCV

# Isotonic regression calibration
calibrated_model = CalibratedClassifierCV(
    model, 
    method='isotonic',
    cv='prefit'
)
calibrated_model.fit(X_val, y_val)

# Verify calibration
from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(
    y_test, 
    calibrated_model.predict_proba(X_test)[:, 1],
    n_bins=10
)
# Plot and verify diagonal alignment

5. Model Explainability

import shap

# Generate SHAP explanations
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global feature importance
shap.summary_plot(shap_values, X_test)

# Individual explanation
def explain_score(applicant_features):
    shap_vals = explainer.shap_values(applicant_features)
    
    # Get top factors
    factors = []
    for idx in np.argsort(np.abs(shap_vals[0]))[-5:]:
        factors.append({
            'name': feature_names[idx],
            'value': applicant_features[0, idx],
            'impact': shap_vals[0, idx],
            'direction': 'positive' if shap_vals[0, idx] > 0 else 'negative'
        })
    
    return factors

Advanced Techniques

Ensemble Methods

Combine XGBoost + LightGBM + Neural Network
Stacking with logistic regression meta-learner
Weighted averaging based on validation performance
Improves robustness and accuracy

Hyperparameter Optimization

Grid Search: Exhaustive but slow
Random Search: More efficient
Bayesian Optimization: Most efficient (Optuna, Hyperopt)
AutoML: H2O.ai, Auto-sklearn for automated tuning

Class Imbalance Handling

SMOTE: Synthetic minority oversampling
Undersampling: Reduce majority class
Class Weights: Penalize false negatives more
Threshold Tuning: Optimize decision boundary

Feature Selection

Remove low-importance features (< 0.1% contribution)
Eliminate highly correlated features (r > 0.95)
Recursive feature elimination (RFE)
L1 regularization (Lasso) for sparsity

Model Validation

Performance Metrics

AUC-ROC: Primary metric (target: 0.82+)
Precision-Recall AUC: For imbalanced data
KS Statistic: Maximum separation between good/bad
Gini Coefficient: 2×AUC - 1
Brier Score: Calibration quality

Stability Testing

PSI: Population Stability Index (< 0.1 stable)
CSI: Characteristic Stability Index
Out-of-time validation: Test on future data
Stress testing: Recession scenarios

Fairness Validation

Demographic parity across protected groups
Equal opportunity (TPR equality)
Calibration by subgroup
Disparate impact ratio (> 0.80)

Deployment Strategy

Model Serialization

import joblib
import mlflow

# Save model artifacts
joblib.dump(model, 'credit_model_v1.pkl')
joblib.dump(scaler, 'scaler_v1.pkl')
joblib.dump(feature_names, 'features_v1.pkl')

# MLflow tracking
mlflow.log_model(model, "credit_scoring_model")
mlflow.log_metrics({
    'auc': auc,
    'precision': precision,
    'recall': recall
})

A/B Testing

Run new model on 10% of traffic initially
Compare approval rates, default rates, revenue
Gradually increase to 25% → 50% → 100%
Revert if performance degrades

Monitoring

Model Performance: Track AUC, precision, recall weekly
Data Drift: Compare feature distributions
Prediction Drift: Monitor score distribution changes
Business Metrics: Approval rate, default rate, revenue

Retraining Schedule

Quarterly: Standard retraining with new data
Monthly: If rapid market changes
Triggered: If PSI > 0.25 or performance drops > 5%
Annual: Major model overhaul with architecture changes

Next chapter covers ensuring fairness, ethics, and regulatory compliance in credit scoring systems.