π Chapter 2: Risk Assessment and Classification Frameworks
2.1 Understanding AI Risk Categories
Risk assessment forms the foundation of effective AI safety protocols. Unlike traditional software systems, AI models present unique risk profiles that require specialized evaluation frameworks. The process begins with systematic identification of potential harms, followed by categorization based on severity, likelihood, and controllability.
Modern AI risk frameworks recognize four primary risk categories:
- Performance Risks: Failures in accuracy, reliability, or robustness that lead to incorrect decisions
- Security Risks: Vulnerabilities to adversarial attacks, data poisoning, or model theft
- Societal Risks: Harms related to bias, fairness, privacy, and broader social impacts
- Control Risks: Challenges in maintaining human oversight and preventing unintended autonomous actions
Each category requires distinct assessment methodologies, mitigation strategies, and monitoring approaches. Organizations must evaluate their AI systems across all four dimensions to develop comprehensive safety protocols.
2.2 NIST AI Risk Management Framework
The National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) provides a structured approach to identifying, assessing, and managing AI-related risks. Released in 2023 and updated in 2025, the framework emphasizes a socio-technical perspective that considers both technical characteristics and broader organizational context.
The NIST AI RMF is organized around four core functions:
| Function | Purpose | Key Activities |
|---|---|---|
| GOVERN | Establish culture of risk management | Define roles, policies, and oversight structures |
| MAP | Understand AI system context | Identify stakeholders, document use cases, analyze impacts |
| MEASURE | Assess and benchmark risks | Test performance, evaluate fairness, monitor outcomes |
| MANAGE | Prioritize and respond to risks | Implement controls, document decisions, enable transparency |
The framework is designed to be flexible and adaptable across different sectors, organization sizes, and AI applications. It does not prescribe specific technical solutions but rather provides a common language and process for risk management.
2.2.1 GOVERN Function Deep Dive
The GOVERN function establishes accountability structures and processes for AI risk management. This includes defining organizational roles such as AI Safety Officers, establishing review boards, and creating policies that integrate AI considerations into existing risk management practices. Governance also encompasses documentation requirements, incident reporting procedures, and mechanisms for stakeholder input.
2.2.2 MAP Function in Practice
Mapping involves comprehensive documentation of the AI system's intended purpose, operational context, and potential impacts. This includes identifying all stakeholders who may be affected by the system, documenting data sources and characteristics, and analyzing how the system fits within broader socio-technical systems. The mapping phase often reveals unanticipated risks that become apparent only when the full context is understood.
2.3 EU AI Act Risk Classification
The European Union's AI Act takes a fundamentally different approach, categorizing AI systems into predefined risk levels based on their application domain and potential for harm. This risk-based regulatory framework became fully effective in 2024 and has influenced AI regulation globally.
| Risk Level | Definition | Examples | Requirements |
|---|---|---|---|
| Unacceptable | Poses clear threat to safety or fundamental rights | Social scoring by governments, real-time biometric surveillance | Prohibited entirely |
| High Risk | Significant potential for harm in critical domains | Medical devices, critical infrastructure, law enforcement tools | Conformity assessment, risk management, transparency |
| Limited Risk | Requires transparency for informed user decisions | Chatbots, emotion recognition, deepfakes | Disclosure to users that they're interacting with AI |
| Minimal Risk | No significant risk to rights or safety | Spam filters, AI-enabled video games | No specific obligations (voluntary codes encouraged) |
High-risk AI systems under the EU AI Act must meet stringent requirements including:
- Risk management system throughout the entire lifecycle
- Data governance ensuring training data quality and relevance
- Technical documentation providing detailed system information
- Record-keeping with automatic logging of events
- Transparency and provision of information to users
- Human oversight measures including "stop" buttons
- Robustness, accuracy, and cybersecurity measures
2.4 Threat Modeling for AI Systems
Threat modeling adapts traditional cybersecurity practices to the unique characteristics of AI systems. The process systematically identifies potential attack vectors, evaluates their likelihood and impact, and prioritizes mitigation efforts.
AI-specific threats include:
| Threat Category | Attack Vector | Impact | Mitigation Strategy |
|---|---|---|---|
| Data Poisoning | Malicious data injection during training | Backdoors, performance degradation | Data provenance tracking, anomaly detection |
| Adversarial Examples | Crafted inputs exploiting model weaknesses | Incorrect predictions, bypassing safety controls | Adversarial training, input validation |
| Model Extraction | Querying to steal model functionality | IP theft, enabling further attacks | Query limiting, output perturbation |
| Prompt Injection | Malicious instructions in user input | Unauthorized actions, data exfiltration | Input sanitization, prompt hardening |
| Model Inversion | Reconstructing training data from model | Privacy violations, data exposure | Differential privacy, access controls |
εΌηδΊΊι (Hongik Ingan)
"Benefit All Humanity"
Comprehensive risk assessment ensures AI systems serve the common good by identifying and addressing potential harms before deployment, protecting vulnerable populations and maintaining public trust in AI technologies.
2.5 Sector-Specific Risk Frameworks
Different industries face distinct AI-related risks, leading to development of specialized assessment frameworks tailored to sector-specific concerns and regulatory requirements.
2.5.1 Healthcare AI Risk Assessment
Medical AI systems present unique risks related to patient safety, diagnostic accuracy, and clinical workflow integration. Assessment frameworks for healthcare AI emphasize validation against diverse patient populations, failure mode analysis, and integration with existing clinical decision support systems. The FDA has developed specific guidance for AI/ML-based medical devices, including requirements for predetermined change control plans.
2.5.2 Financial Services AI Risk
Financial institutions must assess AI systems for fair lending compliance, market manipulation risks, and systemic financial stability impacts. Model risk management frameworks in banking typically include independent model validation, ongoing performance monitoring, and stress testing under adverse scenarios. The Federal Reserve and OCC provide detailed guidance on model risk management that applies to AI systems.
2.5.3 Autonomous Vehicle Safety Assessment
Self-driving car AI requires assessment of physical safety risks, cybersecurity vulnerabilities, and ethical decision-making in unavoidable accident scenarios. Frameworks like NHTSA's AV Test Initiative and ISO 26262 functional safety standard guide comprehensive risk evaluation for automotive AI systems.
2.6 Quantitative Risk Scoring Methodologies
While many AI risks resist precise quantification, structured scoring methodologies help prioritize resources and communicate risk levels to stakeholders. Several approaches have emerged for AI risk scoring:
| Methodology | Approach | Strengths | Limitations |
|---|---|---|---|
| Risk Matrix | Likelihood Γ Impact scoring | Simple, intuitive, widely understood | Oversimplifies complex risks |
| Failure Modes & Effects Analysis (FMEA) | Systematic evaluation of failure scenarios | Comprehensive, structured process | Time-consuming, may miss novel failure modes |
| Bow-Tie Analysis | Visual representation of risk paths | Shows preventive and mitigative controls | Complex for systems with many risk paths |
| Monte Carlo Simulation | Probabilistic modeling of risk scenarios | Handles uncertainty quantitatively | Requires extensive data for parameter estimation |
2.7 Continuous Risk Reassessment
AI system risks are not static. Models can drift over time as input distributions change, new attack vectors emerge, and societal norms evolve. Effective risk management requires continuous reassessment through ongoing monitoring and periodic comprehensive reviews.
Key triggers for risk reassessment include:
- Significant changes in model performance metrics
- Deployment to new populations or contexts
- Discovery of novel attack techniques affecting similar systems
- Regulatory changes or updated guidance
- Incident reports from users or monitoring systems
- Scheduled periodic reviews (typically quarterly or semi-annually)
2.8 Documenting Risk Assessment
Thorough documentation of risk assessments serves multiple purposes: providing evidence of due diligence for regulators, facilitating organizational learning, enabling reproducibility, and communicating risks to stakeholders. The WIA AI Safety Protocol specifies standard documentation formats that streamline this process while ensuring completeness.
Essential elements of risk assessment documentation include:
- System description and intended use case
- Identified risks with severity and likelihood scores
- Assessment methodology and tools used
- Assumptions and limitations of the assessment
- Implemented controls and residual risks
- Responsible parties and oversight structures
- Reassessment schedule and triggers
Summary
Effective AI risk assessment requires systematic evaluation across multiple dimensions: technical performance, security vulnerabilities, societal impacts, and control challenges. Leading frameworks like NIST AI RMF and the EU AI Act provide structured approaches, while sector-specific methodologies address domain-particular concerns. Risk assessment is not a one-time activity but an ongoing process that must adapt as systems evolve and new threats emerge.
Key takeaways include:
- AI risks span performance, security, societal, and control domains
- Multiple frameworks exist; organizations should select approaches matching their context
- Threat modeling must account for AI-specific attacks like data poisoning and adversarial examples
- Sector-specific risks require tailored assessment methodologies
- Risk assessment must be continuous, with regular reassessment triggered by system changes or new information
Review Questions
- What are the four core functions of the NIST AI Risk Management Framework?
- Explain the difference between "high risk" and "limited risk" AI systems under the EU AI Act.
- What is data poisoning, and how does it differ from adversarial examples?
- Why do financial institutions require specialized AI risk assessment frameworks?
- List three triggers that should prompt reassessment of an AI system's risk profile.
- How does the WIA AI Safety Protocol support standardized risk assessment documentation?
Looking Ahead
In Chapter 3, we will explore Security Considerations for AI Systems, diving deep into cybersecurity challenges unique to machine learning models. We'll examine defense strategies against adversarial attacks, secure deployment architectures, and the emerging field of AI-specific security testing. You'll learn practical techniques for hardening AI systems against malicious actors while maintaining usability and performance.