🚨 Intrusion Detection

In the modern digital landscape, cyber threats evolve at an unprecedented pace. Traditional perimeter defenses—firewalls and antivirus software—are no longer sufficient. Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) provide the critical capability to monitor network traffic, detect malicious activity, and respond to threats in real-time.

This comprehensive guide covers the WIA-SEC-016 standard for intrusion detection and prevention, establishing international best practices for protecting digital infrastructure.

What is Intrusion Detection?

Intrusion detection is the process of monitoring network or system activities for malicious actions or policy violations. An IDS analyzes traffic patterns, compares them against known attack signatures, and alerts security teams when suspicious activity is detected.

Key capabilities include:

• Real-time Monitoring: Continuous analysis of network traffic 24/7
• Threat Detection: Identification of attacks ranging from port scans to advanced persistent threats (APTs)
• Alert Generation: Immediate notification of security incidents
• Forensic Analysis: Detailed packet captures for post-incident investigation

IDS vs. IPS: Understanding the Difference

While the terms are often used interchangeably, IDS and IPS serve distinct purposes:

IDS (Intrusion Detection System) operates in passive mode:

• Monitors a copy of network traffic via network tap or SPAN port
• Alerts on detected threats but cannot block traffic
• Zero risk of disrupting legitimate traffic
• Ideal for compliance monitoring and forensics

IPS (Intrusion Prevention System) operates in inline mode:

• Sits directly in the network path
• Can actively block or modify malicious traffic
• Provides real-time protection
• Requires careful tuning to avoid false positives that block legitimate traffic

The Evolution of Intrusion Detection

Intrusion detection has evolved through several generations:

First Generation (1980s-1990s): Simple signature matching against known attack patterns. Limited to detecting variations of documented exploits.

Second Generation (2000s): Protocol analysis and stateful inspection. Ability to detect evasion techniques like packet fragmentation.

Third Generation (2010s): Behavioral analysis and anomaly detection. Machine learning algorithms identify zero-day attacks by detecting deviations from normal traffic patterns.

Fourth Generation (2020s-present): AI-powered threat hunting with automated response. Integration with SIEM, SOAR, and threat intelligence platforms for comprehensive security orchestration.

Why Intrusion Detection Matters

• Compliance: Required by PCI DSS, HIPAA, ISO 27001, and other regulatory frameworks
• Early Warning: Detects reconnaissance and early-stage attacks before damage occurs
• Visibility: Provides insight into network traffic and security posture
• Incident Response: Supplies critical forensic data for investigating breaches
• Defense in Depth: Adds essential layer to multi-layered security strategy

Effective intrusion detection requires multiple complementary approaches. This chapter explores the three primary detection methodologies employed by modern IDS/IPS systems.

Signature-Based Detection

Signature-based detection—also called pattern matching or misuse detection—identifies attacks by comparing network traffic against a database of known attack signatures. Each signature defines the unique characteristics of a specific exploit.

How it Works:

1. IDS maintains database of attack signatures (rules)
2. Each packet is analyzed and compared against signatures
3. Match triggers alert with signature ID and details
4. Signatures updated regularly from threat intelligence feeds

Example Signature (Snort syntax):

alert tcp any any -> $HOME_NET 80 (
  msg:"SQL Injection - UNION SELECT";
  content:"UNION"; nocase; http_uri;
  content:"SELECT"; nocase; http_uri;
  classtype:web-application-attack;
  sid:2100498;
)

Advantages:

• Extremely accurate for known attacks (>99% precision)
• Low false positive rate when properly tuned
• Fast processing (<1ms latency per signature)
• Easy to understand and audit

Limitations:

• Cannot detect zero-day exploits (unknown attacks)
• Requires constant signature updates
• Vulnerable to evasion techniques (encoding, obfuscation)
• Signature database can grow large (100,000+ rules)

Anomaly-Based Detection

Anomaly detection identifies threats by recognizing deviations from established baselines of normal behavior. This approach can detect previously unknown attacks but typically generates more false positives.

Process:

1. Baseline Establishment: Monitor network for 30-90 days to create statistical profile of normal traffic
2. Anomaly Scoring: Compare current traffic against baseline using statistical or ML models
3. Threshold Evaluation: Alert when anomaly score exceeds configured threshold
4. Continuous Learning: Update baseline to adapt to legitimate network changes

Detected Anomalies:

• Unusual traffic volume (potential DDoS)
• Atypical protocol distribution (e.g., sudden increase in ICMP)
• Connection to previously unseen IP addresses
• Access to resources at unusual times
• Abnormal packet sizes or inter-arrival times

Statistical Methods:

• Mean/Standard Deviation: Flag values >3 standard deviations from mean
• Time-Series Analysis: Detect patterns that deviate from historical trends
• Clustering: Group similar traffic; outliers indicate anomalies

Protocol Analysis

Protocol analysis—also called stateful protocol analysis—validates network traffic against protocol specifications defined in RFCs. This method detects attacks that exploit protocol weaknesses or violate protocol standards.

Deep Packet Inspection (DPI):

IDS examines both packet headers and payload content, validating:

• Header Fields: Ensure valid values for TTL, flags, checksums
• Protocol State: Verify correct TCP handshake sequence, proper session teardown
• Application Layer: Parse HTTP, DNS, TLS, SMB to detect malformed requests

Protocol Anomalies Detected:

• Malformed HTTP requests (missing headers, invalid methods)
• DNS queries exceeding maximum length (potential DNS tunneling)
• TLS certificate validation failures
• SMB requests with suspicious file paths (directory traversal)
• TCP packets with contradictory flags (SYN+FIN)

Evasion Detection:

Protocol analysis identifies techniques attackers use to evade IDS:

• Fragmentation: Splitting attack across multiple packets
• Encoding: Unicode, Base64, URL encoding to obfuscate payloads
• Timing Attacks: Slow scans that fall below rate thresholds

Hybrid Approach

Modern IDS/IPS systems employ all three methodologies in concert:

• Signature-based detection for known, high-confidence threats
• Anomaly detection for zero-day and insider threats
• Protocol analysis to prevent evasion and ensure compliance

This defense-in-depth strategy maximizes detection coverage while managing false positive rates.

Network-based Intrusion Detection Systems (NIDS) monitor traffic across network segments, providing comprehensive visibility into communication patterns and potential threats. Deployed at strategic network chokepoints, NIDS serve as the first line of defense against external and lateral attacks.

Architecture Components

1. Packet Capture Interface

The packet capture module intercepts network traffic for analysis:

• Libpcap/WinPcap: Standard library for packet capture on Unix/Windows
• PF_RING: High-performance packet capture for multi-gigabit networks
• AF_PACKET: Linux kernel packet socket for zero-copy capture
• DPDK: Data Plane Development Kit for 10-100 Gbps throughput

2. Packet Decoder

Decodes protocol layers from Ethernet to application:

• Layer 2: Ethernet, VLAN tags
• Layer 3: IPv4, IPv6, ICMP, ARP
• Layer 4: TCP, UDP, SCTP
• Layer 7: HTTP, DNS, TLS, FTP, SSH, SMTP

3. Preprocessors

Normalize and reassemble traffic before detection:

• TCP Stream Reassembly: Reconstructs data streams from packet sequences
• IP Defragmentation: Reassembles fragmented IP packets
• HTTP Normalization: Decodes URL encoding, handles chunked encoding
• Protocol Decoders: Parse application protocols for deep inspection

4. Detection Engine

Core component that applies detection rules:

• Multi-pattern matching using Aho-Corasick algorithm
• Regular expression evaluation (PCRE)
• Statistical anomaly detection
• Protocol validation logic

5. Alert Output Module

Formats and transmits alerts to monitoring systems:

• Syslog (local or remote)
• JSON to SIEM platforms
• PCAP logging for forensics
• Database storage (MySQL, PostgreSQL)

Deployment Models

Passive Monitoring (Tap/SPAN)

IDS receives copy of traffic without disrupting network flow:

• Network Tap: Physical device that splits optical or copper signal
• SPAN/Mirror Port: Switch feature that copies traffic to monitoring port
• Advantages: Zero impact on network, cannot cause outage
• Limitations: Cannot block attacks, alert-only mode

Inline Mode (IPS)

Traffic flows through IDS, enabling active prevention:

• Deployed as gateway between network segments
• Can drop malicious packets in real-time
• Requires high availability (HA) configuration
• Adds minimal latency (<5ms typical)

Performance Considerations

Throughput Requirements:

• Small Office: 100 Mbps - 1 Gbps
• Enterprise: 10 Gbps - 40 Gbps
• Data Center: 100 Gbps+

Optimization Techniques:

• Multi-threading: Parallel packet processing across CPU cores
• Hardware Acceleration: FPGA or GPU offload for signature matching
• Rule Pruning: Disable irrelevant signatures for environment
• Load Balancing: Distribute traffic across multiple sensors

Strategic Placement

Optimal sensor placement maximizes visibility:

• Perimeter: Monitor all traffic entering/exiting organization
• DMZ: Protect public-facing servers (web, email, DNS)
• Internal Segments: Detect lateral movement between zones
• Data Center: Monitor traffic to critical assets and databases

While NIDS monitors network traffic, Host-based Intrusion Detection Systems (HIDS) provide visibility into individual systems. HIDS agents installed on servers, workstations, and critical endpoints detect attacks that evade network monitoring.

What HIDS Monitors

1. File Integrity Monitoring (FIM)

Detects unauthorized changes to critical files:

• System binaries (/bin, /sbin, C:\Windows\System32)
• Configuration files (/etc, Windows Registry)
• Application files (web server root, database files)
• Cryptographic verification using SHA-256 hashes

2. Log Analysis

Monitors system and application logs for security events:

• Authentication logs (successful/failed logins)
• Privilege escalation attempts (sudo, UAC)
• System errors and crashes
• Application-specific logs (web server, database)

3. Process Monitoring

Tracks process execution and behavior:

• New process creation (parent-child relationships)
• Unusual command-line arguments
• Suspicious process names (mimicking system processes)
• Resource consumption anomalies (CPU, memory)

4. Rootkit Detection

Identifies hidden malware:

• Kernel module verification
• Hidden process/file detection
• System call hooking detection
• Comparison of kernel vs. user-space views

Advantages of HIDS

• Encrypted Traffic: Can inspect traffic before encryption or after decryption
• Insider Threats: Detects malicious actions by authorized users
• System-Level Visibility: Sees activities invisible to NIDS (local file access, process execution)
• Compliance: Required by many regulations for critical systems

Popular HIDS Solutions

• OSSEC: Open-source HIDS with log analysis and FIM
• Wazuh: OSSEC fork with enhanced SIEM integration
• Tripwire: Commercial FIM solution
• AIDE: Advanced Intrusion Detection Environment (Linux)

Deployment Best Practices

• Deploy on all critical servers and administrative workstations
• Centralize log collection to prevent tampering
• Baseline normal behavior before enabling alerts
• Monitor HIDS agent health and connectivity
• Integrate with SIEM for correlation with network events

Intrusion Prevention Systems extend IDS capabilities with active defense. Deployed inline, IPS can block attacks in real-time, providing immediate protection rather than just alerting.

IPS Actions

When a threat is detected, IPS can execute various responses:

• Drop Packet: Silently discard malicious packet
• Reset Connection: Send TCP RST to terminate session
• Block IP Address: Dynamically add firewall rule to block source IP
• Rate Limit: Throttle suspicious traffic rather than complete block
• Modify Packet: Remove malicious payload while allowing benign traffic

Deployment Considerations

High Availability (HA)

IPS is a critical network component requiring redundancy:

• Active-Passive: Primary IPS with standby unit for failover
• Active-Active: Multiple IPS units share traffic load
• Heartbeat Monitoring: Detect failures and trigger failover <2 seconds
• State Synchronization: Maintain session tables across units

Performance Impact

Inline deployment adds latency—minimization strategies:

• Hardware acceleration (FPGA, ASIC)
• Optimized rule sets (fast patterns, efficient regex)
• Dedicated high-performance hardware
• Target: <5ms latency at line rate

Tuning for Accuracy

IPS requires careful tuning to avoid blocking legitimate traffic:

False Positive Management:

• Start in detection-only mode; monitor for false positives
• Whitelist trusted sources and applications
• Adjust signature thresholds based on environment
• Use threat intelligence to prioritize high-confidence signatures

Testing Methodology:

1. Lab Testing: Validate rules in isolated environment
2. Pilot Deployment: Enable IPS on non-critical segment
3. Gradual Rollout: Expand to production incrementally
4. Continuous Monitoring: Track blocked traffic and user complaints

Traditional signature-based detection cannot keep pace with the volume and sophistication of modern threats. Machine learning offers powerful techniques to detect zero-day attacks and advanced persistent threats (APTs) by identifying behavioral anomalies.

Supervised Learning

Train models on labeled datasets of benign and malicious traffic:

Random Forest Classifier:

• Ensemble of decision trees for classification
• Features: packet size, protocol, inter-arrival time, entropy
• Accuracy: 92-95% on test data
• Fast inference: <1ms per packet

Neural Networks:

• Deep learning for complex pattern recognition
• Architecture: Multi-layer perceptron with 50+ input features
• Training: 100,000+ labeled samples
• Can detect novel attack variants signature-based systems miss

Unsupervised Learning

Detect anomalies without labeled training data:

Clustering (K-Means):

• Group similar traffic patterns
• Outliers indicate potential attacks
• Useful for discovering unknown attack types

Autoencoders:

• Neural network that learns to compress and reconstruct normal traffic
• High reconstruction error indicates anomaly
• Effective for zero-day detection

Challenges

• Training Data: Requires large, representative datasets
• False Positives: ML models typically have 5-10% false positive rate
• Adversarial ML: Attackers can craft traffic to evade ML detectors
• Explainability: Difficult to understand why ML model flagged traffic

Hybrid Approach

Combining signatures and ML maximizes effectiveness:

• Use signatures for known, high-confidence threats (low false positives)
• Use ML for anomaly detection and zero-days (higher false positives but catches novel attacks)
• Prioritize ML alerts based on risk score
• Feed confirmed ML detections back into signature database

Security Information and Event Management (SIEM) platforms aggregate logs from across the enterprise, enabling correlation analysis that identifies complex attack patterns invisible to individual systems. IDS integration with SIEM is essential for comprehensive threat detection.

Integration Methods

1. Syslog

Standard protocol for log transmission:

• RFC 5424 format with structured data
• UDP (port 514) or TCP (port 6514) transport
• TLS encryption for sensitive alerts
• Widely supported by all SIEM platforms

2. API Integration

Direct API calls for rich alert data:

• Splunk HTTP Event Collector (HEC)
• Elasticsearch REST API
• Custom webhooks for SOAR platforms
• JSON payloads with complete alert context

3. Agent-Based Forwarding

• Splunk Universal Forwarder
• Elastic Beats (Filebeat, Packetbeat)
• Monitors IDS log files and forwards to SIEM

Correlation Use Cases

Multi-Stage Attack Detection:

Correlate IDS alerts with other security events:

1. IDS detects port scan from external IP
2. Firewall logs show connection attempt to discovered service
3. IDS alerts on SQL injection attempt
4. Web server logs show suspicious database queries
5. SIEM Correlation: Identifies coordinated attack spanning multiple systems

Insider Threat Detection:

• User logs in from unusual location (authentication logs)
• Accesses sensitive files not typically used (HIDS file monitoring)
• Large data transfer to external IP (IDS network monitoring)
• SIEM Alert: Potential data exfiltration by insider

Alert Enrichment

SIEM enhances IDS alerts with additional context:

• Threat Intelligence: IP reputation, malware family, known campaigns
• Asset Information: Criticality, owner, vulnerabilities
• User Context: Job role, recent activities, access privileges
• Historical Data: Previous alerts from same source, attack trends

Automated Response

SIEM triggers automated playbooks based on IDS alerts:

• High-severity alert → Isolate affected host via firewall/NAC
• Malware C2 detected → Block domain at DNS level
• Repeated failed authentication → Lock account, notify SOC
• Data exfiltration → Snapshot system for forensics, alert CISO

Successful IDS/IPS deployment requires careful planning, implementation, and ongoing maintenance. This chapter provides actionable guidance for enterprise deployments.

Pre-Deployment Planning

1. Define Objectives

• Compliance requirements (PCI DSS, HIPAA)
• Threat landscape (external attacks, insider threats)
• Asset criticality (protect crown jewels first)
• Incident response capabilities

2. Network Assessment

• Map network topology and traffic flows
• Identify chokepoints for sensor placement
• Measure traffic volumes to size hardware
• Document VLANs, subnets, and security zones

3. Resource Allocation

• Hardware: Servers, network taps, high-speed NICs
• Software: IDS/IPS licenses, signature subscriptions
• Personnel: SOC analysts, incident responders
• Budget: Initial capex + ongoing opex

Implementation Phase

Step 1: Lab Testing

• Deploy IDS in isolated lab environment
• Test with sample attacks (Metasploit, exploit frameworks)
• Validate alert generation and delivery
• Benchmark performance under load

Step 2: Pilot Deployment

• Deploy on non-critical network segment
• Monitor in detection-only mode for 2-4 weeks
• Tune rules to reduce false positives
• Document lessons learned

Step 3: Production Rollout

• Deploy incrementally across network
• Start with perimeter, then internal segments
• Enable IPS blocking after tuning period
• Maintain 24/7 monitoring capability

Tuning and Optimization

Rule Management:

• Disable signatures irrelevant to environment (e.g., Windows rules on Linux network)
• Adjust thresholds for noisy signatures
• Create custom rules for organization-specific threats
• Update signatures daily from threat feeds

Performance Tuning:

• Allocate CPU cores to detection threads
• Increase packet buffers to prevent drops
• Enable hardware acceleration where available
• Monitor CPU, memory, and packet loss metrics

Operational Excellence

Alert Triage:

1. Severity Classification: Prioritize Critical → High → Medium → Low
2. Enrichment: Add context from SIEM, threat intel, asset database
3. Validation: Confirm true positive through packet analysis
4. Response: Execute incident response playbook
5. Documentation: Record findings in ticketing system

Metrics and Reporting:

• Alert volume trends (daily, weekly, monthly)
• Top triggered signatures
• Mean time to detect (MTTD) and respond (MTTR)
• False positive rate
• Compliance status (uptime, signature currency)

Continuous Improvement

• Weekly review of high-severity alerts and false positives
• Monthly rule optimization sessions
• Quarterly penetration testing to validate detection
• Annual architecture review and technology refresh

Common Pitfalls to Avoid

• Alert Fatigue: Too many low-priority alerts → analysts miss critical events
• Insufficient Tuning: High false positive rate erodes confidence
• Lack of Integration: IDS operating in silo without SIEM correlation
• Outdated Signatures: Failure to update rules → missed detections
• No Incident Response: Alerts generated but not acted upon

弘益人間 · Through vigilant monitoring and rapid response, we protect the digital infrastructure that benefits all humanity.