👁️ Chapter 4: Sensors & Perception Systems

4.1 Sensor Suite Overview

Humanoid robots require comprehensive sensing capabilities to perceive their environment, understand their own state, and interact safely with humans. WIA-ROB-019 defines minimum sensor requirements across multiple modalities, ensuring robots can operate reliably in diverse conditions.

4.1.1 Sensor Classification

Sensors are classified into two primary categories:

Proprioceptive Sensors

Measure the robot's internal state—joint positions, velocities, forces, and body orientation. Essential for control and state estimation.

• Joint Encoders: Optical or magnetic encoders measuring angular position. Resolution: minimum 12-bit (0.088°), preferred 17-bit (0.0027°). Update rate: 1-10 kHz.
• Inertial Measurement Units: MEMS-based 6-9 axis sensors (accelerometer, gyroscope, magnetometer). Acceleration range: ±16g. Gyro range: ±2000°/s. Update rate: 200-1000 Hz. Noise density: <0.005 °/s/√Hz.
• Force-Torque Sensors: Strain gauge or capacitive sensors at wrists and ankles. 6-axis measurement (Fx, Fy, Fz, Mx, My, Mz). Force range: ±500N. Torque range: ±50Nm. Resolution: 0.1N, 0.01Nm. Update rate: 1-10 kHz.
• Motor Current Sensors: Measure motor currents for torque estimation. Resolution: 16-bit. Sampling rate: 10-50 kHz. Accuracy: ±1% full scale.
• Battery Monitoring: Voltage, current, temperature, state of charge estimation. Integrated battery management system (BMS) with cell balancing.

Exteroceptive Sensors

Perceive the external environment—obstacles, people, objects, sounds, and conditions.

• RGB Cameras: Color image capture for object recognition, navigation, human interaction. Minimum: 2× stereo cameras, 1920×1080 resolution, 30 fps. Preferred: 60 fps, global shutter, HDR capability. FOV: 90-110° horizontal.
• Depth Cameras: Time-of-flight or structured light. Range: 0.5-8m. Depth resolution: 1mm at 1m distance. Frame rate: 30 fps minimum. VGA (640×480) or higher resolution.
• LiDAR: Laser-based 2D or 3D scanning. Point rate: 100,000-300,000 points/sec. Range: 0.1-30m. Accuracy: ±30mm. Angular resolution: 0.25-1.0°. Wavelength: 905nm or 1550nm (eye-safe).
• Tactile Sensors: Pressure-sensitive arrays on fingers and palms. Minimum: 4×4 array per finger. Pressure range: 0-500 kPa. Spatial resolution: 5-10mm. Update rate: 100 Hz.
• Microphones: Array of 4-8 microphones for sound localization and speech recognition. Frequency response: 20 Hz - 20 kHz. SNR: >60 dB. Spacing: 5-15cm for beamforming. Sample rate: 48 kHz.
• Proximity Sensors: Ultrasonic or infrared for close-range obstacle detection. Range: 0.05-3m. Update rate: 20-50 Hz. Coverage: 360° around robot body.

4.2 Vision System Architecture

4.2.1 Camera Configuration

WIA-ROB-019 specifies a standard multi-camera configuration:

Camera	Location	Purpose	Specifications
Stereo Head Cameras	Head/eyes	Primary perception, manipulation, HRI	1080p@30fps, 70mm baseline, calibrated
Wide-Angle Head Camera	Head/forehead	Peripheral awareness, navigation	720p@30fps, 120° FOV, fisheye
Wrist Cameras (optional)	Wrists	Fine manipulation, grasp verification	VGA@60fps, 90° FOV
Foot Cameras (optional)	Ankles	Foot placement verification, terrain	VGA@30fps, downward facing

4.2.2 Computer Vision Pipeline

The vision processing pipeline transforms raw images into actionable information:

Stage 1: Image Preprocessing (5-20ms)

• Distortion correction (lens calibration)
• Brightness/contrast adjustment (histogram equalization)
• Noise reduction (bilateral filtering)
• Color space conversion (RGB to HSV, LAB)

Stage 2: Feature Detection and Extraction (10-50ms)

• Traditional Features: SIFT, SURF, ORB for matching and tracking
• Edge Detection: Canny, Sobel for boundary identification
• Deep Features: CNN-based feature extractors (ResNet, EfficientNet)

Stage 3: Object Detection and Recognition (50-200ms)

• Generic Object Detection: YOLOv8, Faster R-CNN, EfficientDet. Detects 80+ object classes. Accuracy: >60% mAP. Processing: 30-60 fps on edge GPU.
• Instance Segmentation: Mask R-CNN, YOLACT for pixel-level object masks. Critical for grasp planning. Processing: 10-30 fps.
• Specialized Detectors: Face detection (MTCNN, RetinaFace), hand detection, pose estimation (OpenPose, MediaPipe).

Stage 4: Scene Understanding (100-500ms)

• Semantic Segmentation: DeepLab, SegFormer for pixel-wise scene labeling (floor, wall, furniture, etc.). Resolution: 512×512 or higher. Update: 5-15 fps.
• Depth Estimation: Stereo matching, monocular depth (DPT, MiDaS). Accuracy: ±5% at 1-5m range.
• Scene Graphs: Relationships between objects, spatial reasoning.
• Localization: Visual SLAM (ORB-SLAM3, Kimera-VIO) for mapping and localization.

4.3 LiDAR Processing

4.3.1 Point Cloud Processing

LiDAR generates 3D point clouds requiring sophisticated processing:

• Filtering: Remove outliers (statistical, radius), downsample (voxel grid) to reduce density while preserving structure. Typical reduction: 10× to 100×.
• Segmentation: Separate ground plane, walls, obstacles, objects. RANSAC for planar surfaces. Euclidean clustering for object segmentation.
• Registration: Align point clouds from multiple scans or viewpoints. ICP (Iterative Closest Point), NDT (Normal Distributions Transform). Accuracy: <10mm at 5m.
• Mapping: Build 3D occupancy grids or octomap representations. Resolution: 5-20cm voxels. Update rate: 1-10 Hz.

4.3.2 Obstacle Detection

Critical for safe navigation:

• Detect obstacles within planned path (0.5-5m lookahead)
• Classify obstacle type (static, dynamic, person, object)
• Estimate obstacle motion for dynamic avoidance
• Identify traversable vs non-traversable regions
• Detect stairs, slopes, dropoffs, narrow passages

Processing latency: <100ms from sensor data to obstacle representation. False positive rate: <5%. False negative rate: <0.1% (safety critical).

4.4 Audio Processing and Speech Recognition

4.4.1 Sound Source Localization

Microphone arrays enable determining sound direction:

• Method: Time Difference of Arrival (TDOA), beamforming, GCC-PHAT
• Accuracy: ±10° azimuth, ±20° elevation in quiet environments
• Range: 1-10 meters effective
• Latency: <50ms
• Multi-Source: Track up to 3 simultaneous speakers

4.4.2 Speech Recognition Pipeline

Converting speech to text and understanding intent:

Stage 1: Voice Activity Detection (10-30ms)

Detect when speech is present vs silence/noise. WebRTC VAD, energy-based methods, or deep learning (CRNN).

Stage 2: Speech Enhancement (20-100ms)

Noise reduction, echo cancellation, dereverberation. Beamforming, spectral subtraction, deep learning denoisers (SEGAN, WaveNet-based).

Stage 3: Automatic Speech Recognition (100-500ms)

Transcribe speech to text. WIA-ROB-019 requires multi-language support:

• Engine: Whisper, Conformer, or equivalent. Accuracy: >95% WER for clear speech
• Languages: Minimum English + local language. Recommended: 10+ major languages
• Vocabulary: 50,000+ words, expandable for domain-specific terms
• Mode: Online (streaming) with <500ms latency, offline for batch processing

Stage 4: Natural Language Understanding (100-1000ms)

Extract intent, entities, and semantics. Integration with WIA-INTENT standard. Methods: BERT, GPT-based models, rule-based systems for safety-critical commands.

4.5 Tactile Sensing and Contact Detection

4.5.1 Tactile Array Technologies

Multiple technologies for tactile sensing:

• Resistive: Force-sensitive resistors (FSR). Simple, low cost, moderate accuracy. Pressure range: 0-10 psi. Hysteresis: 5-10%.
• Capacitive: Measure capacitance changes under deformation. Better accuracy, higher cost. Resolution: 0.01-0.1 N. Response time: <5ms.
• Piezoelectric: Generate voltage under pressure. Fast response, measure dynamic forces. Less suitable for static force measurement.
• Optical: Camera-based systems measuring surface deformation. High resolution (sub-mm), full-field measurement. Complexity and cost higher.

4.5.2 Contact State Estimation

Determine when and where robot is in contact with environment:

• Binary Contact: Simple threshold detection. Latency: <1ms. Used for collision detection, emergency stops.
• Force Magnitude: Measure contact force. Used for grasp force control, surface pressing tasks. Accuracy: ±5% full scale.
• Force Direction: 3D force vector. Requires multi-axis sensors or tactile arrays. Enables sophisticated manipulation and assembly.
• Contact Geometry: Shape and area of contact region. From high-resolution tactile arrays. Aids grasp stability analysis.
• Slip Detection: Detect object sliding in grasp. High-frequency tactile sensing (>100 Hz) or specialized slip sensors. Enables reactive grasping.

4.6 State Estimation and Sensor Fusion

4.6.1 Floating Base State Estimation

Humanoid robots have no fixed base—their position and orientation must be estimated from sensors. WIA-ROB-019 requires robust state estimation providing:

• 6-DOF base pose (position and orientation) at >200 Hz
• Base linear and angular velocities
• All joint positions and velocities
• Contact states (which feet/hands on ground)
• Accuracy: ±10mm position, ±0.5° orientation (indoors), degraded in GPS-denied outdoors

Sensor Fusion Approaches

Extended Kalman Filter (EKF):

• Fuses IMU, joint encoders, contact sensors, visual odometry
• Real-time operation (>200 Hz)
• Handles sensor noise and bias
• Assumes Gaussian noise and local linearity

Unscented Kalman Filter (UKF):

• Better handling of nonlinear dynamics
• Similar computational cost to EKF
• Improved accuracy for highly nonlinear motion

Invariant EKF (InEKF):

• Exploits Lie group structure of SE(3)
• Better consistency than standard EKF
• Gaining adoption in robotics research

Particle Filter:

• Non-parametric, handles multi-modal distributions
• More computationally expensive (10-100× slower)
• Used when initial pose unknown or large uncertainties

4.6.2 Localization and Mapping

For navigation in larger environments, robots must localize within maps:

Visual SLAM (Simultaneous Localization and Mapping)

• ORB-SLAM3: Feature-based visual-inertial SLAM. Accurate, robust to textureless scenes. Map re-use capability. Processing: 30 fps on edge GPU.
• LSD-SLAM: Direct method, dense or semi-dense reconstruction. Better in low-texture environments. Higher computational cost.
• Kimera: Real-time metric-semantic SLAM with object-level understanding. Research-grade, advancing toward practical deployment.

LiDAR SLAM

• LOAM (Lidar Odometry and Mapping): Feature-based LiDAR SLAM. Real-time operation, accurate trajectory estimation.
• KISS-ICP: Fast ICP-based odometry. Lightweight, suitable for embedded systems.
• Cartographer: Google's graph-based SLAM. Handles loop closures well, produces globally consistent maps.

Multi-Modal Fusion

Combining visual and LiDAR SLAM provides robustness:

• LiDAR robust to lighting changes, accurate metric measurements
• Vision provides rich appearance information, works at longer range
• Fusion handles failure modes of individual modalities
• WIA-ROB-019 recommends multi-modal approach for Level 3 systems

4.7 Perception for Human-Robot Interaction

4.7.1 Person Detection and Tracking

Critical for safe operation near humans:

• Detection: Deep learning person detectors (YOLOv8-pose, DETR). Accuracy: >95% recall. Processing: 15-60 fps.
• Tracking: Multi-object tracking (SORT, DeepSORT, ByteTrack). Maintain person identities across frames. Handle occlusions, re-identification.
• Distance Estimation: From stereo or depth camera. Accuracy: ±5% at 1-5m range. Safety-critical—conservative estimates preferred.
• Proximity Zones: Define zones around robot (immediate: 0-0.5m, near: 0.5-1.5m, awareness: 1.5-5m). Adjust robot behavior based on zone occupancy.

4.7.2 Gesture Recognition

Enable non-verbal communication:

• Hand Gestures: Wave, point, thumbs up/down, stop sign. MediaPipe Hands, hand pose estimation networks. Accuracy: >85%.
• Body Gestures: Beckoning, pushing away, nodding, shaking head. Skeleton-based action recognition (ST-GCN, PoseConv3D).
• Vocabulary: Minimum 10 gestures, expandable for specific applications.
• Latency: <200ms from gesture start to recognition.
• Robustness: Handle partial occlusions, varying speeds, individual differences.

4.7.3 Facial Expression and Emotion Recognition

Understand human emotional states:

• Face Detection: RetinaFace, MTCNN. >99% detection rate for frontal faces.
• Landmark Detection: 68-point or denser facial landmarks (eyes, nose, mouth, eyebrows). Accuracy: ±2 pixels.
• Expression Classification: Recognize 7 basic emotions (happy, sad, angry, surprised, fearful, disgusted, neutral). Deep learning classifiers. Accuracy: 60-80% (varies by individual and culture).
• Gaze Estimation: Determine where person is looking. Accuracy: ±5° in favorable conditions. Important for joint attention and social cues.

Note: Emotion recognition has limitations and ethical considerations. WIA-ROB-019 requires:

• Explicit user consent before emotion analysis
• Clear communication when emotion recognition active
• No use for deception or manipulation
• Awareness of cultural differences in emotional expression
• Fallback to neutral interaction if emotion uncertain

4.8 Environmental Sensing

4.8.1 Temperature, Humidity, Air Quality

Monitor environmental conditions:

• Temperature Sensors: Digital sensors (SHT3x, BME280). Range: -40°C to +85°C. Accuracy: ±0.3°C. Used for thermal management and environment monitoring.
• Humidity Sensors: Capacitive RH sensors. Range: 0-100% RH. Accuracy: ±3% RH.
• Air Quality: VOC (volatile organic compound) sensors, particulate matter (PM2.5) sensors. Detect smoke, gas leaks, poor air quality. Alert humans if hazardous conditions detected.

4.8.2 Light Level Sensing

Adapt camera settings and behaviors:

• Ambient light sensors (lux meters) for automatic camera exposure
• UV sensors for outdoor operation warnings
• Enable robot to request better lighting if vision performance degraded

4.9 Sensor Calibration and Maintenance

4.9.1 Initial Calibration

All sensors must be calibrated before deployment:

• Camera Calibration: Intrinsic parameters (focal length, principal point, distortion) and extrinsic (camera poses relative to robot). Use checkerboard or AprilTag targets. Re-calibrate if cameras moved or adjusted.
• IMU Calibration: Accelerometer and gyroscope biases, scale factors, cross-axis errors. Temperature compensation for bias drift. Field calibration procedures for magnetometer (if used).
• Force-Torque Sensors: Zero-offset calibration (unloaded), gain calibration with known loads, temperature compensation.
• LiDAR-Camera Registration: Estimate transform between LiDAR and camera frames for sensor fusion. Accuracy: <5mm, <1° required.

4.9.2 Online Calibration and Health Monitoring

Sensors can drift or fail during operation:

• IMU Bias Estimation: Continuous online estimation during stationary periods. Update bias models.
• Sensor Fault Detection: Monitor for outliers, frozen readings, out-of-range values. Automatic switching to redundant sensors if available.
• Performance Monitoring: Track vision algorithm accuracy, localization drift, detection rates. Alert operator if degradation detected.
• Preventive Maintenance: Schedule regular re-calibration (cameras: 6 months, F/T sensors: 3 months, IMUs: 12 months).

4.10 Chapter Summary

This chapter examined the sensor systems and perception algorithms enabling humanoid robots to understand their environment and themselves. We covered proprioceptive sensors for self-state estimation, exteroceptive sensors for environmental perception, vision processing pipelines, LiDAR processing, audio/speech systems, tactile sensing, state estimation through sensor fusion, and perception for human interaction.

Robust perception is foundational to intelligent robot behavior. The specifications in WIA-ROB-019 ensure robots have sufficient sensory capabilities to operate safely and effectively in real-world conditions while maintaining consistent performance across platforms.

In Chapter 5, we'll explore how robots use this perceptual information to interact naturally with humans through speech, gestures, and social protocols.