Chapter 4

Stereo Vision and Passive Sensing

Stereo vision systems mimic human binocular vision using two or more cameras to perceive depth through parallax. As a passive technique requiring no active illumination, stereo vision excels in outdoor environments and applications where adding light sources is impractical.

Fundamental Principles

Stereo vision operates on the principle of triangulation using disparity - the difference in position of the same point when viewed from two different perspectives. Human vision uses this exact mechanism: our two eyes separated by approximately 6.5cm baseline provide slightly different views of the world, which our brain processes to perceive depth.

In a stereo camera system, two cameras are separated by a known baseline distance. When both cameras observe the same point in 3D space, that point appears at different pixel coordinates in each camera's image. The horizontal displacement between these corresponding points is the disparity, which is inversely proportional to depth.

The mathematical relationship is elegantly simple: Depth = (Baseline × Focal Length) / Disparity. This means that to measure depth precisely, we must: accurately calibrate the baseline distance between cameras, know the cameras' focal lengths precisely, and determine disparity with sub-pixel accuracy.

Stereo Matching Algorithms

The core challenge in stereo vision is the correspondence problem: given a pixel in the left image, find the corresponding pixel in the right image that represents the same physical point in the scene. This is computationally intensive, as each pixel in one image must potentially be compared with many pixels in the other image.

Block Matching Methods

Block matching algorithms compare small rectangular regions (blocks) between left and right images to find correspondences. Sum of Absolute Differences (SAD) and Sum of Squared Differences (SSD) are common similarity metrics. These methods are computationally efficient and well-suited for hardware implementation, but they assume that corresponding regions have similar intensities and textures, which may not hold for specular surfaces, occlusions, or regions with repetitive patterns.

Semi-Global Matching (SGM)

SGM represents a more sophisticated approach that balances accuracy and computational efficiency. Instead of making independent matching decisions for each pixel, SGM enforces smoothness constraints: neighboring pixels should have similar disparities unless there's an edge in the image. The algorithm aggregates matching costs along multiple 1D paths through the image, approximating a computationally expensive 2D global optimization with efficient 1D optimizations.

SGM has become the de facto standard for high-quality real-time stereo matching, used in Intel RealSense D400 series and many autonomous vehicle perception systems. Modern implementations achieve dense disparity maps at VGA resolution and 30+ fps using GPU or FPGA acceleration.

Deep Learning Approaches

Recent advances in deep learning have produced stereo matching networks that outperform classical algorithms, especially in challenging conditions like textureless regions, occlusions, and reflective surfaces. Networks like PSMNet, GCNet, and GANet learn to extract features and compute matching costs from large datasets of stereo images with ground truth depth.

The advantage of learned approaches is their ability to handle challenging scenarios that violate traditional assumptions. For instance, they can correctly match regions with different apparent intensities due to surface properties or lighting variations. The disadvantage is computational cost - deep networks require significant GPU resources for real-time operation.

Calibration

Accurate calibration is absolutely critical for stereo vision systems. Calibration determines both intrinsic parameters (focal length, principal point, lens distortion for each camera) and extrinsic parameters (relative position and orientation between cameras).

Intrinsic Calibration

Each camera is calibrated individually using a known pattern, typically a checkerboard. By capturing multiple images of the pattern at different positions and orientations, the calibration algorithm can solve for the camera's internal parameters. Modern calibration uses the Zhang method or similar techniques, achieving sub-pixel accuracy for distortion correction.

Stereo Calibration

After individual camera calibration, stereo calibration determines the 3D transformation (rotation and translation) between the two cameras. This establishes the baseline - the most critical parameter for depth accuracy. A 1% error in baseline translates directly to 1% error in depth measurements.

The calibration process produces rectification parameters that transform the images so corresponding points lie on the same horizontal scan line (epipolar rectification). This reduces the 2D correspondence search to a 1D search along scan lines, dramatically improving computational efficiency and reliability.

Intel RealSense D400 Series

Intel's RealSense D400 series represents one of the most successful commercial stereo vision products, with millions of units deployed in robotics, drones, retail analytics, and industrial automation. Understanding its architecture illustrates practical stereo system design.

Hardware Architecture

The D435, the most popular D400 model, integrates:

• Dual Global Shutter Cameras: Two 1280x720 global shutter cameras with 87° diagonal FOV, separated by 50mm baseline. Global shutter (all pixels exposed simultaneously) prevents motion artifacts that rolling shutter sensors exhibit with moving objects.
• Infrared Projector: A pattern projector that adds texture to otherwise featureless surfaces, improving stereo matching in textureless environments like white walls. This active stereo approach combines the robustness of passive stereo with the texture advantages of structured light.
• Vision Processor: An ASIC (Application Specific Integrated Circuit) that implements hardware-accelerated SGM for real-time depth computation. The processor generates depth maps at up to 90 fps at 1280x720 resolution.
• RGB Camera: A separate 1920x1080 color camera for texture mapping and traditional computer vision tasks, aligned with the stereo cameras' field of view.
• IMU: 6-axis inertial measurement unit (accelerometer + gyroscope) for motion tracking and visual-inertial odometry applications.

Performance Characteristics

Parameter	D435 Value	Notes
Depth Range	0.3 - 10m	Optimal accuracy 0.5-3m
Depth Accuracy	±2% @ 2m	±40mm error at 2m distance
Resolution	1280x720	Adjustable, down to 640x480
Frame Rate	Up to 90 fps	Resolution dependent
Baseline	50mm	Fixed, factory calibrated
Field of View	87° × 58°	Diagonal 95°
Power	~3W typical	USB bus powered

Active vs Passive Stereo

Pure passive stereo relies entirely on natural scene texture for matching. While this works well for outdoor scenes with rich visual detail, it fails on textureless surfaces like painted walls or uniform materials. Active stereo projects a pattern to create artificial texture.

Pattern Projection Strategies

Different systems use different pattern projection approaches:

• Random Dots: Intel RealSense projects a pseudo-random pattern of infrared dots, similar to structured light but optimized for stereo matching rather than triangulation. The dots add high- frequency texture that improves correlation matching.
• Structured Patterns: Some systems project line patterns or coded patterns that provide both stereo correspondence and structured light triangulation depth as complementary modalities.
• Adaptive Projection: Advanced systems analyze the scene and project patterns only in regions lacking natural texture, saving power and reducing interference.

Multi-Camera Systems

While two cameras define the minimum stereo system, multi-camera configurations provide advantages:

Triple Camera Systems

Three cameras in a linear arrangement enable dual baseline operation: a short baseline (cameras 1-2) provides dense matching for close objects, while a long baseline (cameras 1-3) extends maximum range and improves accuracy at distance. This configuration is popular in automotive applications where the system must handle both near-field obstacle detection and far-field object tracking.

Omnidirectional Stereo

Multiple stereo pairs arranged around a platform create 360-degree depth sensing. This is critical for autonomous vehicles and mobile robots that need complete environmental awareness. Challenges include calibrating many cameras, managing data bandwidth, and fusing overlapping depth maps.

Stereo Matching Algorithms Comparison

Algorithm	Approach	Accuracy	Speed	Best For
Block Matching (SAD/SSD)	Local window comparison	Moderate	Very Fast	Real-time embedded systems
Semi-Global Matching (SGM)	1D path aggregation	High	Fast	Automotive, robotics
Graph Cut	Global energy minimization	Very High	Slow	Offline 3D reconstruction
Deep Learning (PSMNet)	CNN-based feature matching	Very High	Moderate (GPU)	Challenging scenes, research

Applications

Robotics and Drones

Stereo vision is the dominant depth sensing technology for mobile robots and drones due to its passive operation (no laser power required), outdoor capability (insensitive to ambient light), and simultaneous color and depth capture. Applications include:

• Obstacle detection and avoidance for autonomous navigation
• 3D mapping and SLAM (Simultaneous Localization and Mapping)
• Object recognition and grasping in warehouse automation
• Precision landing for delivery drones
• Agricultural robots for crop monitoring and harvesting

Autonomous Vehicles

Many autonomous vehicles use stereo cameras as a low-cost complement to lidar. While lidar provides superior range and accuracy, stereo cameras offer color information for traffic sign recognition, lane marking detection, and semantic scene understanding. The Subaru EyeSight system, deployed in millions of vehicles, uses stereo vision for collision avoidance, adaptive cruise control, and lane keeping assistance.

Augmented Reality

AR headsets like Microsoft HoloLens use stereo depth sensing for spatial mapping and hand tracking. The stereo cameras enable the system to understand room geometry, detect surfaces for virtual object placement, and track hand gestures for interaction - all critical capabilities for immersive AR experiences.

Challenges and Limitations

Depth Accuracy Analysis

The depth accuracy of stereo systems is governed by fundamental geometric relationships. For a given baseline (B) and focal length (f), the depth resolution (δz) depends on disparity measurement uncertainty (δd):

δz = (z² × δd) / (B × f)

This shows that depth error grows quadratically with distance. At 1 meter with 50mm baseline, 1280 pixel width, and 0.1 pixel disparity uncertainty, depth resolution is approximately 3mm. At 5 meters, this degrades to 75mm - a 25× worse resolution despite only 5× more distance.

To improve depth resolution, systems can: increase baseline (but this reduces minimum range and increases occlusion issues), increase focal length (but this reduces field of view), increase sensor resolution (but this increases computation), or improve matching accuracy through better algorithms.

Future Directions

• Event-Based Stereo: Neuromorphic cameras that generate events on brightness changes enable ultra-low latency stereo matching and operation in high-speed scenarios.
• Learning-Based Refinement: Hybrid approaches using classical stereo for initial depth estimation and neural networks for refinement, occlusion filling, and confidence estimation.
• Multi-Modal Fusion: Combining stereo depth with other modalities (ToF, lidar, radar) to leverage complementary strengths.
• Self-Calibration: Algorithms that detect and compensate for calibration drift using visual odometry and scene consistency constraints.

Summary

Review Questions

1. What are the key advantages of standardizing 3D image sensor technologies? How does it benefit manufacturers, developers, and end users?
2. Compare and contrast Time-of-Flight (ToF) and structured light sensing technologies. What are the strengths and limitations of each approach?
3. Explain the four-phase architecture of WIA standards. Why is this progressive approach beneficial for adoption and evolution?
4. Describe the main sources of error in 3D depth sensing. How can systematic errors be distinguished from random errors, and what techniques address each type?
5. What role does calibration play in achieving accurate 3D measurements? Describe the difference between factory calibration and field verification.
6. How do point cloud processing techniques transform raw sensor data into useful information for applications? Give examples of common processing operations.
7. Explain how sensor fusion can improve 3D sensing performance. What are some examples of complementary sensor combinations?
8. What factors should be considered when integrating 3D sensors into real-time systems? Discuss latency, power consumption, and computational requirements.