Chapter 4
Stereo vision systems mimic human binocular vision using two or more cameras to perceive depth through parallax. As a passive technique requiring no active illumination, stereo vision excels in outdoor environments and applications where adding light sources is impractical.
Stereo vision operates on the principle of triangulation using disparity - the difference in position of the same point when viewed from two different perspectives. Human vision uses this exact mechanism: our two eyes separated by approximately 6.5cm baseline provide slightly different views of the world, which our brain processes to perceive depth.
In a stereo camera system, two cameras are separated by a known baseline distance. When both cameras observe the same point in 3D space, that point appears at different pixel coordinates in each camera's image. The horizontal displacement between these corresponding points is the disparity, which is inversely proportional to depth.
The mathematical relationship is elegantly simple: Depth = (Baseline × Focal Length) / Disparity. This means that to measure depth precisely, we must: accurately calibrate the baseline distance between cameras, know the cameras' focal lengths precisely, and determine disparity with sub-pixel accuracy.
The core challenge in stereo vision is the correspondence problem: given a pixel in the left image, find the corresponding pixel in the right image that represents the same physical point in the scene. This is computationally intensive, as each pixel in one image must potentially be compared with many pixels in the other image.
Block matching algorithms compare small rectangular regions (blocks) between left and right images to find correspondences. Sum of Absolute Differences (SAD) and Sum of Squared Differences (SSD) are common similarity metrics. These methods are computationally efficient and well-suited for hardware implementation, but they assume that corresponding regions have similar intensities and textures, which may not hold for specular surfaces, occlusions, or regions with repetitive patterns.
SGM represents a more sophisticated approach that balances accuracy and computational efficiency. Instead of making independent matching decisions for each pixel, SGM enforces smoothness constraints: neighboring pixels should have similar disparities unless there's an edge in the image. The algorithm aggregates matching costs along multiple 1D paths through the image, approximating a computationally expensive 2D global optimization with efficient 1D optimizations.
SGM has become the de facto standard for high-quality real-time stereo matching and is widely used in autonomous vehicle perception systems; hardware stereo processors such as the one in the Intel RealSense D400 series are commonly described as using SGM-class matching (Intel does not publish the exact algorithm). Modern implementations achieve dense disparity maps at VGA resolution and 30+ fps using GPU or FPGA acceleration.
Recent advances in deep learning have produced stereo matching networks that outperform classical algorithms, especially in challenging conditions like textureless regions, occlusions, and reflective surfaces. Networks like PSMNet, GCNet, and GANet learn to extract features and compute matching costs from large datasets of stereo images with ground truth depth.
The advantage of learned approaches is their ability to handle challenging scenarios that violate traditional assumptions. For instance, they can correctly match regions with different apparent intensities due to surface properties or lighting variations. The disadvantage is computational cost - deep networks require significant GPU resources for real-time operation.
Accurate calibration is absolutely critical for stereo vision systems. Calibration determines both intrinsic parameters (focal length, principal point, lens distortion for each camera) and extrinsic parameters (relative position and orientation between cameras).
Each camera is calibrated individually using a known pattern, typically a checkerboard. By capturing multiple images of the pattern at different positions and orientations, the calibration algorithm can solve for the camera's internal parameters. Modern calibration uses the Zhang method or similar techniques, achieving sub-pixel accuracy for distortion correction.
After individual camera calibration, stereo calibration determines the 3D transformation (rotation and translation) between the two cameras. This establishes the baseline - the most critical parameter for depth accuracy. A 1% error in baseline translates directly to 1% error in depth measurements.
The calibration process produces rectification parameters that transform the images so corresponding points lie on the same horizontal scan line (epipolar rectification). This reduces the 2D correspondence search to a 1D search along scan lines, dramatically improving computational efficiency and reliability.
Intel's RealSense D400 series is a representative commercial stereo vision product, widely used in robotics, drones, retail analytics, and industrial automation. Understanding its architecture illustrates practical stereo system design.
The D435, the most popular D400 model, integrates:
| Parameter | D435 Value | Notes |
|---|---|---|
| Depth Range | 0.3 - 10m | Optimal accuracy 0.5-3m |
| Depth Accuracy | <2% at 2m (Intel specification) | Within about 40mm at 2m distance |
| Resolution | 1280x720 | Lower resolutions selectable, e.g. 848x480, 640x480 |
| Frame Rate | Up to 90 fps | At 848x480 or lower; 30 fps at 1280x720 |
| Baseline | 50mm | Fixed, factory calibrated |
| Field of View | 87° × 58° | Diagonal 95° |
| Power | USB bus powered | USB 3 connection recommended |
Pure passive stereo relies entirely on natural scene texture for matching. While this works well for outdoor scenes with rich visual detail, it fails on textureless surfaces like painted walls or uniform materials. Active stereo projects a pattern to create artificial texture.
Different systems use different pattern projection approaches:
While two cameras define the minimum stereo system, multi-camera configurations provide advantages:
Three cameras in a linear arrangement enable dual baseline operation: a short baseline (cameras 1-2) provides dense matching for close objects, while a long baseline (cameras 1-3) extends maximum range and improves accuracy at distance. This configuration is popular in automotive applications where the system must handle both near-field obstacle detection and far-field object tracking.
Multiple stereo pairs arranged around a platform create 360-degree depth sensing. This is critical for autonomous vehicles and mobile robots that need complete environmental awareness. Challenges include calibrating many cameras, managing data bandwidth, and fusing overlapping depth maps.
| Algorithm | Approach | Accuracy | Speed | Best For |
|---|---|---|---|---|
| Block Matching (SAD/SSD) | Local window comparison | Moderate | Very Fast | Real-time embedded systems |
| Semi-Global Matching (SGM) | 1D path aggregation | High | Fast | Automotive, robotics |
| Graph Cut | Global energy minimization | Very High | Slow | Offline 3D reconstruction |
| Deep Learning (PSMNet) | CNN-based feature matching | Very High | Moderate (GPU) | Challenging scenes, research |
Stereo vision is the dominant depth sensing technology for mobile robots and drones due to its passive operation (no laser power required), outdoor capability (insensitive to ambient light), and simultaneous color and depth capture. Applications include:
Many autonomous vehicles use stereo cameras as a low-cost complement to LiDAR. While LiDAR provides superior range and accuracy, stereo cameras offer color information for traffic sign recognition, lane marking detection, and semantic scene understanding. The Subaru EyeSight system uses stereo vision for collision avoidance, adaptive cruise control, and lane keeping assistance.
AR headsets combine stereo cameras and depth sensors to perceive space. Microsoft HoloLens 2, for example, uses several grayscale cameras for head tracking and a ToF depth camera for spatial mapping and hand tracking. This sensor combination enables the system to understand room geometry, detect surfaces for virtual object placement, and track hand gestures for interaction - all critical capabilities for immersive AR experiences.
The depth accuracy of stereo systems is governed by fundamental geometric relationships. For a given baseline (B) and focal length (f), the depth resolution (δz) depends on disparity measurement uncertainty (δd):
δz = (z² × δd) / (B × f)
This shows that depth error grows quadratically with distance. At 1 meter with 50mm baseline, 1280 pixel width and 87° horizontal FOV (focal length ≈674 pixels), and 0.1 pixel disparity uncertainty, depth resolution is approximately 3mm. At 5 meters, this degrades to 75mm - a 25× worse resolution despite only 5× more distance.
To improve depth resolution, systems can: increase baseline (but this reduces minimum range and increases occlusion issues), increase focal length (but this reduces field of view), increase sensor resolution (but this increases computation), or improve matching accuracy through better algorithms.