Classic stereo matching or a neural network: which gives a robot better depth?
Luís Felipe Rodrigues Dutra and four teammates built a full stereo pipeline with an iPhone 15, from camera calibration to 3D point clouds, and compared OpenCV's StereoSGBM with deep-learning disparity. On their tabletop scenes, the network gave denser depth in low-texture areas, while SGBM with geometric constraints gave more reliable object depth and was easier to interpret.
Stereo vision lets a robot estimate depth from two images, but depth maps often have holes or noise, especially on low-texture surfaces. This team built a complete pipeline for tabletop scenes: they calibrated an iPhone 15 with 30 checkerboard photos at varied angles and distances, estimated disparity with OpenCV's Semi-Global Block Matching (StereoSGBM) and with a deep-learning model, converted disparity into depth and 3D point clouds, and filled missing depth using image and planar cues. The deep-learning model produced denser disparity, particularly in low-texture areas, and handled lighting changes better. SGBM combined with geometric constraints produced more reliable and interpretable object depth with less compute. The team concluded that a hybrid of the two is the practical choice for robots.
Keywords · stereo vision · StereoSGBM · disparity estimation · camera calibration · point clouds · depth completion
Who did this work
- On this project
- Drove the technical work and shaped the comparison between geometric and deep-learning depth.
Students named here agreed to be featured. 4 other team members are not named. Role notes come from the Project Lead's feedback.
1Why robots still struggle with depth
Two cameras a known distance apart can measure depth, the same way two eyes do. The hard part is matching each pixel in one image to the right pixel in the other.
Classic matching works well on textured surfaces but leaves holes on plain ones. Neural networks fill those holes, but cost more compute and are harder to inspect. The team built both and compared them on the same scenes.
2One pipeline, two ways to estimate disparity
- Captured clean images. Used an iPhone 15 and a rigid matte checkerboard to avoid glare and bending.
- Calibrated. Took 30 photos at varied angles and distances to estimate intrinsic parameters and distortion.
- Estimated disparity two ways. Tuned StereoSGBM (block size 3–11, smoothness penalties P1 and P2) and ran a deep-learning disparity model on the same pairs.
- Reconstructed. Undistorted and rectified, then converted disparity to depth and to 3D point clouds.
- Completed depth. Filled missing depth using image cues and plane fitting with RANSAC.
3Denser versus more reliable
| Criterion | StereoSGBM + geometry | Deep learning |
|---|---|---|
| Depth density | Gaps on low-texture surfaces | Denser, including low-texture areas |
| Object depth on tabletop scenes | More reliable | Less reliable |
| Interpretability | High: parameters map to clear effects | Lower |
| Compute | Lower | Higher |
4The poster
The team's final poster. Names of teammates who are not featured are blurred.
5What this does not show
- No numerical depth-error comparison against ground truth is reported.
- Tests used the team's own tabletop scenes and one phone camera.
- The method was not run on a robot; real-time deployment is listed as future work.
6What this work shows
Researcher in Computer Vision and Robotics
Principal Computer Vision Engineer with 15 years in robotics and machine vision, across consumer robotics, healthcare, factory automation and embedded systems.
The team built the pipeline and ran the comparison. The Project Lead guided the work and reviewed it.
Robotics · Computer vision · Autonomous systems
A pipeline that runs end to end, with an honest comparison of two methods, reads well to engineering reviewers.
7What the Project Lead observed
The Project Lead wrote feedback on the project and rated the demonstrated capability on a four-level scale.
System-Level Insight
- 1Foundational Application
- 2Applied Reasoning
- 3Independent Problem Solving
- 4System-Level Insight
Level 4 of 4: anticipates broader implications, risks or downstream effects, and connects technical decisions to real-world impact.
- Framing the problem
- Showed a clear understanding of stereo fundamentals and real constraints: calibration sensitivity, lighting variation and compute cost.
- Decisions and trade-offs
- Compared approaches on accuracy, robustness and resource limits to reach system-level design choices.
- Evidence and testing
- Tested under varying data quality and conditions, and identified failure modes through structured experiments.
- Contribution
- Luís provided technical leadership and shaped a well-structured comparison; the Academic Coordinator noted his collaborative leadership and clear communication.
Source: the Project Lead's and Academic Team's written feedback, 2025.
References
- Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. IEEE TPAMI, 30(2), 328–341.
- Brown, M. & Hartley, R. (2004). Camera calibration with radial distortion and decentering distortion. IEEE TPAMI, 26(7).
- Mayer, N. et al. (2016). A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. CVPR.
Join the Computer Vision and Robotics – Amazon Robotics Project
Work with a robotics vision engineer on real perception problems and leave with a pipeline you can run.
