One ordinary camera, a full 3D scene, and objects placed inside it
A team of three students built a 3D perception pipeline from a single RGB camera: camera tracking, 3D reconstruction and object detection, each checked against ground truth. Their camera estimate landed within 2% of the true focal length, and the closest car in their test scene was placed 0.36 m from its true position.
Robots and vehicles need to know where they are and what is around them, often with only a camera. This team built a perception pipeline from a single RGB camera. COLMAP estimated the camera's internal parameters in an unknown environment, ORB-SLAM3 tracked the camera in real time and built a sparse map, Gaussian Splatting produced a dense 3D scene, and YOLO detected objects and placed them in 3D. On the TUM RGB-D benchmark, the estimated focal length was within 2% of the calibrated value. In a test scene, the averaged position estimate for the closest car landed 0.36 m from its true center; for cars further away the error rose to about 1 m. The team also measured how longer Gaussian Splatting training improved image quality at the cost of time, from about 5 minutes to 36 minutes.
Keywords · robot perception · monocular SLAM · ORB-SLAM3 · COLMAP · Gaussian Splatting · YOLO
Who did this work
- Concentration focus
- AI + Quantum & Cybersecurity
- Concentration focus
- AI + Physical Intelligence
- Concentration focus
- AI + Physical Intelligence
Students are not named. Year of study is as of fall 2026. Direction and goal come from each student's own program plan.
1Why one camera is hard
Depth sensors and lidar make 3D perception easier, but they add cost and weight. A single camera is cheap and everywhere, but it gives no direct depth, and its own settings may be unknown in a new environment.
The team asked how far a single ordinary camera could go: estimate its own parameters, track where it is in real time, rebuild the scene in detail, and place detected objects inside that scene, with each step measured.
2Three modules, each measured on its own
- Estimated the camera. Used COLMAP to recover the camera's focal length and center without a calibration target, and compared them to the calibrated values in the TUM RGB-D benchmark.
- Tracked in real time. Explained why COLMAP is too slow for live use (it matches unordered images with heavy features) and switched to ORB-SLAM3, which uses lightweight features and matches only nearby frames.
- Rebuilt the scene densely. Fed the sparse map and images into 3D Gaussian Splatting and measured quality against training time.
- Placed objects in 3D. Added YOLO object detection and connected it to the reconstructed scene in the SIBR viewer.
3Each stage checked against ground truth
| Camera parameter | Estimated by COLMAP | Calibrated value | Difference |
|---|---|---|---|
| Focal length, x | 526.76 | 517.31 | +1.8% |
| Focal length, y | 508.83 | 516.47 | −1.5% |
| Training iterations | Time | PSNR | Gaussian splats |
|---|---|---|---|
| 7,000 | ~5 min | 21.08 | 553,482 |
| 10,000 | ~8 min | 21.82 | 664,218 |
| 30,000 | ~36 min | 26.17 | 775,845 |
Object placement was checked against ground truth for three cars at increasing distance. For the closest car, individual detections were off by 0.50 m on average and the averaged estimate by 0.36 m. For the two cars further away, the averaged estimate was off by about 0.95–1.0 m, from far fewer valid detections (51 and 8, against 86).
4Selected slides
Selected slides from the team's final presentation. Student names are not shown.
5What this does not show
- Results come from a benchmark sequence and a few scenes the team recorded, not a deployed robot.
- The slides do not report end-to-end runs on a robot.
- Object placement was checked on three cars in one scene, and error grew from 0.36 m to about 1 m with distance. The team lists a combined real-time system as future work.
6What this work shows the team can do
Researcher in Robot Perception and Manipulation
CEO of ROBOX and advisor at XYZ Robotics. MIT EECS PhD. Best Paper Finalist at IROS and ICRA. First place in the Amazon Robotics Challenge stowing task (2017).
The team built and tested the pipeline and presented it. The Project Lead guided the technical direction and reviewed the work.
Robotics · Computer vision · Autonomous systems
A working perception pipeline with measured stages works well as a portfolio piece.
7What the Project Lead observed
After the project, the Project Lead wrote an evaluation of each student and placed each one on a four-level scale. This section shows the highest level reached on the team. The notes below describe the team's project. Individual ratings are not published.
Independent Problem Solving
- 1Foundational Application
- 2Applied Reasoning
- 3Independent Problem Solving
- 4System-Level Insight
Level 3 of 4: evaluates alternatives, adapts the approach based on evidence or constraints, and works with meaningful independence.
- Framing the problem
- Analyzed why COLMAP could not run in real time and moved to ORB-SLAM, which relies on lightweight ORB features, limits matching to nearby frames, and uses a constant-motion model.
- Decisions and trade-offs
- Combined sparse mapping with dense Gaussian Splatting reconstruction, studying the neural methods behind high-quality results.
- Evidence and testing
- Integrated YOLO object detection so the reconstructed map carries meaning, and connected the Gaussian Splatting pipeline to the SIBR viewer and its detection module.
- Teamwork
- One student led the team and organized the overall effort throughout the project.
Source: the Project Lead's written evaluations, November 2025.
References
- Campos, C. et al. (2021). ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM. IEEE Transactions on Robotics, 37(6).
- Schönberger, J. L. & Frahm, J.-M. (2016). Structure-from-motion revisited. CVPR.
- Kerbl, B. et al. (2023). 3D Gaussian Splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4).
- Sturm, J. et al. (2012). A benchmark for the evaluation of RGB-D SLAM systems. IROS.
Join the AI and Robotics for Mobile Robot Manipulation – Boston Dynamics Project
This outcome came out of this project. Work with a robotics researcher on real perception problems, and leave with a system you can show.
