Turn separate 3D scans into one space you can move through.
Toshihiro Koizumi and four teammates reconstructed scenes from 2,880 photos with 3D Gaussian Splatting, stitched separately trained scenes together with ICP, and built a viewer with a trajectory map, portals and region triggers. To fit training on limited GPU memory, they trained in chunks and reduced roughly 600,000 initial points to about 70,000.
Gaussian Splatting renders photorealistic 3D scenes in real time, but most tools show each reconstruction as an isolated, static scene. This team built a system for exploring several connected reconstructions as one environment. They ran COLMAP on 2,880 photos, trained the roughly 600,000 initial points in chunks to avoid running out of GPU memory, and reduced them to about 70,000 optimized points before global fine-tuning. Separately trained scenes were aligned with the Iterative Closest Point (ICP) algorithm. On top of the SIBR viewer they added a 2D trajectory map, region-based teleport prompts, live camera position, and the ability to place images and 3D models in the scene. The poster reports what still breaks: ICP did not stitch perfectly, noisy points had to be removed by hand, and memory limits forced lower resolution on large datasets.
Keywords · 3D Gaussian Splatting · ICP · COLMAP · interactive viewer · AR/VR · GPU memory
Who did this work
- On this project
- Led the interactive interface, the trajectory map and the virtual portals.
Students named here agreed to be featured. 4 other team members are not named. Role notes come from the Project Lead's feedback.
1Why one reconstruction is not enough
Gaussian Splatting builds a 3D scene from ordinary photos and renders it fast enough to move through in real time. That makes it attractive for AR and VR.
In practice, each reconstruction is a separate, static scene, and a large area is too big to train as one model on a normal GPU. The team asked how to combine several reconstructions into one space a person could actually explore.
2Reconstruct in pieces, then stitch and add interaction
- Reconstructed each scene. Ran SIFT feature matching and COLMAP on 2,880 photos, then trained Gaussian Splatting.
- Trained in chunks. Split the roughly 600,000 initial points into chunks to avoid GPU out-of-memory errors, reduced them to about 70,000 optimized points, then fine-tuned globally.
- Stitched scenes. Aligned overlapping point clouds from separate reconstructions with ICP so a viewer can move between them.
- Built the interface. Added a trajectory map, region-based teleport prompts, live camera position, and images and 3D models placed in the scene, rendered inside the CUDA splatting pass.
3What the system does, and what still breaks
| Feature | Status in the poster |
|---|---|
| Multi-scene stitching with ICP | Working, but not perfect; noisy points removed by hand |
| Trajectory map and live camera position | Working |
| Region-based teleport between scenes | Working |
| Images and 3D models placed in the scene | Working; general scene editing not finished |
| Large reconstructions at full resolution | Limited by GPU memory |
4The poster
The team's final poster. Names of teammates who are not featured are blurred.
5What this does not show
- The poster reports no image-quality metrics (such as PSNR or SSIM) for the reconstructions.
- Stitching quality depended on manual cleanup of point clouds.
- Results cover the team's own captured scenes; other environments may behave differently.
- The interactive features were demonstrated, not tested with users.
6What this work shows
Researcher in Computer Graphics and Deep Learning
Researcher at MIT CSAIL, Computational Design and Fabrication Group. Published at SIGGRAPH, CVPR and ACM Transactions on Graphics. Interned at NVIDIA, Meta Reality Labs and Google.
The team built the system and wrote the poster. The Project Lead guided the direction and ran regular progress checks.
Computer graphics · AR/VR · 3D vision
A working interactive viewer, with a clear list of what still breaks, is a strong portfolio demo.
7What the Project Lead observed
The Project Lead wrote feedback on the project and rated the demonstrated capability on a four-level scale.
System-Level Insight
- 1Foundational Application
- 2Applied Reasoning
- 3Independent Problem Solving
- 4System-Level Insight
Level 4 of 4: anticipates broader implications, risks or downstream effects, and connects technical decisions to real-world impact.
- Framing the problem
- Took Gaussian Splatting from static visualization toward interactive AR/VR use, and merged several large reconstructions into one navigable environment.
- Decisions and trade-offs
- Chose methods that balance visual quality, compute cost and interactivity under limited hardware.
- Execution
- Steady, disciplined progress: identified technical problems early and engineered practical fixes.
- Contribution
- Toshihiro led the interactive interface, trajectory visualization and virtual portals, adding to both robustness and usability.
Source: the Project Lead's and Academic Team's written feedback, 2025.
References
- Kerbl, B. et al. (2023). 3D Gaussian Splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4).
- Besl, P. J. & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE TPAMI, 14(2).
- Schönberger, J. L. & Frahm, J.-M. (2016). Structure-from-motion revisited. CVPR.
Join the Computer 3D Graphics and Deep Learning – NVIDIA Project
Work with a graphics researcher on 3D reconstruction and leave with a system you can show.
