Integration of Multi-Sensor SLAM and Gaussian Splatting for Real-Time Photorealistic Scene Reconstruction

Master Thesis at ifp - Haibo Zhu

Haibo Zhu

Duration: 6 months
Completion: September 2026
Supervisor: Dr.-Ing. Wei Zhang
Examiner: apl. Prof. Dr.-Ing. Norbert Haala

Introduction

Photorealistic scene reconstruction with metric geometry is important for mobile mapping, robotic perception, augmented and virtual reality, and digital twins. For these applications, a useful map should combine geometric accuracy, photorealistic novel-view rendering, and computational efficiency by exploiting complementary camera texture and LiDAR geometry.

Tightly coupled Gaussian SLAM aims to reconstruct scenes while estimating motion online, but real-time computation budgets and continuously evolving poses can make it difficult to isolate mapping decisions from tracking effects. This thesis therefore adopts a post-SLAM setting: trajectories, calibration, timing, and observations are fixed before reconstruction, allowing the study to focus on how heterogeneous geometric evidence should guide persistent map construction and refinement.

Conventional point-cloud or surfel maps support localization and measurement but not continuous novel-view rendering. 3D Gaussian Splatting (3DGS) instead offers an explicit representation with efficient differentiable rendering. Yet, when heterogeneous geometric inputs are merged into a shared Gaussian representation, their measurement origin and spatial coverage can be obscured.

 

 

Methodology

The thesis proposes SAGE (Source-Aware Geometric Evidence), a two-stage framework that transforms fixed RGB images, poses, LiDAR observations, and learned depth into one persistent Gaussian map.

Figure 1: SAGE uses fixed-pose RGB and geometric observations for source-aware construction in Stage I, then refines continuous Gaussian parameters with fixed topology in Stage II.

SAGE distinguishes direct LiDAR Center observations, coverage-extending LiDAR Fused data, and SPNet depth for LiDAR-unsupported areas. Each Gaussian retains its source, score, identity, and creation frame throughout the map lifecycle.

Stage I initializes the map only from LiDAR and proposes candidates where opacity is insufficient or rendered depth conflicts with evidence. Source-specific regions and priorities admit, deduplicate, colorize, and add suitable points. Each commit follows a grow-optimize-prune cycle. Optimization combines the current frame with replayed keyframes, while source-aware pruning controls weak support and population growth. In Stage II, topology is frozen: no Gaussian is inserted, removed, split, or relabelled. RGB, LiDAR depth, and surface-normal supervision refine only position, scale, rotation, color, and opacity.

Experiments and Results

SAGE is evaluated on Replica, TUM RGB-D, R3LIVE, and six Odin1 sequences. Table 1 compares SAGE with published baseline results compiled from Lang et al. (2025b, Table III) on the R3LIVE dataset (Lin and Zhang, 2022).

Table 1: R3LIVE rendering comparison. Each cell reports PSNR (dB) / SSIM / LPIPS. D, C, and P denote degenerate, campus, and park sequences.

SAGE has the highest PSNR on four of six sequences and leads all three metrics on D0 and D1. Because values follow each source's protocol rather than controlled reruns, they demonstrate competitiveness but not universal superiority.

On Odin1 Downtown1, source-aware growth and pruning raise PSNR by 1.305 dB and SSIM by 0.0197, lower LPIPS by 0.0263, and use 4.9% fewer Gaussians. Depth MAE rises by 0.0022 m, revealing the trade-off.

Figure 2: Source-aware growth and pruning move the operating point toward higher PSNR with a smaller population.

Stage II improves aggregate PSNR, SSIM, LPIPS, and depth MAE on all four tested scenes. PSNR improves on 67.6–75.7% of accepted frames and depth MAE on 80.3–92.0%, without changing Gaussian identities or provenance.

Figure 3: Stage-I and Stage-II error heatmaps and RGB renderings for four scenes.

On Replica, SAGE runs at 8.00 mapping FPS with an 11.8 MB mean checkpoint and 3.15 GB reported memory. Large outdoor maps reach 1.80–2.11 million Gaussians and 17.2–19.0 GB, showing that long-term growth remains unbounded.

Conclusion

SAGE separates population-changing construction from topology-frozen refinement. Across synthetic, RGB-D, outdoor, and self-recorded data, its source-aware lifecycle improves reconstruction without expanding the population during Stage II.

Current limitations include rule-based source reliability, weak modeling of reflections, and no hard memory bound. Future work should add calibrated uncertainty, view-dependent appearance, and resource-aware submaps or out-of-core storage.

References

  1. Lin, J., and Zhang, F. (2022). R3LIVE: A robust, real-time, RGB-colored, LiDAR-inertial-visual tightly-coupled state estimation and mapping package. ICRA, 10672–10678. DOI
  2. Matsuki, H., Murai, R., Kelly, P. H. J., and Davison, A. J. (2024). Gaussian Splatting SLAM. CVPR, 18039–18048. Paper
  3. Wang, H., Wang, J., and Agapito, L. (2023). Co-SLAM: Joint coordinate and sparse parametric encodings for neural real-time SLAM. CVPR, 13293–13302. Paper
  4. Lang, X., Li, L., Wu, C., et al. (2025a). Gaussian-LIC: Real-time photo-realistic SLAM with Gaussian Splatting and LiDAR-inertial-camera fusion. ICRA, 8500–8507. DOI
  5. Sun, L. C., Bhatt, N. P., Liu, J. C., et al. (2024). MM3DGS-SLAM: Multi-modal 3D Gaussian Splatting for SLAM using vision, depth, and inertial measurements. IROS, 10159–10166. arXiv
  6. Lang, X., Lv, J., Tang, K., et al. (2025b). Gaussian-LIC2: LiDAR-inertial-camera Gaussian Splatting SLAM. arXiv:2507.04004. arXiv

Ansprechpartner

Dieses Bild zeigtNorbert Haala

Norbert Haala

apl. Prof. Dr.-Ing.

Stellvertretender Institutsleiter

Zum Seitenanfang