MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusion

ICCV 2025

Zihan Wang, Jeff Tan, Tarasha Khurana*, Neehar Peri*, Deva Ramanan
Carnegie Mellon University
* Equal contribution.
Paper (CVF) Video Code Data (ExoRecon)

Overview

MonoFusion reconstructs dynamic 4D scenes from sparse-view videos captured by four static cameras. It aligns monocular geometry and motion estimates across views and time, using depth priors and semantic features to initialize reconstruction. The paper evaluates novel-view synthesis on Panoptic Studio and ExoRecon, a subset of Ego-Exo4D. The input is synchronized multi-camera video with known camera parameters; monocular fusion describes the reconstruction strategy.


Given sparse-view videos of dynamic scenes, our approach reconstructs 3D geometry and motion, enabling extreme novel view synthesis, 3D tracking, and feature distillation. Our sparse-view (4-camera) setup strikes a balance between ill-posed reconstructions from casual monocular captures and well-constrained reconstructions from dense multi-view studio captures.

4D Scene Reconstruction that Supports Free-View Synthesis


Examples from ExoRecon, a curated six-scene subset of Ego-Exo4D covering diverse human activities. Training views are omitted from the displays below.

Bike Repair

Ego View Synthesis (Follow the Dance!)


With 4 camera set-up, we can enable ego view synthesis at anywhere in the scene, boosting possible embodied applications! [Left: gt provided by EgoExo4D, Right: our synthesised view] (Different colour is led by different camera sensors, foreground in our reconstruction has been removed)

More Results!


HealthCare - CPR

Music - Piano

Cooking - Scramble Egg

Sports - Football

Panoptic - Baseball


Abstract

We address the problem of dynamic scene reconstruction from sparse-view videos. Prior work often requires dense multi-view captures with hundreds of calibrated cameras (e.g. Panoptic Studio) - such multi-view setups are prohibitively expensive to build and cannot capture diverse scenes in-the-wild. In contrast, we aim to reconstruct dynamic human behaviors, such as repairing a bike or dancing, from a small set of sparse-view cameras with complete scene coverage (e.g. four equidistant inward-facing static cameras). We find that dense multi-view reconstruction methods struggle to adapt to this sparse-view setup due to limited overlap between viewpoints. To address these limitations, we carefully align independent monocular reconstructions of each camera to produce time- and view-consistent dynamic scene reconstructions. Extensive experiments on PanopticStudio and Ego-Exo4D demonstrate that our method achieves higher quality reconstructions than prior art, particularly when rendering novel views.

Contribution

  • We repurpose Ego-Exo4D for sparse-view reconstruction and highlight the challenge of reconstructing skilled human behaviors in dynamic environments.
  • We demonstrate that monocular reconstruction methods can be extended to the sparse-view setting by carefully incorporating monocular depth and foundational priors.
  • We extensively ablate our design choices and show that we achieve state-of-the-art performance on challenging sequences from Ego-Exo4D.

    • Method Overview


    Structure

    As dynamic scene reconstruction from sparse views is extremely challenging, we present two key insights to initialize plausible geometry and motion:

  • Initializing consistent scene geometry via confidence-aware spatio-temporal alignment
  • Initializing motion trajectories by clustering per-point 3D semantic features distilled from 2D foundation models

    • Experiments


    Concrete steps to validate our points:

    • Perfect training views (prove correct implementation)
    • Perfect near-novel (5°) view synthesis (illustrate that the method works, like other approaches)
    • Great extreme-novel (45°) view synthesis (demonstrate free-viewpoint rendering, no one did it before)
    • Held-out camera evaluation (90°) (use 3 out of 4 cameras for training and leave 1 out for qualitative results)

    Selected Qualitative Results

    45° Novel View Synthesis Comparison (1)
    Structure
    45° Novel View Synthesis Comparison (2)
    Structure
    45° Novel View Synthesis Comparison (3)
    Structure

    Existing monocular methods and their extension to multi-view produce poor results rendered from a drastically different novel view. MV-SOM improves upon SOM in 45° novel-view synthesis. Our method's careful point cloud initialization and feature-based motion bases further improve on MV-SOM.

    Input and evaluation scope

    The experiments use synchronized sparse-view videos with known camera parameters. The primary setup uses four static cameras; a separate held-out-camera setting trains on three views. Evaluation includes eight Panoptic Studio scenes and six ExoRecon scenes, with near-view and 45-degree novel-view comparisons. See Section 4 for the exact split, baselines and metrics.

    Limitations. Reconstruction depends on foreground masks and monocular priors. Missed body parts and long occlusions can cause mask or tracking failures. See the paper's limitations discussion.

     /  Download

    Acknowledgements

    Zihan Wang is currently supported by funding from Bosch Research. Zihan would like to thank all the coauthor for discussion and paper writing. Beyond that, Jeff for deep discussion about this project, Tarasha for debugging advice, Neehar for high-level insight, and Prof. Deva Ramanan for insightful guidance and advice. Outside of the author list, we would like to thank Nikhil Keetha and Jay Karhade for their great suggestions.