Input Video Novel View 1 Novel View 2 Novel View 3
Reconstructing dynamic objects from monocular videos is a severely underconstrained and challenging problem, and recent work has approached it in various directions. However, owing to the ill-posed nature of this problem, there has been no solution that can provide consistent, highquality novel views from camera positions that are significantly different from the training views. In this work, we introduce Neural Parametric Gaussians (NPGs) to take on this challenge by imposing a two-stage approach: first, we fit a low-rank neural deformation model, which then is used as regularization for non-rigid reconstruction in the second stage. The first stage learns the object’s deformations such that it preserves consistency in novel views. The second stage obtains high reconstruction quality by optimizing 3D Gaussians that are driven by the coarse model. To this end, we introduce a local 3D Gaussian representation, where temporally shared Gaussians are anchored in and deformed by local oriented volumes. The resulting combined model can be rendered as radiance fields, resulting in high-quality photo-realistic reconstructions of the non-rigidly deforming objects, maintaining 3D consistency across novel views. We demonstrate that NPGs achieve superior results compared to previous works, especially in challenging scenarios with few multi-view cues.
Overview of our method. We present a two-stage method. In stage 1 (left), we learn a coarse point model, which is parameterized through low-rank coefficients from an MLP. In stage 2 (right), we optimize 3D Gaussians in local volumes, defined by the point sets. The figure distinguishes between parts that are shared over time (), individual for each time step (), and fixed-function (). MLP weights θ, Gaussian interpolation weights w, scales S, rotations R and harmonic coefficients h are shared over time and the deformation is purely modeled by the low-rank coefficients αi, leading to a different coarse point model for each frame.
@article{das2023neural,
title={Neural parametric gaussians for monocular non-rigid object reconstruction},
author={Das, Devikalyan and Wewer, Christopher and Yunus, Raza and Ilg, Eddy and Lenssen, Jan Eric},
journal={arXiv preprint arXiv:2312.01196},
year={2023}
}The website template is borrowed from Nerfies.