PixelGaussian: Generalizable 3D Gaussian Reconstruction From Arbitrary Views

Xin Fei1,2,*  Wenzhao Zheng1,2,✉  Yueqi Duan1  Wei Zhan2 
Masayoshi Tomizuka2  Kurt Keutzer2  Jiwen Lu1 

1Tsinghua University    2UC Berkeley 

*Work done during an internship at UC Berkeley, ✉Corresponding author



  arXiv     Paper     Code  

Overview

Sample Image

Most existing generalizable 3D Gaussian splatting methods (e.g., pixelSplat, MVSplat) assign a fixed number of Gaussians to each pixel, leading to inefficiency in capturing local geometry and overlap across views. Differently, our PixelGaussian dynamically adjusts the Gaussian distributions based on geometric complexity in a feed-forward framework. With comparable efficiency, PixelGaussian (trained using 2 views) successfully generalizes to various numbers of input views with adaptive Gaussian densities.

Our Pipeline

Sample Image

Given multi-view input images, we initialize 3D Gaussians using a lightweight image encoder and cost volume. Cascade Gaussian Adapter (CGA) then dynamically adapts both the distribution and quantity of Gaussians. By leveraging local image features, Iterative Gaussian Refiner (IGR) further refines Gaussian representations via deformable attention. Finally, novel views are rendered from the refined 3D Gaussians using rasterization-based rendering.

Sample Image

(a) CGA comprises a keypoint scorer followed by a series of hypernetworks that produce context-aware thresholds to guide the splitting and pruning of Gaussians. (b) IGR further facilitates direct image-Gaussian interactions, enabling Gaussian representations to capture and extract local geometric features more effectively.

Experiment Results

Quantitative Results

Sample Image
  • PixelGaussian achieves the best performance on the two representative datasets RealEstate10K and ACID.
  • Trained with 2 reference views, PixelGaussian can generalize to more views.

PixelGaussian achieves adaptive numbers of Gaussians

Sample Image

PixelGaussian achieves faster rendering FPS due to fewer Gaussians

Sample Image

Visualizations

Sample Image

PixelGaussian gradually adjust the Gaussian density

Sample Image

Larger Gaussian density in complex areas

Sample Image

Contact

If you have any questions, please feel free to contact us:

  • Xin Fei: feix21@mails.tsinghua.edu.cn
  • Wenzhao Zheng: wenzhao.zheng@outlook.com

BibTeX

        
          @article{fei2024pixel,
            title={PixelGaussian: Generalizable 3D Gaussian Reconstruction From Arbitrary Views},
            author={Fei, Xin and Zheng, Wenzhao and Duan, Yueqi and Zhan, Wei and Tomizuka, Masayoshi and Keutzer, Kurt and Lu, Jiwen},
            journal={arXiv preprint arXiv:2410.18979},
            year={2024}
        }