16 Mar, 2024

                                               [Runyi Yang](<https://runyiyang.github.io/>) / [Runyi’s Blogs](<https://runyiyang.notion.site/Runyi-s-Blogs-f52d6bf73e104c51a4f5e80529b6a9b6>)

Siggraph 2023 Best Paper

https://github.com/graphdeco-inria/gaussian-splatting

Abstract

• Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

1 Introduction

Goal

Achieve real-time rendering & Create the representations with fast optimization time.

Basic Idea

• Use 3D Gaussians as a flexible and expressive scene representation.
  • Initialize with sparse SfM points (High quality)
  • Or initialize with random points (Low quality)
• Use rasterization technique and 𝛼-blending to render. (Fast rendering)
• 3D Gaussians Property
  • 3D position
  • opacity 𝛼
  • anisotropic covariance
  • spherical harmonic (SH) coefficients

2. 3D Gaussians

Given the sparse point cloud $P_c$, the initial step in our methodology involves the establishment of Gaussian fields. This is achieved by projecting the 3D coordinates $x \in P_c$ onto Gaussian primitives $p \in P$ as described by the following equation:

$$ p(x) = \exp(-\frac{1}{2}(x)^T \Sigma^{-1}(x)) $$

where the Σ is defined as 3D covariance matrix in the world space. To ensure the positive semi-definiteness and to uphold the physical interpretation of the covariance matrix, 3DGS introduced an efficient methodology for its decomposition. This approach utilizes an ellipsoid configuration to represent the 3D Gaussian covariance. The decomposition of Σ is achieved using a scaling matrix S and a rotation matrix R, as expressed in the equation:

$$ \Sigma = RSS^TR^T $$

This representation of anisotropic covariance is particularly advantageous for optimization processes. The subsequent sections of this paper will delve into the methods employed for optimizing Gaussians in accordance with the scene.