3D GAN Inversion with Pose Optimization

Jaehoon Ko1, Kyusun Cho1, Daewon Choi1, Kwangrok Ryoo1,2, Seungryong Kim1,
1Korea University, 2LG AI Research

WACV 2023

arXiv Code

Abstract

With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image. We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs.

Our Optimization Scheme

overview_image


Pre-trained Networks for Initialization

We pre-train a camera pose estiamtor and latent code encoder for better initialization of the optimization process, by training the respective newtorks on a generated pseudo pair dataset.

Optimization using Depth-based Warping

We use the depth map calculated from NeRF parameters and compare the estimated depth from the obtained camrea pose and projected depth from a canoncial viewpoint which allows us to optimize the latent code and camera pose simultaneously.

Generater Tuning with Depth Regularization

We fine tune the generator for a more accurate reconstruction of the given image by adding a depth regularization term to the loss function.

Reconstruction Results

overview_image
overview_image

Related Links

Our method was implemented on EG3D, the state-of-the-art nerf-based 3D-aware GAN model.

Our method was built upon Pivotal Tuning, a popular and simple method for 2D GAN inversion.

Our depth-based optimization scheme was inspired by monocular depth estimation literature.

We borrowed the idea of using a depth smoothness regularization during the generator tuning step from RegNeRF.

There's a lot of excellent work that was introduced around the same time as ours, but most of them requires ground-truth camera pose or 3D morphable models.

BibTeX

@article{ko20233d,
  author    = {Ko, Jaehoon and Cho, Kyusun and Choi, Daewon and Ryoo, Kwangrok and Kim, Seungryong},
  title     = {3D GAN Inversion with Pose Optimization},
  journal   = {WACV},
  year      = {2023},
}