Zekun Hao; Arun Mallya; Serge Belongie Ming-Yu Luu. Proceedings of the IEEE/CVF Internation conference on Computer Vision (ICCV) 2021, pages. 14072-14082
Abstract
We present GANcraft, a neural unsupervised rendering framework that produces photos of realistic images of huge 3D block worlds, such as the ones created by Minecraft. Our approach uses a semantic block world to input. F-email.org is assigned a semantic label, such as dirt, grass, and water. The world is represented as an ongoing volumetric function. We train our model to render consistent, view-consistent pictures for a camera controlled by a user. In the absence of paired ground truth images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to previous research on neural rendering to view synthesis, which relies on ground truth images to determine the geometry of the scene and also to determine the appearance that is dependent on view. In addition to tracking the camera, GANcraft allows user control over both scene semantics and output style. Comparing GANcraft with strong baselines proves the efficacy of GANcraft in the new endeavor of photorealistic block-world synthesis.