Zekun Hao, Arun Mallya, Serge Belongie Ming-Yu Luu and Ming-Yu Liu; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14072-14082
We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds, such as those created in Minecraft. Our approach uses a semantic block world as an input and each block is assigned a semantic label , such as grass, dirt, or water. We model the world as continuous volumetric functions and train our model to render consistent and view-oriented photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we propose an approach to training based on pseudo-ground truth and adversarial training. This is in contrast to previous work using neural rendering that aids in view synthesizing. This requires ground truth images to establish the geometry of the scene and also to determine the view-dependent appearance. GANcraft gives users control over both scene semantics as well as output style. premium ebooks Results from experiments compared to strong baselines demonstrate the efficacy of GANcraft for this unique task of photorealistic 3D block synthesizing.