Image Completion with GANs
From left to right: original protrait, incomplete portrait, portrait completed by the GAN.GitHub repository
Motivation & Objective
This project aims at implementing a Generative Adversarial Network (GAN) for image completion in Python and Pytorch. The architecture, as described in the reference research paper (cf. reference section) is meant to be fed with incomplete images (images with holes of any shape in it) and to complete them. An example is given above: the left image is the original image, the center image is the image given to the trained GAN, and the right image is the output of the GAN.
Such a model can be used on damaged portraits as depicted above, but it can also be used on landscapes to erase unwanted objects or features in an image.
Data
Two instances of the model were trained on two different datasets:- The first dataset used is a subset of CelebA that contains celebrities portraits. All images in this dataset are pretty similar, which eases the learning process.
- The second dataset used is a subset of Places2 that contains day-to-day life scenes. With 400+ unique scene types, this dataset has greater variance, which hindered the learning process and results.
Approach
Without diving in the training process and architecture details, the particularities of this model are the following:- The Generator is not fed with noise but with an incomplete image (a group of pixels are set to black);
- The Generator is first trained by itself, the loss being the pixel-wise distance between the original image and the completed image;
- The Discriminator is composed of two sub-models, one assessing the realness of the whole image, the other one assessing the realness of a sub-image centered on the missing zone;
- The Discriminator is also trained by itself before being combined with the Generator.
References
- Model Architecture: Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Trans. Graph. 36, 4, Article 107 (August 2017), 14 pages
- CelebA dataset: Liu, Ziwei and Luo, Ping and Wang, Xiaogang and Tang, Xiaoou. 2015. Deep Learning Face Attributes in the Wild. Proceedings of International Conference on Computer Vision (ICCV)
- Places2 dataset: Places: A 10 million Image Database for Scene Recognition B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017