[DL輪読会]Pose Manipulation with Identity Preservation

DEEP LEARNING JP [DL Papers] Pose Manipulation with Identity Preservation Keno Harada, B4, UT http://deeplearning.jp/ 1

• http://deeplearning.jp/

● ○ ○ ○ ● ● ● Andrei-Timotei Ardelean, Lucian Mircea Sasu Transilvania University of Brasov Journal : INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL : : ○ https://arxiv.org/pdf/2004.09169.pdf https://github.com/TArdelean/CainGAN landmark , Face Reenactment SPADE .

• https://arxiv.org/pdf/2004.09169.pdf
• https://github.com/TArdelean/CainGAN

● Reenactment ● ● ●

Reenactment: ● , , , ○ ■ Synthesizing Obama: Learning Lip Sync from Audio Image from The Creation and Detection of Deepfakes: A Survey

Reenactment: ● , ○ ○ , photometric, geometric and kinematic complexity ● ○ Image from The Uncanny Valley

Reenactment: ● Warping ○ ○ ■ ■ ■ ■ ■ , occlusion Bringing portraits to life View morphing Deepwarp: Photorealistic image resynthesis for gaze manipulation Unsupervised disentangling of shape and appearance X2face: A network for controlling face generation using images, audio, and pose codes ● Direct synthesis ○ ○ ■ ■ ■ Image-to-image translation with conditional adversarial networks Deep video portraits Video-to video synthesis

Reenactment: ● ● few-shot, one-shot ○ ○ ○ ○ Few-Shot Adversarial Learning of Realistic Neural Talking Head Models Pose Manipulation with Identity Preservation <One-Shot Identity-Preserving Portrait Reenactment Neural Head Reenactment with Latent Pose Descriptors (CVPR 2020)

: CainGAN

:4 ● Generator ○ landmark, ● Targetted Embedder ○ , landmark, (Identity) ● Identity discriminator ○ ○ Identity source ● Pose discriminator ○ ○ Target landmark landmark

: Generator ● Semantic Image Synthesis with Spatially-Adaptive Normalization ○ SPADE ● High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs ○ Pix2PixHD, Multiscale discriminator ● Which Training Methods for GANs do actually Converge? ○ ResNet

: Generator: SPADE ● BatchNorm ○ ○ ● Semantic GauGAN semantic BatchNorm

: Discriminator: Multiscale ● Discriminator, Discriminator Discriminator

: Targeted Embedder Responsibitiliy Spatial embedding Identity embedding ●

:

● Targetted embedder ● SPADE ○ Identity Embedding Semantic ● Semantic ○ Spade ● Few shot vid2vid ● ?? ??