Diffusion-based portrait image generation scheme using plug-in adapters

Portrait image generation is the task of creating a new face image that reflects modification conditions while preserving the identities of the reference portrait images. Although recent deep learning-based generative models have shown excellent image generation performance, they suffer from inefficiencies in this task, including fine-tuning using multiple reference images for each person. To address these issues, we propose a novel diffusion-based portrait image generation method using two plug-in adapters: a semantic adapter and a spatial adapter. The semantic adapter integrates text conditions into the facial identity from a single reference image, while the spatial adapter adjusts the spatial configuration of the same reference face image. In experiments using the LAION-Face dataset, the proposed method achieved 16.7% improvement in Id-Pres and 6.35% improvement in CLIP-TI over state-of-the-art portrait generation methods. Also, we demonstrate through ablation studies that both adapters are essential for efficient portrait image generation.