透過 CLIP 文字編碼器適應進行條件擴散模型中的語義編輯和去偏見

Semantic Editing and Debiasing in Conditional Diffusion Models by CLIP Text-Encoder Adaptation

鄭廷瑋(Ting-Wei Cheng)

指導教授 ： 吳家麟

國立臺灣大學/電機資訊學院/資訊工程學系/碩士(2024年)

https://doi.org/10.6342/NTU202403591

摘要

本論文研究了 CLIP 文本編碼器在條件擴散模型中的適應，以解決語義編輯和去偏見相關的挑戰。我們探討了透過適應 (Adaptation) 在增強生成圖像語義屬性控制方面的有效性，同時減少內在偏見。研究利用了各種解耦策略，並對文字編碼器進行修改，以評估緩解與性別和種族相關偏見的潛力。我們的研究結果表明，通過針對性適應微調文本編碼器可以顯著提高語義控制的精確性和去偏見的有效性。本研究為圖像合成領域的更公平和可控的生成模型的發展做出了貢獻。

關鍵字

語意編輯 ； 去偏見 ； 圖片生成 ； LoRA

並列摘要

This thesis investigates the adaptation of the CLIP text encoder for use in conditionaldiffusion models to address challenges related to semantic editing and debiasing. We explore the effectiveness of low-rank adaptations in enhancing the control over semanticattributes of generated images while simultaneously reducing inherent biases. The studyutilizes various disentanglement strategies and introduces modifications to the text encoder to evaluate the potential for mitigating biases related to gender and ethnicity. Ourfindings indicate that fine-tuning the text encoder with targeted adaptations can significantly improve semantic control’s precision and debiasing effectiveness. This work contributes to the development of more fair and controllable generative models in the field ofimage synthesis.

並列關鍵字

Semantic Editing ； Debiasing ； Image Generative ； LoRA

參考文獻

[1]M. Brack, F. Friedrich, D. Hintersdorf, L. Struppek, P. Schramowski, and K. Kersting. Sega: Instructing text-to-image models using semantic guidance. In A. Oh,T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advancesin Neural Information Processing Systems, volume 36, pages 25365–25389. CurranAssociates, Inc., 2023.

[2] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advancesin neural information processing systems, 34:8780–8794, 2021.

[3] M. D’Incà, E. Peruzzo, M. Mancini, D. Xu, V. Goel, X. Xu, Z. Wang, H. Shi, andN. Sebe. Openbias: Open-set bias detection in text-to-image generative models.ArXiv, abs/2404.07990, 2024.

[4] R. Gandikota, J. Materzynska, T. Zhou, A. Torralba, and D. Bau. Concept sliders:Lora adaptors for precise control in diffusion models, 2023.

[5] E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. Ganspace: Discoveringinterpretable gan controls. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, andH. Lin, editors, Advances in Neural Information Processing Systems, volume 33,pages 9841–9850. Curran Associates, Inc., 2020.25doi:doi:10.6342/NTU202403591