Uni$ extbf{F}^2$ace: Fine-grained Face Understanding and Generation with Unified Multimodal Models

🔼 This figure illustrates the creation of the UniF2ace-130K dataset. The left panel details a three-step pipeline: Step 1 involves collecting high-quality facial images; Step 2 uses GPT-40 and a fine-grained face attribute classification model to generate detailed captions, correcting and enhancing initial captions; and Step 3 creates visual question-answering (VQA) pairs using GPT-4. The right panel showcases examples demonstrating how this process refines GPT-40 generated captions, resulting in more accurate, comprehensive descriptions that include corrected attributes (e.g., gender), enhanced details (e.g., bags under eyes), and inferred information (e.g., a person is talking and seems slightly tired).

read the caption

Figure 2: Pipeline and examples of UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace-130K construction. Left: A three-stage pipeline for building UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace-130K. Step-1: High-quality face images are collected. Step-2: Detailed captions are generated by GPT-4o with a face attribute model trained to classify fine-grained appearance, action, and emotion. Step-3: Question-answering pairs are created. These stages collectively refine GPT-4o-generated captions and produce fine-grained descriptions for VQAs generation. Right: A representative example showcasing UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace-130K’s ability to correct (e.g., gender), enhance (e.g., bags under eyes), and reason (e.g., talking, slight tiredness) in GPT-4o-generated captions.

🔼 UniFace, a unified multimodal model, processes both text and image inputs to perform Text-to-Image (T2I) and Multimodal Understanding (MMU) tasks. Text is tokenized, and images are encoded using VQGAN. These are combined into a single sequence. A noise scheduler then masks a portion of the image tokens. The masked tokens are fed through a Transformer network containing Mixture-of-Experts (MoE) layers. The MoEs are divided into generation and understanding groups. The first group of MoEs operates at the token level, using both shared and routed experts, while the second group operates at the sequence level and incorporates domain-specific features. This hierarchical architecture allows for fine-grained processing of facial features. The model’s training objective combines a D3Diff loss (based on the noise scheduler’s output p_t(x_t|x₀)) and a text autoregressive loss.

read the caption

Figure 3: Our UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace architecture integrates Text-to-Image (T2I) and Multimodal Understanding (MMU) tasks. Text inputs are encoded via a tokenizer, while input images are processed through a VQGAN encoder, merging into a unified token sequence. A noise scheduler masks a subset of image tokens, which are then processed by a Transformer with Mixture-of-Experts (MoE) layers. These MoE layers are grouped for generation and understanding tasks, with the first operating at the token level using shared and routed experts, and the second incorporating domain-specific features at the sequence level. This hierarchical design enables fine-grained facial feature processing. The noise scheduler outputs pt⁢(𝐱t|𝐱0)subscript𝑝𝑡conditionalsubscript𝐱𝑡subscript𝐱0p_{t}(\mathbf{x}_{t}|\mathbf{x}_{0})italic_p start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( bold_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | bold_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) for D3Diff loss computation, combined with text autoregressive loss to form the training objective.

🔼 Figure 4 presents a comparison of face image generation results from five different models: SDXL, TokenFlow, OmniFlow, Show-o, and UniFace. The goal is to showcase UniFace’s superior ability to generate high-fidelity images that accurately reflect the details specified in text prompts. Each model is given the same prompt, and the resulting images are displayed alongside the prompt used to generate them. By comparing the generated images, one can observe that UniFace produces images with significantly more detail and accuracy in capturing fine-grained attributes (such as hair color, clothing, and facial features) compared to the other models. The highlighted attributes in the figure emphasize UniFace’s strength in this area.

read the caption

Figure 4: Comparative analysis of face images generation quality across SDXL [43], TokenFlow [44], OmniFlow [23], Show-o [67], and UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace. Our proposed UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace effectively captures more detailed information from prompts. We highlight fine-grained attributes.

🔼 This figure visualizes the activation frequencies of the Mixture-of-Experts (MoE) layers within the UniFace model. It’s broken down into token-level and sequence-level MoE activations for both understanding and generation tasks. The size of the circles in the heatmaps directly corresponds to the frequency of activation for each expert within each layer. Larger circles represent more frequent activation. The left column shows the understanding tasks (image-to-text), while the right column shows the generation tasks (text-to-image). This allows for a visual comparison of expert usage patterns between the two main tasks and across different layers of the network.

read the caption

Figure 5: Activation frequency of Token-Level and Sequence-Level MoE in different layers. The left column corresponds to understanding tasks, while the right column corresponds to generation tasks. Larger circles indicate experts that are activated more frequently.

🔼 Figure 6 presents a comparison of visual question answering (VQA) performance between UniFace and other state-of-the-art models. UniFace excels at generating detailed and accurate answers to complex questions about facial features. The figure demonstrates UniFace’s ability to capture and express subtle nuances of facial attributes such as hairstyles, earrings, or expressions, offering a superior level of detail and accuracy compared to alternative approaches. GPT-40 scores are also included to provide a benchmark against a powerful large language model.

read the caption

Figure 6: Comparison of visual question-answering results and GPT-4o-based scores.

🔼 This figure shows a comparison of face images generated by different models: SDXL, TokenFlow, OmniFlow, Show-o, and UniFace (the authors’ model). For each model, several example images are displayed alongside a textual description of the generated face. This visual comparison highlights the differences in image quality, detail, and adherence to the prompt across various models. UniFace demonstrates a higher level of detail and accuracy in capturing fine-grained features compared to other models.

read the caption

Figure 1: More comparison of generated face images with other models.

🔼 This figure showcases additional examples of faces generated by the UniFace model. It demonstrates the model’s ability to produce diverse and high-quality images of faces with varying attributes and characteristics.

read the caption

Figure 2: More face images generated by UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace

🔼 Figure 3 showcases a comparison of captioning results from various models, specifically highlighting fine-grained facial attributes. The figure directly compares the captions generated by different models against a ground truth. Blue highlights indicate correctly identified fine-grained attributes, while red highlights indicate errors or omissions. This allows for a visual assessment of each model’s capability in capturing nuanced facial details and the accuracy of their descriptions. Each model’s caption is presented alongside a numerical score reflecting its performance. This visual representation aids in understanding the relative strengths and weaknesses of each model concerning fine-grained detail in facial attribute identification and description.

read the caption

Figure 3: Comparison of captioning results and DeepSeeek-v3-based scores. We highlight fine-grained attributes with blue and errors in answers with red.

🔼 Figure 4 details the prompts used to create the UniF2ace-130K dataset. The process involved three steps. First, a prompt was given to GPT-40 to generate initial captions describing facial attributes. If the image had existing captions (as in the MM-CelebA-HQ dataset), those were incorporated into the prompt. Second, another prompt was used to refine those captions using additional fine-grained attributes. Third, a prompt was given to GPT-4 to generate visual question-answering (VQA) pairs based on the refined captions, aiming for a detailed understanding of facial attributes.

read the caption

Figure 4: Prompts for building dataset. The first and second prompts are to GPT-4o, while the last prompt is to GPT-4. In the first prompt, the content in “[]” is used only when the image data includes built-in captions, such as in MM-CelebA-HQ dataset.

🔼 Table 2 presents a comparison of UniFace’s performance on face understanding tasks against several state-of-the-art multimodal models, including both advanced models focusing solely on understanding and unified multimodal models capable of both understanding and generation. The metrics used assess the model’s ability to extract and analyze detailed facial features from images. UniFace significantly outperforms all other models across all evaluation metrics, showcasing its superior capabilities in fine-grained face understanding.

read the caption

Table 2: Evaluation on face understanding tasks compared with advanced multimodal models and unified multimodal models. Our UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace achieves the highest scores across all metrics, demonstrating superior ability to extract and analyze features from face images.

🔼 This table presents an ablation study comparing the performance of the UniFace model trained with different loss functions. The primary loss function used is D3Diff (Dual Discrete Diffusion), which combines elements of score-matching and masked generative losses. The table shows the results when training with only the score matching loss, only the masked generative loss, and varying the weighting coefficient (α) in the D3Diff loss. Metrics evaluated include VQAscore-CF5, VQAscore-LV, FID, and VLM-score. The best overall results (bold) are achieved with α = 0.01 in the D3Diff loss function, indicating the effectiveness of the dual loss approach. The second-best results (underlined) provide further context for the relative performance of alternative loss functions.

read the caption

Table 3: Performance comparison with different loss. Considering all metrics, the optimal result is achieved with α=0.01𝛼0.01\alpha=0.01italic_α = 0.01 in D3Diff. Bold: Best performance. Underlined: Second best performance.

🔼 This table presents an ablation study evaluating the impact of token-level and sequence-level Mixture-of-Experts (MoEs) on the UniFace model’s performance. It shows the results of experiments where each type of MoE was removed individually to measure their contribution to overall performance. The results demonstrate that both token-level and sequence-level MoEs significantly improve the model’s performance across multiple metrics.

read the caption

Table 4: Performance impact of token-level and sequence-level MoE in UniF2superscriptF2\textbf{F}^{2}F start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTace through ablation study. Both MoEs contribute significant performance improvements.