Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content

🔼 The figure illustrates a quality control process for ensuring the accuracy of annotations in the Q-Eval-100K dataset. A random sample of 5,000 instances from the complete dataset is selected and evaluated by experts to establish ‘golden scores’ – the benchmark for quality. Any subsequent batch of annotations is only accepted if the scores given by annotators for the sampled instances exhibit a high correlation with these expert-determined golden scores. This process helps to maintain a consistent and reliable standard of annotation quality across the entire dataset.

read the caption

Figure 2: Illustration of the Sample and Scrutinize quality control strategy for annotations in Q-Eval-100K. We randomly select a sample of 5K instances from the full dataset, which are then reviewed by experts to establish golden scores. A batch of annotations is approved only if the scores of the sampled instances show a high correlation with these expert-assigned golden scores.

🔼 The figure illustrates the Q-Eval-100K dataset creation process. It starts with prompt design, encompassing diverse prompts created manually and sourced from other datasets. These prompts are fed into various text-to-vision models to generate a wide range of images and videos. Human annotators then subjectively evaluate each generated instance, assessing both its visual quality and alignment with the original prompt. These evaluations are represented by Mean Opinion Scores (MOS). The resulting MOS data then forms a Supervised Fine-Tuning (SFT) dataset. This SFT dataset is specifically designed to train Large Multimodal Models (LMMs), effectively injecting human judgment on visual quality and alignment into the model’s training process.

read the caption

Figure 3: Overview of the Q-Eval-100K construction process. We design a wide range of prompts and employ various text-to-vision models to generate diverse content. Subjective evaluations are then conducted to rate the visual quality and alignment of these generated instances. The resulting scores form the SFT dataset, which can help inject corresponding knowledge into LMMs.

🔼 This figure shows the distribution of Mean Opinion Scores (MOS) for image alignment across different text-to-image generative models. It uses violin plots to represent the distribution of scores for each model, visualizing both the central tendency and the spread of scores. This allows for comparison of the alignment capabilities of various models, showing which models tend to produce more consistent alignments between the generated images and their corresponding text prompts.

read the caption

(a) Image Alignment

🔼 This figure shows the distribution of Mean Opinion Scores (MOS) for video alignment across various text-to-video generation models. The MOS represents human judgment on how well the generated video aligns with the corresponding text prompt. The box plots visualize the range, median, and quartiles of MOS scores for each model. The variations in the distributions reveal the differences in the alignment capabilities of different generation models. Models closer to a score of 5 demonstrate a higher level of alignment, indicating a better correspondence between the visual content of the video and the text prompt.

read the caption

(b) Video Alignment

🔼 This figure is a violin plot showing the distribution of Mean Opinion Scores (MOS) for the visual quality of generated images across various generative models. Each violin represents a different model, and the width of the violin reflects the density of scores at different MOS values. The plot provides a visualization of the performance of various models on the image visual quality, allowing for a comparison of their visual quality capabilities.

read the caption

(c) Image Visual Quality

🔼 This figure (d) shows the distribution of Mean Opinion Scores (MOS) for the visual quality of generated videos in the Q-Eval-100K dataset. It uses violin plots to visualize the MOS distribution across different video generation models, showing the variation in visual quality for each model. The x-axis represents different models, and the y-axis represents the MOS values for visual quality. The violin plot shows both the probability density and the quantiles of the data. The wider the violin, the higher the data density for that model and the higher the variability of the visual quality.

read the caption

(d) Video Visual Quality

🔼 This figure displays violin plots showing the distributions of Mean Opinion Scores (MOS) for visual quality and alignment of generated images and videos from the Q-Eval-100K dataset. Each plot represents a different model’s performance on image alignment, video alignment, image visual quality, and video visual quality, allowing for a comparison of model performance across these categories and media types.

read the caption

Figure 4: MOS distributions for the visual quality and alignment of generated images and videos in the Q-Eval-100K dataset respectively.

🔼 The Q-Eval-Score model uses a multi-stage process to evaluate visual quality and alignment in text-to-vision content. First, a large language model (LMM) is trained on the Q-Eval-100K dataset, learning to assess both visual quality and alignment. Next, context prompts are used to guide the LMM, prompting more detailed and accurate responses. The model outputs probabilities for different rating tokens (e.g., Excellent, Good, Fair, Poor, Bad), which are then converted into final scores. A special ‘Vague-to-Specific’ strategy further refines the alignment scores, particularly for long prompts, by breaking them down into more manageable parts for evaluation.

read the caption

Figure 5: The pipeline of the proposed Q-Eval-Score model involves multiple stages. First, the Q-Eval-100K SFT dataset is used to train the LMM on visual quality and alignment knowledge. Then, context prompts are applied to guide the LMM towards generating more detailed and accurate outputs. Finally, the rating token probabilities are converted into predicted scores. Additionally, long prompt alignment is achieved through a Vague-to-Specific strategy to further refine the model’s responses.

🔼 The figure illustrates the Vague-to-Specific strategy used to evaluate alignment in long prompts. A long prompt, describing a complex scene, is fed into a Large Language Model (LLM). The LLM processes this prompt and outputs two things: 1) a concise summary (the ‘Vague Prompt’) capturing the essence of the original prompt, and 2) several shorter, more specific prompts (the ‘Specific Prompts’) that break down the original prompt’s details. The alignment score is then calculated separately for the Vague Prompt and each Specific Prompt. These individual scores are finally combined using weighted averaging to get a final alignment score that more accurately reflects the overall consistency between the generated image and the original complex prompt.

read the caption

Figure 6: An example of the Vague-to-Specific strategy. The original long prompt is divided by the LLM (QwenLM [5]) into a Vague Prompt and several Specific Prompts. The alignment score is first calculated separately for each part, then combined using weighted averaging to form the final score.

🔼 This radar chart visualizes the overall performance of various models on visual quality assessment. Each axis represents a specific evaluation metric (Instance-level SRCC, Instance-level PLCC, Model-level SRCC, Model-level PLCC for both image and video), and each model’s performance is plotted as a point on the chart. The radial distance of the point from the center indicates the model’s performance score, allowing for a direct comparison across different models and evaluation metrics.

read the caption

(a)

🔼 This radar chart visualizes the overall performance of various models on the alignment aspect of the Q-Eval-100K dataset. The chart presents the Instance-level and Model-level performance of each model, measured using SRCC (Spearman rank correlation coefficient) and PLCC (Pearson linear correlation coefficient). Specifically, it compares the performance of CLIPScore, BLIP2Score, VQAScore, and Q-Eval-Score across different evaluation metrics on both images and videos.

read the caption

(b)

🔼 This figure presents a comparative analysis of visual quality and alignment performance across multiple evaluation methods on the Q-Eval-100K dataset. Two radar charts display the results: one for visual quality and one for alignment. Each chart shows the performance of different methods (indicated by abbreviations such as CLIPScore, BLIP2Score, and Q-Eval-Score) across four evaluation metrics: instance-level Spearman’s rank correlation coefficient (SRCC), instance-level Pearson linear correlation coefficient (PLCC), model-level SRCC, and model-level PLCC. The results are further broken down by image (’-i’) and video (’-v’) modalities. This allows for a comprehensive comparison of the methods’ performance across different evaluation aspects and data types.

read the caption

Figure 7: Radar charts of the overall performance on the Visual Quality and Alignment aspects on Q-Eval-100K, where IR, IP, MR, MP indicate Instance-level SRCC, Instance-level PLCC, Model-level SRCC, Model-level PLCC, and -i, -v represents image and video respectively.

🔼 This figure presents a visual comparison of the results from different evaluation methods, including CLIPScore, BLIP2Score, and Q-Eval-Score. For each of six example image-text pairs, the figure shows the Mean Opinion Score (MOS) from human evaluations alongside the scores assigned by each of the evaluation methods. This comparison allows for a visual assessment of how well each evaluation metric correlates with human judgments of visual quality and alignment.

read the caption

Figure 8: Visualization comparison results.

🔼 Table 2 presents a comparative analysis of various methods for evaluating the visual quality of images generated by text-to-image models, using the Q-Eval-100K dataset. It compares the performance of different metrics, including NIQE, CLIP-IQA, Q-Align, IPCE, and the proposed Q-Eval-Image, across both instance-level and model-level evaluations. The metrics’ performance is measured using SRCC and PLCC, reflecting the rank and linear correlation with human annotations. The table highlights the superior performance of Q-Eval-Image in both instance and model-level evaluations.

read the caption

Table 2: Performance comparison on the visual quality aspect of Q-Eval-100K. Best in bold, second underlined.

🔼 Table 3 presents a performance comparison of different methods for evaluating the alignment aspect of the Q-Eval-100K dataset. It compares the performance of Q-Eval-Score with several popular zero-shot alignment evaluation metrics (CLIPScore, BLIP2Score, and VQAScore). Results are shown for both image and video data, considering both instance-level (individual generated examples) and model-level (overall model performance) accuracy. To provide a fair comparison, results for the zero-shot methods are also included using their default weights (marked with an asterisk). This allows for a comprehensive evaluation of Q-Eval-Score’s performance compared to existing methods in the field.

read the caption

Table 3: Performance comparison on the alignment aspect of Q-Eval-100K. Considering that CLIPScore, BLIP2Score, and VQAScore are popular zero-shot alignment evaluation metrics, we provide the corresponding performance with the official default weight as well (marked with *).

🔼 This ablation study analyzes the impact of different components of the Q-Eval-Score model on its performance. It examines the contribution of supervised fine-tuning (SFT) training, context prompts, cross-entropy (CE) loss, and mean squared error (MSE) loss. The table presents the performance metrics (Instance-level SRCC, Instance-level PLCC, Model-level SRCC, Model-level PLCC) for both image and video data, comparing the full Q-Eval-Score model to variations where one component is removed.

read the caption

Table 4: Ablation Study of Q-Eval-Score.

🔼 This table presents a comparison of alignment evaluation performance on a subset of the Q-Eval-100K dataset containing long prompts (over 25 words). The comparison focuses on the proposed Q-Eval-Score model, specifically contrasting its performance with and without the application of the Vague-to-Specific (V2S) strategy. The results are presented for both image and video data using the Instance-level SRCC and PLCC metrics to show the impact of the V2S strategy on the model’s ability to effectively handle long prompts.

read the caption

Table 5: Performance comparison on the alignment aspect of Q-Eval-100K on the long prompt subset, where w/o V2S and w V2S represents the proposed Q-Eval-Score model with and without the Vague-to-Specific strategy respectively.

🔼 This table presents the cross-dataset validation results of Q-Eval-Score on the GenAI-Bench dataset. Q-Eval-Score, which was originally trained on the Q-Eval-100K dataset, is evaluated on GenAI-Bench to assess its generalization capabilities. The table compares the performance of Q-Eval-Score against several other popular alignment evaluation metrics (CLIPScore, BLIP2Score, and VQAScore), showing the SRCC and PLCC scores for both image and video alignment tasks on GenAI-Bench. An asterisk (*) indicates that the default weights were used for the respective methods.

read the caption

Table 6: Cross-dataset validation performance on GenAI-Bench. The Q-Eval-Score is trained on the Q-Eval-100K and then validated on GenAI-Bench. * indicates using default weight.

🔼 Table 7 provides a detailed breakdown of the prompt design used in the Q-Eval-100K dataset creation. It categorizes the prompts based on three key aspects: Entity Generation (creation of the primary entities within the image or video), Entity Attribute Generation (defining the characteristics of those entities), and Interaction Ability (describing the relationships and actions between entities or with the background). Each aspect is further divided into subcategories, illustrating the diversity and complexity of prompts included. The ‘Count’ column indicates the number of prompts within each subcategory.

read the caption

Table 7: Detailed Descriptions of Entity Generation, Entity Attribute Generation, and Interaction Ability

🔼 This table presents the cross-validation results of different models (Q-Align, CLIPScore, BLIP2Score, and Q-Eval-Score) on four datasets: AGIQA-Quality, T2VQA, AGIQA-Align, and TIFA160. All models were pre-trained using the Q-Eval-100K dataset. The results, reported as SRCC/PLCC (Spearman Rank Correlation Coefficient/Pearson Linear Correlation Coefficient), demonstrate the generalization performance of each model across different datasets and tasks (visual quality and alignment).

read the caption

Table 8: Cross-validation (All pre-trained on Q-Eval-100K).