WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

🔼 Figure 2 showcases examples from the WISE benchmark dataset, illustrating its breadth and complexity. WISE evaluates Text-to-Image (T2I) models’ ability to generate images based on complex semantic prompts requiring world knowledge and logical reasoning. The figure presents example prompts and corresponding images across three main categories: Cultural Common Sense, Spatio-Temporal Reasoning, and Natural Science. Each category is further divided into 25 sub-domains, demonstrating the diverse range of prompts that test a model’s understanding of the world. The prompts intentionally go beyond simple keyword matching; instead, they require models to perform logical inference and integrate their world knowledge to generate accurate and relevant images.

read the caption

Figure 2: Illustrative samples of WISE from 3 core dimensions with 25 subdomains. By employing non-straightforward semantic prompts, it requires T2I models to perform logical inference grounded in world knowledge for accurate generation of target entities.

🔼 The figure shows a detailed breakdown of the WISE benchmark dataset, which is composed of three main categories: Cultural Common Sense, Spatio-Temporal Reasoning, and Natural Science. Each category is further divided into several subdomains (25 in total), providing a comprehensive evaluation of the models’ understanding of diverse aspects of world knowledge during image generation. The subdomains within each category are visually represented, offering a clear overview of the benchmark’s structure and scope.

read the caption

Figure 3: Detailed composition of WISE, consisting of 3 categories and 25 subdomains.

🔼 This figure illustrates the WISE (World Knowledge-Informed Semantic Evaluation) framework’s four-stage evaluation process. It highlights how WISE assesses generated images’ alignment with world knowledge across three core dimensions (Cultural Common Sense, Spatio-temporal Reasoning, and Natural Science). Two example prompts are shown: ‘a candle in space’ (Natural Science) and ‘a close-up of a maple leaf in summer’ (Spatio-temporal Reasoning). Both prompts reveal limitations in the models’ understanding of fundamental scientific and seasonal facts, respectively, resulting in a consistency score of 0. This demonstrates WISE’s ability to effectively identify knowledge-related conflicts in generated images.

read the caption

Figure 4: Illustration of the WISE framework, which employs a four-phase verification process (Panel I to IV) to systematically evaluate generated content across three core dimensions. The two representative cases, science-domain input “candle in space” violates oxygen-dependent combustion principles, while spatiotemporal-domain “close-up of summer maple leaf” contradicts botanical seasonal patterns, both receiving 0 in consistency (see Evaluation Metrics in Panel III), confirming the benchmark’s sensitivity in world knowledge conflicts.

🔼 This table presents the average WiScore achieved by various text-to-image (T2I) models on a simplified version of the WISE benchmark. The prompts in this simplified version, rewritten using GPT-40, are more direct and less reliant on world knowledge compared to the original WISE prompts. The table displays the normalized WiScore (divided by 2) for each model across six categories (Cultural, Time, Space, Biology, Physics, Chemistry) and an overall average. This allows for a comparison of model performance when faced with less complex prompts, highlighting how well models can generate images based on simpler instructions.

read the caption

Table 2: Normalized WiScore results on rewritten prompts. These prompts were simplified from the original WISE benchmark using GPT-4o (e.g., “The plant often gifted on Mother’s Day” to ”Carnation”).

🔼 This table presents the consistency scores achieved by various text-to-image (T2I) models across different subdomains within the WISE benchmark. The WISE benchmark evaluates the models’ ability to generate images that accurately reflect the semantic content and world knowledge embedded within a set of prompts. The subdomains represent different categories of semantic complexity (Cultural Common Sense, Spatio-temporal Reasoning, Natural Science), each further divided into several specific sub-categories. The consistency score indicates how accurately the generated image aligns with the intended meaning of the prompt. Higher scores suggest better alignment between the image and the prompt’s intended meaning, demonstrating the model’s stronger understanding and successful integration of world knowledge.

read the caption

Table 3: Consistency score of different models.

🔼 This table presents a quantitative evaluation of realism in images generated by various text-to-image (T2I) models. Realism is assessed across five categories: Cultural Common Sense, Time, Space, Biology, and Physics, along with an overall Realism score. The models are categorized as either dedicated T2I models or unified multimodal models, allowing for a comparison of the performance of the different model architectures in generating realistic imagery. Each score likely reflects the average realism rating across a significant number of image generation tasks within that particular category. Higher scores indicate more realistic image generation.

read the caption

Table 4: Realism scores of different models.

🔼 This table presents the Aesthetic Quality scores achieved by various text-to-image (T2I) models. The scores are a crucial part of the WiScore metric and reflect the overall artistic appeal and visual quality of the images generated by each model. The models are categorized into dedicated T2I models and unified multimodal models, and scores are provided for each model across different subdomains within the WISE benchmark.

read the caption

Table 5: Aesthetic Quality scores of different models.

🔼 This table presents the consistency scores achieved by various text-to-image (T2I) models when evaluated on a modified version of the WISE benchmark. The ‘rewritten prompts’ are simplified versions of the original WISE prompts, making them more direct and less reliant on complex world knowledge. The simplification was done using GPT-40 to convert prompts such as ‘The plant often gifted on Mother’s Day’ to the simpler prompt ‘Carnation.’ The scores represent how accurately and completely each model’s generated image reflects the simplified prompt, indicating the models’ ability to understand and represent basic concepts visually. The table is broken down into six categories: Cultural, Time, Space, Biology, Physics, and Chemistry, representing the semantic domains of the prompts, and then provides an overall score across all categories. It allows for a comparison of model performance on simplified prompts, offering insights into the balance between world knowledge integration and basic visual understanding in the generation process.

read the caption

Table 6: Consistency scores of different models on rewritten prompts. These prompts were simplified from the original WISE benchmark using GPT-4o (e.g., “The plant often gifted on Mother’s Day” to ”Carnation”).

🔼 This table presents the realism scores achieved by various text-to-image (T2I) models when prompted with simplified versions of the WISE benchmark prompts. The original WISE prompts, designed to assess the model’s world knowledge, were rewritten using GPT-40 to be more direct and image-focused. The scores indicate how realistically each model generated images based on these simplified prompts. Higher scores suggest more realistic image generation. The table breaks down the scores by subdomain within the three main WISE categories: Cultural Common Sense, Spatio-temporal Reasoning, and Natural Science, as well as providing an overall average realism score for each model.

read the caption

Table 7: Realism scores of different models on rewritten prompts. These prompts were simplified from the original WISE benchmark using GPT-4o (e.g., “The plant often gifted on Mother’s Day” to ”Carnation”).

🔼 This table presents the Aesthetic Quality scores achieved by various text-to-image (T2I) models when evaluated using simplified prompts. These simplified prompts were generated by GPT-40, which rephrased complex, knowledge-rich prompts from the WISE benchmark into more concise, image-centric descriptions (e.g., replacing “The plant often gifted on Mother’s Day” with simply “Carnation”). The scores are categorized by model type (Dedicated T2I and Unified MLLM), and further broken down by sub-domain (Cultural, Time, Space, Biology, Physics, Chemistry) to show performance variation across different knowledge types. Higher scores indicate better aesthetic quality in the generated images, according to the evaluation metric used in the WISE benchmark. The table allows comparison of model performance on simplified prompts, providing insights into the impact of prompt complexity on the ability of T2I models to generate high-quality, aesthetically pleasing images.

read the caption

Table 8: Aesthetic Quality scores of different models on rewritten prompts. These prompts were simplified from the original WISE benchmark using GPT-4o (e.g., “The plant often gifted on Mother’s Day” to ”Carnation”).