WritingBench: A Comprehensive Benchmark for Generative Writing

🔼 This figure illustrates the four-stage query construction pipeline used to create WritingBench queries. It starts with initial query generation using LLMs, then uses a refinement pool containing five writing requirements (three core competencies—personalization, stylistic adjustments, and format specifications—highlighted with black borders) and an expression type (purple). Checked strategies, refining the initial queries, are applied to produce multi-requirement prompts (color-coded text), with red phrases referencing gray-shaded writing support materials provided in the initial query. The process ensures diverse writing tasks covering various domains and includes integrating heterogeneous sources of materials. The implementation details are described in Section 3.1 of the paper.

read the caption

Figure 2: Construction pipeline of WritingBench queries. The refinement pool contains five writing requirements (three core competencies with black borders) and one expression type (purple). Checked strategies refine initial queries into multi-requirement prompts (color-coded text) with red phrases referencing gray materials. Implementation details in Section 3.1.

🔼 This figure shows a donut chart illustrating the distribution of WritingBench’s 1239 queries across six primary domains and their corresponding 100 secondary subdomains. Each primary domain is represented by a segment of the donut chart and its size corresponds to the number of queries in that domain. The secondary subdomains are further broken down within each primary domain, and the number of queries (Num) and the number of subdomains (Sub) are indicated for each primary domain. The chart visually represents the breadth and depth of WritingBench’s coverage of various writing tasks.

read the caption

Figure 3: Domain categories in WritingBench. Inner represents the 6 primary domains and outer depicts 100 secondary subdomains (Sub = subdomains per category, Num = dataset entries).

🔼 This figure showcases the dynamic criteria generation process within WritingBench. A writing query is provided as input, and the system automatically generates five evaluation criteria, each with a detailed description and a 10-point scoring rubric. The diverse background colors highlight the different types of requirements (e.g., formatting, style, content). This illustrates how WritingBench adapts its evaluation to each writing task, providing a more nuanced and comprehensive assessment compared to traditional static evaluation methods.

read the caption

Figure 4: Example of dynamically generating criteria for a writing query in WritingBench. Different background colors represent various types of requirements.

🔼 Figure 5 is a heatmap visualization showing the performance of various LLMs (large language models) across 100 subdomains within six primary domains of the WritingBench benchmark. Each subdomain represents a specific type of writing task (e.g., writing a scientific paper, a legal document, a poem etc.). The color intensity represents the average score achieved by each model on each subdomain, with red indicating higher scores and blue representing lower scores. This figure allows for a detailed comparison of the strengths and weaknesses of different LLMs in various writing scenarios.

read the caption

Figure 5: Scores of different models across different subdomains in WritingBench. Red indicates higher score and blue refers to lower score. The figures are the average score of each subdomain for different models.

🔼 This figure presents a performance comparison of various large language models (LLMs) on the WritingBench benchmark across different input lengths. The x-axis represents the range of input lengths (in tokens), and the y-axis shows the corresponding average scores achieved by each LLM. The different colored lines represent different LLMs, allowing for a visual comparison of how their performance varies with input length. This illustrates the impact of input length on the ability of each model to generate high-quality writing.

read the caption

Figure 6: Scores of different models across various input lengths on the WritingBench.

🔼 This figure shows the performance of various LLMs (large language models) on the WritingBench benchmark across different output lengths. The x-axis represents output length in tokens, and the y-axis represents the average score achieved by each model. Each line represents a different LLM, illustrating how well each model performs at generating text of varying lengths. The plot helps to identify strengths and weaknesses of the LLMs in producing longer or shorter responses, highlighting models better suited for generating longer-form content. The scores likely reflect a composite of quality metrics such as fluency, coherence, and relevance.

read the caption

Figure 7: Scores of different models across various output lengths on the WritingBench.

🔼 This table presents a statistical overview of the WritingBench dataset, categorized by six primary domains and 100 subdomains. It shows the number of queries, average and maximum token counts for inputs and outputs, and the distribution of queries across different length categories (less than 1k tokens, 1k-3k tokens, 3k-5k tokens, and more than 5k tokens). The table also details the distribution of queries based on three key writing requirements: style, format, and length.

read the caption

Table 2: Data statistics for WritingBench categorized by domain, requirement, and length.

🔼 This table presents the WritingBench performance evaluation results for various Large Language Models (LLMs). The evaluation was conducted using a critic model and focuses on six writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, and Advertising & Marketing) across three writing requirements (Style, Format, and Length). Scores range from 1 to 10, reflecting the quality of LLM writing in each domain and requirement. The ‘C’ column signifies the category-specific scores, providing a more granular view of LLM performance on particular aspects within each domain.

read the caption

Table 3: WritingBench performance of different LLMs across 6 domains and 3 writing requirements evaluated with our critic model (scale: 1-10). The six domains include: (D1) Academic & Engineering, (D2) Finance & Business, (D3) Politics & Law, (D4) Literature & Art, (D5) Education, and (D6) Advertising & Marketing. The three writing requirements assessed are: (R1) Style, (R2) Format, and (R3) Length. Here, “C' indicates category-specific scores.

🔼 This table presents the results of a human evaluation experiment comparing different methods for generating evaluation criteria in a writing benchmark. Specifically, it examines the agreement between human judges and three different approaches: using static, globally uniform criteria; static, domain-specific criteria; and dynamic, query-dependent criteria. The experiment uses two LLMs, ChatGPT-4 (referred to as ChatGPT) and Claude-3.5-Sonnet (referred to as Claude), as judges to assess the consistency of each criteria generation method. The scores represent the percentage of agreement between the human judges and the respective LLM judge.

read the caption

Table 4: Comparison of human consistency scores across different criteria generation methods. ChatGPT corresponds to ChatGPT-4o-latest, Claude corresponds to Claude-3.5-Sonnet.

🔼 This table presents the performance evaluation results of writing models on two benchmarks: WritingBench and EQBench. The models are categorized as either trained on the full dataset (’-all’) or a filtered subset (’-filtered’) of high-quality data curated using the WritingBench framework. The scores provide a quantitative comparison of the models’ writing capabilities, highlighting the impact of data filtering on model performance.

read the caption

Table 5: Performance evaluation of our writing models on two benchmarks.’-filtered’ indicates models trained with filtered data, while ’-all’ refers to those trained with the full dataset.

🔼 This table presents the prompt used to initiate the generation of writing queries in the WritingBench benchmark. The prompt instructs a language model to produce 10 distinct writing requests under a specified secondary domain, while remaining within the context of a primary domain. It emphasizes the need for detailed and specific requests that reflect realistic user needs and tone, and specifies the desired JSON format for the output.

read the caption

Table 6: Initial query generation prompt introduced in Section 3.1.1.

🔼 This table lists the 20 subdomains categorized under Academic & Engineering and Finance & Business domains in the WritingBench benchmark. For each subdomain, a brief description of the type of writing task it represents is provided. This helps clarify the range of writing tasks covered within these two broad domains.

read the caption

Table 7: Subdomains in Academic & Engineering and Finance & Business.

🔼 This table lists the 100 secondary subdomains included in the WritingBench benchmark, categorized under the primary domains of Politics & Law and Literature & Art. For each subdomain, a concise description is provided to clarify the type of writing task involved.

read the caption

Table 8: Subdomains in Politics & Law and Literature & Art.

🔼 Table 9 lists the 100 secondary subdomains categorized under the primary domains of Education and Advertising & Marketing in the WritingBench benchmark. For each subdomain, a brief description is provided, explaining the type of writing task involved. This table provides a comprehensive overview of the diverse writing tasks included within the benchmark.

read the caption

Table 9: Subdomains in Education and Advertising & Marketing.

🔼 This table presents a set of guidelines used to refine and enhance the initial writing queries generated by the model. These guidelines aim to increase the diversity and practical applicability of the queries by incorporating specific requirements for length, format, style, personalization, and content. The guidelines are designed to create writing tasks that better represent real-world scenarios.

read the caption

Table 10: Query refinement guidance pool introduced in Section B.5.

🔼 This table describes the prompt used in the query refinement stage of the WritingBench construction. The prompt guides the refinement of initial writing queries generated by LLMs, incorporating specific requirements and considering factors like length, format, style, and content. The prompt is structured to ensure consistency and to guide the annotators to generate high-quality writing queries that align with real-world writing scenarios.

read the caption

Table 11: Query refinement prompt introduced in Section 3.1.1.