HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

2025-01-31 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This paper introduces HoWToBench, a large-scale Chinese writing benchmark, and Tree-of-Writing (ToW), a novel evaluation framework for assessing Large Language Models' (LLMs) human-level writing capabilities. HoWToBench comprises 12 genres and 1302 instructions across three task categories: contextual completion, outline-guided writing, and open-ended generation, curated from expert-written sources with a 96.85% quality pass rate. ToW addresses the "Negotiation Inconsistency" in LLM-as-a-judge methods by explicitly modeling the aggregation weights of sub-features in a tree-structured workflow, achieving a 0.93 Pearson correlation with human judgments. The framework demonstrates robustness against textual disturbances, unlike traditional overlap-based metrics and popular LLM-as-a-judge practices. The study also reveals a negative correlation between input length and content-related scores in guided tasks, indicating that simply increasing input information does not improve nuanced writing quality.

Key takeaway

For research scientists evaluating LLM writing, adopting the Tree-of-Writing (ToW) framework and HoWToBench benchmark can significantly enhance the reliability and human alignment of your assessments. You should move beyond simplistic metrics and implicit LLM-as-a-judge methods to explicitly model evaluation criteria, especially for open-ended and nuanced writing tasks. This approach will provide more robust and interpretable insights into model performance across diverse genres and task complexities.

Key insights

ToW and HoWToBench offer a robust, human-aligned framework for evaluating LLM writing beyond simple instruction-following.

Principles

• Explicitly model evaluation sub-feature aggregation weights.
• Human-level writing evaluation requires multi-dimensional assessment.
• Longer inputs do not guarantee higher quality in nuanced writing.

Method

Tree-of-Writing (ToW) simulates human judgment via a tree-structured workflow, assigning explicit weights to content, format, and impression nodes, then aggregating scores through depth-first traversal.

In practice

• Use ToW for more reliable LLM writing evaluation.
• Consider HoWToBench for diverse Chinese writing tasks.
• Prioritize quality over input length for complex LLM generation.

Topics

• Tree-of-Writing Framework
• LLM Writing Evaluation
• HoWToBench Benchmark
• Human Judgment Correlation
• Text Generation Robustness

Code references

• ZhuoerFeng/ACL2026-Tree-of-Writing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing — Tree-of-Writing (ToW) offers a robust, tree-structured method for evaluating LLM writing, correlating highly with human judgment.
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — LLMs struggle with complex, enterprise-scale PDF-to-JSON extraction, especially as schema breadth increases.
• RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity — LLMs prioritize learned role preferences over dynamic contextual cues in social role conflict scenarios.
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.
• LLMbench: A Comparative Close Reading Workbench for Large Language Models — LLMbench enables close reading of LLM outputs by visualizing probabilistic text generation for humanistic analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.