Calibrated Surprise: An Information-Theoretic Account of Creative Quality

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Science & Research — Mathematics & Computational Sciences, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

The paper "Calibrated Surprise: An Information-Theoretic Account of Creative Quality" (April 2026) introduces an objective, mathematical framework for evaluating creative writing. It defines good writing as "calibrated surprise," where comprehensive constraints (ethos, mythos, lexis, dianoia) narrow the feasible solution space, making the resulting choices appear least predictable from an unconstrained perspective. This framework utilizes Shannon's mutual information, $I(X;Y)=H(X)-H(XY)$, with "calibrated" corresponding to conditional entropy $H(XY)$ approaching zero, and "surprise" to information entropy $H(X)$ increasing. The authors define "accuracy" as the convergence of author intent, reader expectation, and the logic of reality. Computational verification using Qwen1.5-7B log-probabilities on 20 paired literary passages (12 Chinese, 8 English) consistently shows higher mutual information for high-quality texts than for degraded versions, supporting the central prediction. This work establishes a theoretical foundation for a follow-up Creative Quality Alignment (CQA) methodology and a professional evaluation benchmark.

Key takeaway

For AI Scientists and Research Scientists developing or evaluating generative writing models, this information-theoretic framework offers a robust, objective measure of creative quality. You should shift from subjective rubrics or indirect feedback to calibrating models' internal conditional distributions $P(x|y)$ using high signal-to-noise expert Chain-of-Thought data. This approach ensures models concentrate probability mass on truly accurate, non-mediocre writing choices, directly improving creative output alignment.

Key insights

Good creative writing is "calibrated surprise," where full constraints yield rare, accurate choices.

Principles

• Full-dimensional accuracy and mediocrity are mutually exclusive.
• More constraints reduce uncertainty in writing choices.
• Macro decisions naturally contribute more information.

Method

Creative quality is measured by Shannon's mutual information $I(X;Y)=H(X)-H(XY)$, where $X$ is the writing choice and $Y$ is the intersection of full-dimensional reality constraints. $H(XY)\to 0$ (calibration) implies high $H(X)$ (surprise).

In practice

• Use LLM logprobs to approximate an ideal reader's probability judgment.
• Fine-tune LLMs with expert CoT data to calibrate $P(x|y)$ for quality.

Topics

• Information Theory
• Creative Writing Quality
• Mutual Information
• LLM Evaluation
• Calibrated Surprise
• Creative Quality Alignment

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• Calibrated Surprise: An Information-Theoretic Account of Creative Quality — Creative quality arises from "calibrated surprise," where converging constraints yield unpredictable, yet logically sound, choices.
• AI Next Token kaise choose karti hai? — AI selects the next token probabilistically, balancing context fit with controlled randomness for creativity.
• UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs — LLMs struggle significantly with generating samples calibrated to true underlying distributions, hindering their use in simulations.
• InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning — High-quality LLM reasoning traces are informationally dense, showing low uncertainty convergence and monotonic progress in entropy reduction.
• Identifying Information from Observations with Uncertainty and Novelty — Identification and sample complexity are unified across computational, statistical, and PAC-Bayes learning frameworks.
• The stories behind our published research from last year — Gelman's diverse research portfolio emphasizes Bayesian methods, statistical workflow, and critical re-evaluation of scientific practices.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.