CREATE: Testing LLMs for Associative Creativity

2026-03-10 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

CREATE is a new benchmark designed to evaluate the creative associative reasoning capabilities of large language models. This benchmark requires models to generate multiple distinct and meaningful paths connecting different concepts within their parametric knowledge. Models are scored based on the specificity and diversity of these generated paths, rewarding a larger set of strong, varied connections. The task mirrors real-world creative challenges like hypothesis generation, featuring a vast search space while allowing for objective grading. Initial evaluations of frontier models indicate that while stronger models achieve higher creative utility, benchmark saturation remains challenging due to the task's complexity and answer multiplicity. Interestingly, "thinking models" do not consistently outperform others, even with increased token budgets, and current creative prompting techniques offer only marginal improvements.

Key takeaway

For research scientists developing or evaluating large language models, you should consider integrating CREATE into your assessment pipeline to rigorously test associative reasoning. This benchmark offers an objective framework for measuring creative utility, highlighting areas where current models, including "thinking models," still struggle. Focusing on methods that enhance path specificity and diversity will be crucial for advancing model creativity.

Key insights

CREATE evaluates models' creative associative reasoning by generating diverse, specific concept paths.

Principles

• Creativity involves novel, meaningful concept connections.
• High specificity and diversity define strong associative paths.

Method

Models generate multiple distinct paths connecting concepts from their parametric knowledge, scored on specificity and diversity, with higher scores for more strong, diverse paths.

In practice

• Develop methods for associative creativity.
• Test models on hypothesis generation tasks.

Topics

• CREATE Benchmark
• Associative Reasoning
• Creative AI Evaluation
• Frontier Model Performance
• Generative Pathfinding

Best for: Research Scientist, AI Researcher, AI Scientist, Prompt Engineer

Related on AIssential

• Automated Creativity Evaluation of Language Models Across Open-Ended Tasks — The framework offers a scalable, domain-agnostic method to evaluate LLM creativity by separating measurement from task.
• IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization — IntElicit uses dialogue policy optimization to assess contextualized creativity by reducing confounders and rewarding elicitation of participant reasoning.
• The Lovelace Test Revisited — Modern LLMs pass the original Lovelace Test due to the practical impossibility of explaining their complex, emergent outputs.
• DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios — Fine-grained creativity assessment in complex domains can be data-efficiently automated using hierarchical metrics and mixed-granularity LLM training.
• GENIE: A Fine-Grained Measure for Novelty — GENIE offers a fine-grained metric to measure task-specific novelty in LLM outputs, surpassing holistic evaluations.
• Automated Creativity Evaluation of Language Models Across Open-Ended Tasks — A new domain-agnostic framework automates large language model creativity evaluation across diverse open-ended tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.