Evaluating Generative AI: Did Astral Codex Ten Win His Bet on AI Progress?

2026-02-19 · Source: Surge AI Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Data Science & Analytics · Depth: Advanced, long

Summary

This analysis evaluates the compositionality capabilities of advanced image generation models, specifically DALL·E 2 and Google's Imagen, in response to complex prompts. Initially, DALL·E 2 struggled with multi-object composition, often misinterpreting relationships and attributes (e.g., a red cat on a blue dog next to a purple lake, with a black pig flying). The article then examines a bet made by Scott of Astral Codex Ten, who claimed that newer, larger models would resolve such failures. Scott declared victory after Imagen generated images for five complex prompts, modified to feature robots due to Imagen's Terms of Service. However, human evaluators (Surgers) found that only one of Imagen's generations (robot in factory looking at a cat in a top hat) clearly met the prompt's criteria, with others exhibiting significant inaccuracies in object placement, attributes, or scene context. A direct comparison between DALL·E and Imagen showed mixed results, with Surgers preferring DALL·E for two prompts and Imagen for three, though one of Imagen's wins was deemed ambiguous.

Key takeaway

For AI scientists and computer vision engineers developing or deploying image generation models, this analysis highlights the persistent challenge of compositionality. Your models, even advanced ones like Imagen, may still misinterpret complex multi-object prompts, leading to inaccurate outputs. You should integrate robust human evaluation into your development and testing workflows to identify and address these subtle compositional failures, rather than relying solely on internal metrics or developer self-assessment.

Key insights

Human evaluation reveals current image generation models still struggle with complex compositional prompts, despite scaling improvements.

Principles

• Compositionality remains a key challenge for AI.
• Scaling alone does not guarantee compositional understanding.

Method

Human evaluators assessed image accuracy against complex prompts, identified misinterpretations, and compared model performance using Likert scales and qualitative feedback.

In practice

• Use human evaluation for creative AI model deployment.
• Design prompts carefully to avoid ambiguity.
• Anticipate compositional errors in complex image generation.

Topics

• Text-to-Image Synthesis
• Compositionality
• DALL-E
• Imagen
• Human Evaluation

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, AI Researcher, AI Engineer, Data Scientist

Related on AIssential

• Why Most AI Art Prompts Fail. And How to Write Better Ones — AI art generation is a linguistic process where precise prompts are crucial for consistent, intentional image outcomes.
• When AI goes haywire: the case of the skyscraper and the slide trombone — Generative AI lacks common sense and real-world understanding, relying solely on statistical patterns from training data.
• Bridging Creative Intent and Visual Quality: Creator-Driven Recurrent Video Generation with Agentic Feedback Loops — CHIEF enables creator-driven, iterative video generation by integrating human creative direction with agentic, subjective feedback loops.
• AI, Design, and the Power of Open Models — Image generation models must prioritize user control, accurate text, and customizability for professional design workflows.
• Novel Problems in VLA [R] — Novel VLA research opportunities exist in addressing current technique limitations, particularly in detailed image/video annotation for generative models and simulation-based action learning.
• MIT Professor on Generative AI & Computer Vision: Part 2 — AI's future involves integrating multimodal generative models for robust simulation and human-like intelligence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Surge AI Blog.