Narrative Landscape: Mapping Narrative Dispositions Across LLMs

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study by Jung et al. introduces a quantitative framework to profile Large Language Model (LLM) dispositions, defined as stable, model-specific output regularities under controlled elicitation. This framework employs a structured narrative constraint-selection task, administered across six frontier models and three distinct instruction types. Disposition is operationalized through two key dimensions: "consistency," measured by Jaccard similarity of cross-replication selection overlap, and "diversity," quantified using the inverse Simpson index for dispersion across options. The research further presents "Narrative Landscape," a PCA-based visualization tool that maps each model's selection profile into a shared space for direct comparison. Results indicate a clear rigidity–exploration spectrum among model families and demonstrate that instruction types significantly alter the geometry of selection spaces, even when scalar metrics appear similar, highlighting qualitative differences in selection topologies.

Key takeaway

For NLP Engineers evaluating LLM behavior or selecting models for narrative generation, you should move beyond scalar metrics to understand model dispositions. Recognize that instruction types fundamentally shift output topologies, even if consistency or diversity scores seem similar. Utilize quantitative frameworks like Narrative Landscape to visualize and compare model-specific rigidity-exploration spectrums, ensuring your chosen LLM aligns with desired narrative consistency and diversity requirements.

Key insights

LLM dispositions can be quantitatively profiled using consistency and diversity metrics, revealing model-specific output regularities.

Principles

• LLM output regularities are model-specific dispositions.
• Instruction types alter LLM selection space geometry.
• Scalar metrics can mask qualitative output differences.

Method

Profile LLM dispositions using a structured narrative constraint-selection task. Measure consistency via Jaccard similarity and diversity via inverse Simpson index. Visualize with PCA-based Narrative Landscape.

In practice

• Compare LLM output consistency and diversity.
• Analyze how instruction types shift LLM behavior.
• Use PCA to visualize model selection profiles.

Topics

• LLM Disposition
• Narrative Generation
• Model Evaluation
• Jaccard Similarity
• Inverse Simpson Index
• PCA Visualization

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey — LLMs bridge abstract narrative theories with NLP, enabling scalable story generation and understanding despite definitional challenges.
• Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact — Apparent psychological profiles of LLMs are primarily measurement artifacts driven by directional response bias, not inherent traits.
• Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models — LLM consistency for clinical tasks varies significantly, demanding careful evaluation beyond single-output metrics.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative content in LLM pretraining data is measurable and unevenly distributed, impacting model capabilities.
• Do Large Language Models Always Tell The Same Stories? — Large Language Models consistently produce narratives that are more homogeneous than human-written stories, even with common mitigation efforts.
• SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations — Effective LLM mediation requires multi-dimensional evaluation across diverse socio-cognitive contexts to reveal adaptation gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.