Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework

2026-04-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

This work introduces a quantitative framework for evaluating narrative quality by analyzing linguistic features. It proposes a methodology that extracts 33 quantitative linguistic features, categorized into lexical, syntactic, and semantic groups, to automatically assess narratives. An experiment on a corpus of 23 books, comprising both canonical masterpieces and self-published works, demonstrated the system's ability to cluster narratives and distinguish between professionally edited and self-published texts with high accuracy using a similarity matrix. The methodology was further validated against a human-annotated dataset, where it significantly outperformed traditional story-level evaluation metrics, confirming the efficacy of linguistic features in determining narrative quality.

Key takeaway

For research scientists developing automated content analysis tools, this framework offers a robust method for objectively evaluating narrative quality. You should consider integrating quantitative linguistic features, such as those proposed, into your models to enhance the precision of narrative assessment and distinguish between different levels of textual professionalism.

Key insights

Linguistic features can quantitatively assess narrative quality, distinguishing professional from self-published works.

Principles

• Narrative quality has quantifiable linguistic indicators.
• Linguistic features can differentiate text professionalism.

Method

The method extracts 33 quantitative linguistic features (lexical, syntactic, semantic) from narratives, then uses a similarity matrix to cluster and evaluate texts, validated against human annotations.

In practice

• Automate narrative quality assessment.
• Identify professionally edited texts.
• Analyze linguistic patterns in literature.

Topics

• Quantitative Stylistic Framework
• Narrative Quality Assessment
• Linguistic Feature Extraction
• Lexical Features
• Syntactic Features

Best for: Research Scientist, AI Scientist, NLP Engineer, Data Scientist

Related on AIssential

• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework — Linguistic features can quantitatively assess narrative quality, distinguishing professional from self-published works.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative structure in web-scale LLM pretraining data is measurable and unevenly distributed, impacting model reasoning.
• Which sentence is doing the most work in your favourite novel ? I tried to find out. — Machine learning can identify semantically distinctive sentences in texts, revealing structural or narrative pivots.
• How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups — Extrinsic discourse evaluation reveals intrinsic MT quality doesn't guarantee downstream success, especially in goal-oriented interactions.
• A Cross-Genre Analysis of Discourse Relation Signaling in the GUM Corpus — The paper analyzes discourse relation signaling across diverse genres in the GUM Corpus.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.