Your Language Model Cannot Say Certain Sentences.

2026-06-21 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

This article asserts that language models, particularly those utilizing a softmax layer for next-word prediction, are inherently limited in their output capabilities. It claims that certain sentences are mathematically impossible for these models to generate, a constraint stemming not from training data or model size, but from the rank of a matrix involved in the final computational step. The author intends to demonstrate this impossibility through a step-by-step, manual proof using a minimal four-word vocabulary and a very small model, emphasizing that the final stage of a language model's operation can be fully understood via basic arithmetic. This fundamental limitation applies to all models ending in a softmax.

Key takeaway

For AI Scientists and Machine Learning Engineers designing or evaluating language models, you should recognize that models ending in a softmax layer possess inherent mathematical limitations on their output. This means certain word sequences are impossible, not just improbable. Understanding this matrix rank constraint is crucial for debugging unexpected model behaviors or when assessing the true generative capacity of your systems. Consider this fundamental ceiling when interpreting model failures or successes.

Key insights

Language models with softmax layers are mathematically forbidden from generating certain sentences due to matrix rank limitations.

Principles

• Softmax-based LMs have inherent output ceilings.
• Matrix rank dictates prediction possibility.
• Mathematical impossibility, not unlikelihood.

Method

The proposed proof involves manual arithmetic with a tiny model and a four-word vocabulary to demonstrate matrix rank limitations on next-word predictions.

Topics

• Language Models
• Softmax Layer
• Matrix Rank
• Next-Word Prediction
• Generative AI Limitations
• Mathematical Proof

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Yes it's just doing compression. No it's not the diss you think it is. — Optimizing language models for next-word prediction via log-likelihood is a form of data compression, which can be linked to understanding.
• Language in the brain — Statistical learning and synaptic plasticity are insufficient for modeling human language competency.
• Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective — Softmax attention behaves linearly in the large-prompt, infinite-token limit, simplifying theoretical analysis.
• MIT study explains why scaling language models works so reliably — Superposition, where concepts overlap in limited dimensional space, mechanistically explains language model scaling laws.
• A Neural Approach to Discourse Relation Signal Detection — A neural network and \"delta-softmax\" metric quantify discourse signal strength and ambiguity, improving relation identification.
• Relational Linear Properties in Language Models: An Empirical Investigation — A new Kullback-Leibler divergence probing method efficiently tests relational linearity in language models, showing variations across models and sensitivity to phrasing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.