Language Models as Measurement Apparatus for Culture

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cultural Studies, AI Research Methodology · Depth: Expert, quick

Summary

The paper "Language Models as Measurement Apparatus for Culture" posits that NLP's application to cultural phenomena constitutes a material-discursive practice, where the measurement apparatus—comprising the language model, data, annotation, and evaluation—actively participates in shaping the cultural reality it aims to quantify, rather than merely observing it. Drawing on Karen Barad's concept of the agential cut, the author explains how substantive design choices within the apparatus establish contingent boundaries between the phenomenon and the instrument. This boundary is inherently entangled because language models have already internalized much of the cultural material they are designed to measure. The analysis is supported by three case studies involving television and film dialogue, which explore structure, interaction, and deviation, alongside three examinations of the apparatus itself, focusing on the erasure of cultural markers, attunement to historical material, and agency within an agentic workflow. The paper advocates for a research program that is theory-driven, empirically rigorous, and culturally contingent, emphasizing each agential cut as a conscious methodological and ethical commitment.

Key takeaway

For AI Scientists and Research Scientists designing cultural measurement systems, recognize that your language model apparatus actively shapes the cultural reality it quantifies. You must treat each "agential cut"—the boundary between phenomenon and instrument—as a conscious methodological and ethical commitment. Carefully consider how your model's design choices and internalized cultural material influence measurement outcomes, ensuring empirical rigor and cultural contingency.

Key insights

Language models actively constitute the cultural reality they measure, not just record it.

Principles

• Measurement apparatus shapes cultural reality.
• Agential cuts define phenomenon-instrument boundaries.
• LMs internalize cultural material, creating entanglement.

Method

The paper proposes a research program for cultural measurement using LMs that is theory-driven, empirically rigorous, and culturally contingent, treating each "agential cut" as a conscious methodological and ethical commitment.

In practice

• Analyze TV/film dialogue for structure.
• Examine apparatus for cultural marker erasure.
• Assess LM attunement to historical data.

Topics

• Language Models
• Cultural Measurement
• NLP Ethics
• Agential Cut
• Material-Discursive Practice
• Television Dialogue Analysis

Best for: AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• How to Shape the Engineering Culture in Software Companies — David Grizzanti, at InfoQ Dev Summit Boston, proposed an anthropological approach to shaping engineering culture, stressing that deep understanding of an organization's existing behaviors, power dynam…
• ConlangCrafter Turns AI to Imagining Languages — ConlangCrafter generates diverse, rule-abiding constructed languages, outperforming general LLMs in consistency and diversity.
• AI will make language barriers disappear – and diminish our understanding of other cultures — AI translation, while efficient, risks eroding genuine cultural understanding and the human element of communication.
• Agentic AI Translate: An Agentic Translator Prototype for Translation as Communication Design — Translation in the GenAI era shifts from text conversion to communication design, guided by explicit specifications.
• Machine ‘culture’ beyond language — Language reliably shifts an LLM's "psychological profile," influencing social orientation and cognitive style.
• The social sciences need tools for the 21st century | Letters — Social science needs new tools and methods to improve replicability and advance understanding of human behavior.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.