PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino

2026-06-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

PACUTE is a new diagnostic benchmark comprising 4,600 tasks designed to evaluate morphological understanding in Filipino. This language features productive infixation, reduplication, and diacritic-driven lexical distinctions. Large language models (LLMs) often struggle with these structures due to subword tokenization, which obscures character-level and morphological boundaries. PACUTE employs a hierarchical diagnostic framework with six compositional levels to pinpoint where this understanding fails. Evaluation of open-weight LLMs showed near-chance performance on morpheme decomposition. Frontier commercial models performed better, recovering individual affixes, but remained significantly below character-level ceilings on compositional tasks like morpheme transformations and syllabification. The research identifies productive morphological composition, not just character access, as the primary bottleneck for Filipino word-structure understanding.

Key takeaway

For NLP engineers working with morphologically rich languages, especially those with non-concatenative structures like Filipino, current LLM tokenization and morphological understanding are significant hurdles. You should prioritize evaluating models using benchmarks like PACUTE to identify specific weaknesses in compositional morphology. Consider developing custom tokenization strategies or architectural modifications that explicitly handle infixes, reduplication, and diacritics to improve performance beyond simple affix recovery.

Key insights

Subword tokenization hinders LLM understanding of complex morphology, especially in languages like Filipino.

Principles

• Subword tokenization obscures morphological structure.
• Non-concatenative morphology challenges standard tokenizers.
• Productive morphological composition is a key bottleneck.

Method

PACUTE uses a hierarchical diagnostic framework with six compositional levels to localize where morphological understanding breaks down in LLMs.

In practice

• Evaluate LLMs on morphological understanding using PACUTE.
• Focus on compositional tasks for complex morphology.
• Address subword tokenization limitations for inflected languages.

Topics

• PACUTE
• Filipino Language
• Morphological Understanding
• Large Language Models
• Subword Tokenization
• Non-concatenative Morphology
• Diagnostic Benchmarks

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew — Tokenization strategy critically impacts Transformer models' ability to represent complex morphology, especially in non-concatenative languages.
• Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese — LLMs struggle with flexible phonological understanding despite recalling pronunciations, indicating a research gap.
• AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages — Models exhibit a significant "syntax gap" in processing African languages, struggling with their structural diversity.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• Pattern-and-root inflectional morphology: the Arabic broken plural — The model reverses traditional Semitic morphology, prioritizing patterns over roots for Arabic noun inflection.
• Compression-based Language Complexity under Register Variation in Portuguese — Compression-based metrics effectively measure linguistic complexity and register variation across languages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.