Tokenization Granularity and Medical Term Representations in Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study by Vojtech Lanz and Pavel Pecina investigates how tokenization granularity impacts the representation of medical terminology within existing pretrained language models. Diverging from prior work on contextualized or fine-tuned models, this research focuses on isolated term representations. The authors developed an intrinsic definition retrieval task using UMLS term-definition pairs, with comparisons to WordNet. Findings reveal that despite substantial fragmentation of medical terminology, models largely maintain semantic alignment between terms and their definitions. However, tokenization granularity still correlates with retrieval performance, indicating that effects previously observed in downstream biomedical tasks are reflected even in isolated term representations. Encoder models particularly benefit from whole-token preservation, while decoder LLMs exhibit tokenization effects mainly at deeper retrieval ranks.

Key takeaway

For NLP Engineers developing medical language models, understanding tokenization granularity is crucial. Your choice of tokenization strategy directly impacts how well models represent medical terms and perform on retrieval tasks. If you are working with encoder models, prioritize methods that preserve whole tokens to enhance performance. For decoder LLMs, be aware that tokenization effects might only become apparent at deeper retrieval ranks, requiring more thorough evaluation.

Key insights

Tokenization granularity impacts medical term representations in LMs, even for isolated terms, affecting retrieval performance.

Principles

• Tokenization granularity affects term representation.
• Semantic alignment is robust to fragmentation.
• Retrieval performance correlates with granularity.

Method

The study introduced an intrinsic definition retrieval task using UMLS term-definition pairs, comparing results against WordNet to assess semantic alignment and retrieval performance.

In practice

• Prioritize whole-token preservation for encoders.
• Evaluate tokenization for medical NLP tasks.
• Consider deeper ranks for decoder LLM analysis.

Topics

• Tokenization Granularity
• Medical Terminology
• Language Models
• Encoder-Decoder Architectures
• Definition Retrieval
• UMLS Dataset

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• Token Embeddings : From numbers to meaning — Token embeddings convert arbitrary token IDs into dense, learned vectors that capture semantic relationships through contextual co-occurrence.
• (1D) Ordered Tokens Enable Efficient Test-Time Search — Coarse-to-fine tokenization improves test-time search steerability in autoregressive generative models.
• Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew — Tokenization strategy critically impacts Transformer models' ability to represent complex morphology, especially in non-concatenative languages.
• Data Augmentation for Historical NER: A Systematic Comparison of Lexical and LLM-based Approaches — This paper systematically compares lexical and LLM-based data augmentation for Historical Named Entity Recognition.
• What is Tokenization? All you need to know — Tokenization is a critical NLP decision impacting model efficiency, robustness, and language representation, not just a preprocessing step.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.