Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

2026-05-24 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cheminformatics · Depth: Expert, extended

Summary

A controlled comparison of byte-pair encoding (BPE) and Unigram-LM tokenizers on chemistry SMILES strings reveals they do not converge, building near-disjoint subword vocabularies. Across 22 matched conditions, including diverse, drug-like, and natural-products corpora, and both pre-tokenization boundary policies, the cross-algorithm Jaccard overlap on learned pieces never exceeds 0.161, and is at most 0.05 when weighted toward high-frequency pieces. Unigram-LM segments held-out molecules into 29–41% more tokens, indicating agreement on "where" to cut but not "how deeply". BPE's segmentation is a strict coarsening of Unigram-LM's on 80–99% of molecules. This separation persists even at eight times the headline vocabulary scale (V=256, 512, 1024), establishing the subword algorithm as a critical modeling decision, not a default.

Key takeaway

For AI Scientists and Machine Learning Engineers developing chemical language models, selecting a SMILES tokenizer is a critical design decision, not a default. You should explicitly evaluate both BPE and Unigram-LM based on your model's specific requirements for token granularity and vocabulary composition. This choice directly influences effective sequence length, embedding learnability, and the chemical interpretability of learned representations, impacting downstream model performance.

Key insights

BPE and Unigram-LM tokenizers yield near-disjoint vocabularies and different segmentation granularities for chemistry SMILES.

Principles

• Subword algorithm choice significantly impacts chemistry SMILES tokenization.
• Natural language tokenizer priors may not transfer to chemistry SMILES.
• Vocabulary overlap is minimal, especially for high-frequency pieces.

Method

Controlled comparison of BPE and Unigram-LM on a fixed 165-token OpenSMILES base across corpus typologies, vocabulary sizes, and boundary policies.

In practice

• Consider Unigram-LM for finer-grained, near-atomic SMILES segmentation.
• Use BPE for coarser, larger-piece SMILES tokenization, spanning rings.
• Evaluate tokenizer choice based on downstream model granularity needs.

Topics

• Chemistry Language Models
• SMILES Tokenization
• Byte-Pair Encoding
• Unigram-LM
• Subword Algorithms
• Molecular Representation

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Augmenting Molecular Language Models with Local $n$-gram Memory — Integrating local $n$-gram memory via MolGram efficiently resolves the locality gap in molecular language models, improving performance with fewer parameters.
• SupraBench: A Benchmark for Supramolecular Chemistry — Current LLMs lack robust supramolecular chemistry reasoning, showing significant headroom across all benchmark tasks.
• Agentic Molecular Recovery via Molecule-Aware Exploration — Molecular recovery should prioritize identity preservation and semantic alignment over mere validity restoration for invalid SMILES.
• A Systematic Evaluation of Molecular Mixture Behavior Prediction — Evaluating molecular mixture ML models requires decomposing error into pure-compound and non-ideal interaction components, moving beyond absolute accuracy.
• De novo molecular generation with optical property preconditioning at the token level — Token-conditioned autoregressive models can generate OLED molecules with targeted optical properties, but control varies by chemical motif.
• Tokenization — The Making of RAG (Part 2) — Tokenization is RAG's foundational step, critically influencing accuracy, cost, and context preservation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.