Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

2026-06-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A novel semantic pre-training framework enhances the Tsetlin Machine (TM) for interpretable text classification by transferring knowledge from pre-trained language models (PLMs) like BERT. The method involves two stages: first, unlabeled text samples are embedded using a PLM and grouped into 200 semantic clusters via K-means or Top2Vec. These cluster–sample pairs then pre-train a Non-Negated Tsetlin Machine (NTM) with enhanced Type I feedback, which learns interpretable semantic keywords. In the second stage, labeled samples are augmented with these cluster descriptors to fine-tune a standard TM. This approach significantly outperforms vanilla and embedding-based TMs, achieving accuracy competitive with BERT-based models, often within 1-2% of BERT-large on datasets such as AG-News, NYT, DBpedia, R8, and R52, while preserving full interpretability. Selecting 20 high-confidence keywords further optimizes performance.

Key takeaway

For AI Scientists and Machine Learning Engineers building interpretable text classifiers in high-stakes domains, you can now achieve performance competitive with BERT-large models without sacrificing transparency. This framework allows you to deploy robust, explainable Tsetlin Machines by pre-training them with semantic clusters from language models. Consider implementing this approach to meet stringent interpretability requirements in fields like legal or medical text analysis, ensuring both accuracy and clear decision-making logic.

Key insights

Pre-training Tsetlin Machines with semantic clusters from language models achieves BERT-competitive accuracy while maintaining full interpretability.

Principles

• Tsetlin Machines can balance high accuracy with inherent interpretability.
• Semantic pre-training effectively transfers contextual knowledge from PLMs to Boolean models.
• Non-negated clauses in TMs enhance interpretability and semantic pattern learning.

Method

Embed unlabeled text with a PLM, cluster into semantic groups (e.g., 200), then pre-train a Non-Negated Tsetlin Machine (NTM) with cluster-sample pairs to extract high-confidence semantic keywords. Augment labeled data with these keywords to fine-tune a vanilla TM.

In practice

• Utilize 200 clusters for optimal semantic coherence when pre-training TMs.
• Select around 20 high-confidence keywords per cluster to maximize fine-tuning performance.
• Apply this NTM pre-training for text classification in domains requiring high transparency.

Topics

• Tsetlin Machine
• Explainable AI
• BERT
• Semantic Clustering
• Text Classification
• Natural Language Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability — Combining semantic clustering from PLMs with Tsetlin Machines yields interpretable, high-performing text classification without embeddings.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• The Paper That Funded a Fortune — Data-driven word clustering, maximizing mutual information, preceded modern embeddings and influenced quantitative finance.
• How Machines Understand Words: Token Classification in NLP — Token classification is foundational for NLP, enabling machines to interpret language by labeling individual words and phrases.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.