Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language

2026-06-15 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

bangla-smollm-135m is a compact 135-million parameter decoder-only foundational large language model specifically designed for efficient language modeling in the Bangla script. Developed to address the computational challenges of deploying large models on edge, mobile, and decentralized hardware for low-resource, non-Latin scripts, this model employs a deterministic intersect-and-append token merging strategy. This method, applied between TituLLMs and SmolLM2-135M, effectively mitigates subword script fragmentation without compromising early pretrained parameter states. In zero-shot multi-task benchmark evaluations, including PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU, bangla-smollm-135m demonstrates performance comparable to or exceeding models twice its size, such as Gemma-3-270m, and achieves parity with models in the 1B parameter tier. The model is publicly available at rnnandi/bangla-smollm-135m.

Key takeaway

For NLP Engineers developing LLMs for low-resource languages, you should consider compact, specialized architectures like bangla-smollm-135m. This model demonstrates that strategic token merging can achieve performance parity with 1B parameter models, enabling efficient deployment on computationally constrained edge or mobile systems. Evaluate similar efficiency-focused adaptation techniques to overcome subword fragmentation and reduce infrastructure costs for non-Latin script applications.

Key insights

A compact 135M parameter LLM for Bangla achieves 1B-tier performance by efficiently merging tokens, overcoming low-resource script challenges.

Principles

• Efficiency can surpass scale for low-resource languages.
• Token merging can prevent script fragmentation.
• Preserve early pretrained states during adaptation.

Method

A deterministic intersect-and-append token merging strategy, applied between TituLLMs and SmolLM2-135M, adapts foundational models for non-Latin scripts while preserving early pretrained parameter states.

In practice

• Deploy 135M models on edge/mobile systems.
• Use token merging for non-Latin script adaptation.
• Evaluate with PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, Bangla_MMLU.

Topics

• Foundational LLMs
• Bangla Language Processing
• Low-Resource NLP
• Model Efficiency
• Token Merging Strategy
• Edge AI Deployment

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models — Cross-lingual fairness and tokenization efficiency are not fundamentally at odds in multilingual LLMs.
• Bangla Key2Text: Text Generation from Keywords for a Low Resource Language — A new 2.6M Bangla keyword-text dataset enables supervised keyword-driven text generation for low-resource languages.
• This AI Paper Introduces TinyLoRA, A 13-Parameter Fine-Tuning Method That Reaches 91.8 Percent GSM8K on Qwen2.5-7B — TinyLoRA enables high LLM performance with only 13 parameters by leveraging reinforcement learning.
• Granite 4.1 LLMs: How They’re Built — Granite 4.1 LLMs achieve strong performance through rigorous multi-stage data curation and reinforcement learning, even with smaller dense architectures.
• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-aware LLM evaluation requires analyzing full training trajectories, not just final performance metrics.
• Instant LLM Updates with Doc-to-LoRA and Text-to-LoRA — Hypernetworks generating on-demand LoRA adapters enable instant LLM customization and information internalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.