Convolution for Large Language Models

2026-07-20 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study investigates integrating lightweight depthwise convolutions into large language models (LLMs) to provide local inductive bias, complementing Transformer self-attention. Macro-level ablation across 17 locations in a Qwen3 Transformer block found optimal results when convolution was applied to projected queries, keys, and values before attention. A subsequent micro-level analysis favored a residual depthwise convolution with a kernel size of 3, without additional normalization or activation. This design consistently improved average accuracy on seven downstream benchmarks across Qwen3 models and various pre-training data budgets, adding less than 0.01% parameters. Representation-level analysis indicated the convolution makes repeated token IDs more sensitive to their immediate context.

Key takeaway

For machine learning engineers optimizing large language models, you should consider integrating lightweight depthwise convolutions. Applying a residual depthwise convolution with a kernel size of 3 to projected queries, keys, and values before self-attention can improve average accuracy on downstream benchmarks. This architectural modification adds less than 0.01% parameters, offering a highly efficient way to enhance local context modeling and token sensitivity in your LLM deployments.

Key insights

Depthwise convolutions enhance LLM locality with minimal parameters, improving context sensitivity.

Principles

• Convolutions provide local inductive bias.
• Residual depthwise convolution (k=3) is effective.
• Apply convolution before attention for best results.

Method

Ablation study compared 17 convolution locations in a Qwen3 Transformer block, favoring application to projected queries, keys, and values before attention, using residual depthwise convolution with kernel size k=3.

In practice

• Integrate k=3 depthwise convolution pre-attention.
• Improve LLM accuracy with <0.01% parameter increase.
• Enhance token context sensitivity.

Topics

• Large Language Models
• Transformers
• Depthwise Convolution
• Self-Attention
• Model Architecture
• Inductive Bias
• Qwen3

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models — Scaling LLMs by expanding computation along the sequence-length dimension with token streams offers a fixed-backbone improvement path.
• Towards Selective Depth Mixing: Attention Residuals for Stable, Reusable Features in Deep Transformers — Novel architectural and inference techniques enhance LLM efficiency and performance by optimizing residual connections, KV cache management, and sampling.
• Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification — Integrating LLMs with VLMs to semantically prune GCN image graphs improves semi-supervised classification accuracy.
• DeepSeek-V4: The Interesting Part Is the Attention Architecture — DeepSeek-V4 uses hybrid compressed attention and mHC to enable million-token context efficiently.
• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• Can Large Language Models Understand Context? — LLMs struggle with nuanced context understanding, especially pre-trained dense models and those with aggressive quantization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.