LUMI: Tokenizer-Agnostic LLM-Based Lossless Image Compression

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

LUMI, an LLM-based Unified Model-agnostic lossless Image compression framework, introduces a novel approach to lossless RGB image compression by decoupling it from traditional tokenizer behavior. Unlike methods that convert pixel data into text tokens for LLMs, LUMI employs a pixel embedding module to map raw intensity and channel information directly into the LLM's continuous embedding space. It integrates intra-patch position encoding to preserve 2D spatial structure and utilizes a 256-way prediction head for native pixel alphabet probabilities. Only the pixel embedding, position encoding, soft-prefix parameters, and prediction head are trained, keeping the LLM backbone frozen. Experiments on natural, medical, and remote-sensing image benchmarks, using LLaMA, Qwen, and Gemma backbones, demonstrate LUMI's competitive compression rates, unified interface across tokenizer families, and enhanced cross-domain robustness compared to existing tokenizer-based LLM compression baselines.

Key takeaway

For Machine Learning Engineers developing image compression solutions, if you are evaluating LLM-based methods, consider LUMI's tokenizer-agnostic framework. Your current tokenizer-dependent approaches may limit cross-domain robustness and require model-family-specific adaptations. By adopting pixel embedding modules and intra-patch position encoding with frozen LLM backbones, you can achieve competitive compression rates and a more unified interface across diverse image types, including medical and remote-sensing data. This shifts your focus to pixel-space adaptation rather than language-symbol modeling.

Key insights

LUMI enables tokenizer-agnostic lossless image compression by adapting frozen LLMs directly in pixel space, enhancing robustness.

Principles

• LLMs can estimate image probabilities without text tokenization.
• Pixel-space adaptation of frozen LLMs is effective.
• Continuous embeddings can represent raw pixel data.

Method

LUMI maps raw pixel data to an LLM's continuous embedding space, adds intra-patch position encoding, and uses a 256-way prediction head, training only these specific modules.

In practice

• Implement pixel embedding modules for raw data input.
• Integrate intra-patch position encoding for spatial structure.
• Adapt frozen LLMs for non-language domain tasks.

Topics

• LLM Image Compression
• Tokenizer-Agnostic
• Pixel Embeddings
• Lossless Compression
• Foundation Models
• Spatial Encoding

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• LUMI: Tokenizer-Agnostic LLM-Based Lossless Image Compression — LUMI enables tokenizer-agnostic, LLM-based lossless image compression by directly embedding pixels into the model's continuous space.
• Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings — LLM unembedding matrices encode an "edge spectrum" subspace biasing embeddings towards frequent, uninformative tokens, which EmbedFilter removes.
• LLM-based Source Code Compression via Thresholded Symbol Ranking — Bounding LLM symbol predictions to top-$T$ ranks significantly improves source code compression ratio and throughput.
• SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices — SigmaScale enhances SVD-based LLM compression by learning activation-aware scaling matrices, improving efficiency.
• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data — LLMs can learn dictionary encodings in-context, enabling lossless prompt compression without fine-tuning and preserving analytical accuracy.
• Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization — Byte-Prefix Marginalization (BPM) enables robust cross-tokenizer on-policy distillation by aligning teacher and student distributions in a shared byte space.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.