I Built a Prompt Compressor. The Most Useful Thing It Found Was Nothing

2026-07-09 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

A new prompt compression tool, TokenLens, explores reducing token costs for Large Language Models by selectively dropping "unimportant" tokens, akin to lossy image compression. The underlying technique, LLMLingua-2, employs a 500-million-parameter bidirectional encoder, trained on GPT-4 compressed transcripts, to classify and extract essential tokens, running efficiently on a MacBook Air CPU with 16GB of RAM. TokenLens functions as a local proxy for Anthropic APIs, measuring token usage and optionally applying compression while strictly avoiding cached prefixes to prevent increased billing. A key finding from its use in a real coding session revealed minimal savings, as prompt caching already effectively compresses system prompts and conversation history. The tool demonstrates significant potential for large, uncached prose inputs like RAG contexts or extensive specifications, though further evaluation of output quality at aggressive compression rates is still needed.

Key takeaway

For AI Engineers optimizing LLM token costs, recognize that prompt caching already handles significant compression for interactive sessions. Your efforts should focus on applying lossy prompt compression techniques, like those in LLMLingua-2, to large, uncached prose inputs such as RAG contexts or extensive specifications. Use tools like TokenLens to safely measure and compress, ensuring they avoid busting existing caches. Always evaluate the impact on model output quality, especially when using aggressive compression rates, to prevent unintended performance degradation.

Key insights

Prompt compression utilizes LLM tolerance for terse input to reduce token costs by extracting essential meaning.

Principles

• Compression is a bet about where information actually lives.
• LLM prompt compression is lossy, relying on model robustness.
• Cache-aware compression is critical to avoid increased costs.

Method

LLMLingua-2 trains a small bidirectional encoder (500M params) on GPT-4 generated token labels (kept/dropped) to perform a fast classification pass, extracting important tokens in original order.

In practice

• Apply prompt compression to large, uncached prose inputs.
• Use local proxies like TokenLens for safe, cache-aware compression.
• Prioritize prompt caching for interactive LLM sessions.

Topics

• Prompt Engineering
• LLM Cost Optimization
• Data Compression
• LLMLingua-2
• TokenLens
• RAG Systems

Code references

• jmcdonald69124/tokenlens

Best for: AI Engineer, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• What is Prompt Caching? Optimize LLM Latency with AI Transformers — Prompt caching stores precomputed LLM input representations (KV pairs) to reduce latency and cost for repeated prompt prefixes.
• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data — LLMs can learn dictionary encodings in-context, enabling lossless prompt compression without fine-tuning and preserving analytical accuracy.
• You’re Not Wasting Tokens by Writing Too Much. You’re Wasting Them by Remembering Too Much — High AI token costs stem from bloated context, not just prompt length.
• The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts — LLM prompt compression algorithms misalign linguistic perplexity with task importance, preserving code syntax but pruning critical math numbers.
• Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing — Proactive, edge-side prompt rewriting with a local LLM significantly reduces token costs for coding agents.
• How an Unexpected Reddit Spike Forced Me to Learn Prompt Caching the Hard Way — Multi-agent LLM pipelines offer significant prompt caching benefits, especially when context is passed sequentially between agents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.