CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield introduces three complementary techniques for training compute-efficient language models, validated on CHERRY-1.8B, a Korean foundation model. Selective Ground Truth Token Training (SGT) focuses supervision on ~15% of semantically crucial tokens, achieving 4.5x per-supervised-token efficiency. Depth compression reduces a 48-layer, 1B-parameter transformer to 6 layers (227M) with recurrent unrolling, yielding a 2.5x parameter reduction while maintaining performance. Finally, a Mixture of Efficient Experts (MoEE) fuses several compressed models, improving loss to 2.789 compared to 2.926 for the best single compressed model at comparable active parameters. These methods aim to maximize utilization of existing capacity rather than solely scaling resources.

Key takeaway

For AI Engineers or ML teams developing language models under tight resource budgets, you should investigate CHERRY's integrated approach to maximize compute efficiency. Concentrating supervision on semantically decisive tokens via SGT offers 4.5x per-token ROI, while depth compression and recurrent unrolling can reduce parameters by 2.5x without significant performance loss. This allows you to build sovereign models with frontier capabilities at a fraction of the cost, especially for domain-specific applications.

Key insights

Compute-efficient LLM training is achieved by concentrating supervision on semantic tokens and architecturally optimizing models via compression and expert fusion.

Principles

• Positive gradient coupling improves unsupervised tokens.
• Linguistic coherence is crucial for wave propagation.
• Transformer depth redundancy allows significant compression.

Method

CHERRY employs selective training (SGT), adjacent-layer merging for depth compression, recurrent unrolling to restore effective depth, and expert fusion (MoEE) with SGT-guided distillation from frontier teachers.

In practice

• Induce meta-cognitive behaviors like self-correction with minimal SSGT supervision.
• Use adjacent-layer merging for 2.5x parameter reduction in LLMs.
• Apply SGT distillation for ~5x cost reduction in knowledge transfer.

Topics

• Selective Ground Truth Token Training
• Depth Compression
• Mixture-of-Experts
• Recurrent Neural Networks
• Foundation Models
• LLM Efficiency
• Sovereign AI

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Progressive Cramming: Reliable Token Compression and What It Reveals — Perfect token reconstruction through cramming does not guarantee semantic utility or preserved downstream model capabilities.
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models — Expert Tying reduces MoE LLM memory footprint by almost 2x by sharing expert parameters across transformer layers with minimal performance impact.
• CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — Efficient language model training is achieved through selective supervision, depth compression, and expert fusion.
• Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models — Adaptive token compression for time series LLMs improves efficiency and performance by leveraging asymmetric information structures.
• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs — Layer-wise compute allocation based on hidden-state similarity enables efficient depth-sparse LLMs without full retraining.
• VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression — Pretrained vision-language models can intrinsically compress visual tokens, enhancing efficiency and potentially improving base model performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.