Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

2025-04-01 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

Code2LoRA is a hypernetwork framework designed to inject repository-level context into code language models (LLMs) with zero inference-time token overhead. It addresses the limitations of existing methods like RAG or per-repository fine-tuning, which are either costly or brittle to evolving codebases. The framework offers two scenarios: Code2LoRA-Static, for stable codebases, converts a single repository snapshot into a LoRA adapter, achieving 63.8% cross-repo and 66.2% in-repo exact match on the static track of the new RepoPeftBench benchmark. Code2LoRA-Evo, for actively developing codebases, maintains an adapter updated per code diff using a GRU hidden state, reaching 60.3% cross-repo exact match on the evolution track (+5.2 pp over a single shared LoRA). Both variants outperform context-injection methods and demonstrate strong generalization on a temporal out-of-distribution holdout set.

Key takeaway

For Machine Learning Engineers building code assistants, Code2LoRA offers a compelling alternative to traditional RAG or costly per-repository fine-tuning. You should consider integrating hypernetwork-generated LoRA adapters to efficiently inject repository context, especially for evolving codebases. This approach eliminates inference-time token overhead and maintains performance as code evolves, providing a more scalable and robust solution for assertion completion and similar tasks.

Key insights

Hypernetworks can generate repository-specific LoRA adapters, efficiently injecting code context into LLMs without inference-time token overhead.

Principles

• Parametric adaptation outperforms context injection for repository knowledge.
• Recurrent aggregation of diffs tracks software evolution better than static snapshots.
• Cross-repository knowledge transfer regularizes adapter generation, reducing variance.

Method

Code2LoRA uses a shared repository encoder (Qwen3-Embedding-0.6B) to create dense embeddings, which a hypernetwork then maps to LoRA weights for a frozen Qwen2.5-Coder-1.5B LLM.

In practice

• Use Code2LoRA-Static for stable codebases to match per-repository LoRA performance.
• Employ Code2LoRA-Evo for active development to adapt to commit history.
• Consider hypernetwork-generated adapters to avoid context window limits.

Topics

• Code Language Models
• Hypernetworks
• LoRA Adapters
• Software Evolution
• Parameter-Efficient Fine-Tuning
• Code Completion Benchmarks

Code references

• huggingface/trl

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• The Evolution of Adaptation: From Full Fine-Tuning to LoRA Mastery — LoRA and QLoRA democratize LLM fine-tuning by drastically reducing computational and memory requirements.
• ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate — ARCA resolves degenerate token credit assignment in LoRA-based LLM-RL by leveraging adapter hidden-state residuals.
• BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution — BenchEvolver automatically hardens coding benchmarks by evolving solutions, restoring model discrimination and providing training data.
• Instant LLM Updates with Doc-to-LoRA and Text-to-LoRA — Hypernetworks generating on-demand LoRA adapters enable instant LLM customization and information internalization.
• The Appeal and Reality of Recycling LoRAs with Adaptive Merging — Adaptive LoRA merging offers limited benefit over training a new LoRA, often acting as regularization rather than knowledge transfer.
• HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support — Domain adaptation and RAG enable smaller LLMs to match larger models in specialized HPC support with fewer resources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.