LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

Low-Rank Convolutional Adaptation (LoCA) is a new Parameter-Efficient Fine-Tuning (PEFT) framework designed for Vision Foundation Models (VFMs), specifically addressing the challenges of adapting convolutional kernels. Unlike traditional Low-Rank Adaptation (LoRA), which is optimized for 2D matrices in transformer self-attention layers, LoCA accounts for the inherent spatial-channel coupling in 4D convolutional tensors. It achieves this by decoupling channel and spatial adaptation, introducing a low-rank channel adaptation for cross-channel mixing and refining spatial bases extracted from pre-trained kernels using Singular Value Decomposition (SVD). Experimental results demonstrate that LoCA effectively preserves pre-trained spatial priors, achieving competitive performance or matching the best current results across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks. This approach mitigates the prohibitive costs and catastrophic forgetting associated with full fine-tuning VFMs.

Key takeaway

For Machine Learning Engineers adapting Vision Foundation Models with convolutional layers, LoCA offers a superior Parameter-Efficient Fine-Tuning alternative to LoRA. If your projects involve fine-grained classification, semantic segmentation, or generative tasks, consider integrating LoCA to preserve spatial priors and achieve competitive performance without the high costs of full fine-tuning. This approach helps mitigate catastrophic forgetting, ensuring more robust model adaptation.

Key insights

LoCA adapts convolutional VFMs by decoupling spatial and channel information, improving PEFT over LoRA for 4D tensors.

Principles

• Convolutional kernels inherently couple spatial and channel data.
• Decoupling spatial and channel adaptation is crucial for PEFT.
• SVD refines pre-trained spatial bases effectively.

Method

LoCA performs low-rank channel adaptation for cross-channel mixing and refines spatial bases from pre-trained kernels using Singular Value Decomposition (SVD).

In practice

• Fine-grained classification
• Domain-generalized semantic segmentation
• Generative benchmarks

Topics

• Vision Foundation Models
• Parameter-Efficient Fine-Tuning
• Low-Rank Adaptation
• Convolutional Adaptation
• Singular Value Decomposition
• Semantic Segmentation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Variable-Rate Deep Image Compression based on Low-Rank Adaptation by Progressive Learning — LoRA-based progressive learning enables efficient variable-rate deep image compression with significant resource savings.
• Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks — Parameter-Efficient Fine-Tuning (PEFT) with adapters and LoRA drastically reduces parameters for instance segmentation while maintaining performance.
• iLoRA: Bayesian Low-Rank Adaptation with Latent Interaction Graphs for Microbiome Diagnosis — iLoRA is a Bayesian graph-conditioned LoRA framework that jointly learns predictions and latent interaction structures for domain adaptation.
• Low-Rank Adaptation Redux for Large Models — Signal processing principles offer a principled framework for advancing LoRA and parameter-efficient fine-tuning methods.
• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models — GUI VLM KV cache compression benefits from spatio-temporal adaptive re-weighting, moving beyond fixed saliency and cutoffs.
• The Eigenvectors of AI: Shared LoRA Subspaces for Continual Learning — Task-specific LoRA adapters converge to a shared low-rank subspace, enabling massive storage and memory reduction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.