BACH: A Bayesian Admixture of Contrastive Heads for Multi-Interest Two-Tower Retrieval

2026-07-09 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

BACH (Bayesian Admixture of Contrastive Heads) is a novel multi-interest two-tower retrieval model designed to overcome limitations in existing systems. Traditional two-tower retrievers compress user interests into a single embedding, while current multi-interest models suffer "routing collapse" from hard-routing training, under-utilizing heads and lacking per-user interest weight estimates. BACH addresses this by framing multi-interest two-tower retrieval as a per-user mixture over heads, optimized via variational inference. This soft mixture trains all heads, mitigating collapse, and generates a per-user weighting of interests directly usable during serving. BACH also supports a shared global-codebook variant for precomputable retrieval. Evaluated on MovieLens-20M, Taobao, and Netflix benchmarks, BACH consistently improves top-of-ranking retrieval over hard-routing multi-interest and single-vector baselines at every head count. The study also found that scoring candidates by their best head, consistent with serving, outperforms target-routed training, with BACH showing further gains.

Key takeaway

For Machine Learning Engineers building multi-interest retrieval systems, you should consider adopting a Bayesian admixture approach like BACH. This method mitigates routing collapse by training all interest heads and provides explicit per-user interest weightings, directly improving serving relevance. Implement a soft mixture model and prioritize scoring candidates by their best head to achieve superior top-of-ranking retrieval performance compared to traditional hard-routing or single-vector baselines.

Key insights

BACH uses Bayesian admixture and variational inference for multi-interest retrieval, mitigating routing collapse and providing per-user interest weights.

Principles

• Soft mixtures prevent head under-utilization.
• Per-user interest weights improve serving.
• Best-head scoring enhances retrieval performance.

Method

BACH casts multi-interest two-tower retrieval as a per-user mixture over heads, fit by variational inference. This soft mixture trains every head and produces per-user interest weightings, supporting a global-codebook variant for precomputable retrieval.

In practice

• Implement soft mixture for multi-interest models.
• Use per-user interest weights in serving.
• Score candidates by their best head.

Topics

• Multi-Interest Retrieval
• Two-Tower Architectures
• Bayesian Admixture
• Variational Inference
• Routing Collapse
• Recommender Systems

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• BayesPO: Bayesian Prompt Optimization via Parallel-Tempered Gradient-Guided Discrete MCMC — Bayesian posterior sampling with gradient-guided MCMC optimizes LLM prompts by defining a posterior over discrete tokens.
• Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval — Hopfield-Fenchel-Young networks unify and extend associative memory models through generalized energy functions and differentiable update rules.
• Distribution-Alignment Bridge for Uncertainty-Aware Text-to-Video Retrieval — DAB reframes text-to-video retrieval as distribution alignment, modeling uncertainty with Gaussian embeddings and refining them via a diffusion-inspired bridge.
• A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems — A convex framework for neural likelihood approximation improves theoretical foundations and practical efficiency in inverse problems.
• ARIA: Adaptive Region-Based Importance Allocation for Conditional Diffusion Distillation — ARIA adaptively allocates training effort in conditional diffusion distillation by tracking teacher-student discrepancy across conditioning regions.
• py/cuTAGI: An Open-Source Library for Tractable Approximate Gaussian Inference in Bayesian Neural Networks — py/cuTAGI enable analytic Bayesian learning in neural networks by treating all quantities as Gaussian random variables, bypassing gradient descent.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.