In-Context Learning for Latent Space Bayesian Optimization

2026-06-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Scientific Machine Learning · Depth: Expert, quick

Summary

In-Context Learning for Latent Space Bayesian Optimization (LSBO) addresses a critical mismatch in how tabular foundation models like TabPFN and TabICL are applied as Bayesian optimization (BO) surrogates. While BO is crucial for sample-efficient design, and LSBO extends it to structured objects like molecules, the latent code-to-objective map in LSBO differs significantly from standard regression tasks used for pretraining in-context models. Researchers tackled this by complementing the pretraining of tabular foundation model surrogates with synthetic optimization tasks defined on a molecular VAE's latent space. This continued-pretraining objective includes a regularizer that anchors the model to its original checkpoint, maintaining its broad regression prior while preventing overspecialization. The resulting model demonstrated strong performance on held-out molecular optimization benchmarks, validating the importance of LSBO-specific adaptation for in-context surrogates.

Key takeaway

For Machine Learning Engineers developing Bayesian optimization solutions for structured data, you should consider domain-specific adaptation for in-context learning surrogates. If your latent space objective differs from standard regression tasks, complementing pretraining with synthetic optimization tasks, anchored to the original model, can significantly improve performance on benchmarks like molecular design. This approach ensures your models retain broad applicability while specializing effectively.

Key insights

Adapting tabular foundation models for latent space Bayesian optimization requires specific pretraining to address domain mismatches.

Principles

• Pretraining distribution is crucial for Bayesian behavior.
• Anchoring to original checkpoint preserves broad prior.
• LSBO requires domain-specific adaptation.

Method

Complement pretraining of tabular foundation model surrogates with synthetic optimization tasks on a molecular VAE's latent space, using a regularizer to maintain the original prior.

In practice

• Apply to molecular design optimization.
• Use VAEs for latent space representation.

Topics

• Bayesian Optimization
• Latent Space Optimization
• In-Context Learning
• Tabular Foundation Models
• Molecular Design
• Variational Autoencoders

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Time-Aware Latent Space Bayesian Optimization — TALBO adapts latent-space Bayesian optimization to dynamic objectives by integrating time into both the surrogate and generative representation.
• In-Context Multiple Instance Learning — Pretraining an in-context learner on synthetic data solves low-label MIL tasks efficiently without gradient updates.
• When Tabular Foundation Models Meet Strategic Tabular Data: A Prior Alignment Approach — PFN-style tabular models require inference-time alignment to overcome prediction bias in strategic data environments.
• Partial Causal Structure Learning for Valid Selective Conformal Inference under Interventions — Selective conformal prediction under interventions can be robustly maintained by learning partial causal structures.
• Learning Perturbations to Extrapolate Your LLM — Learnable continuous perturbations in embedding space significantly improve LLM out-of-domain extrapolation.
• TERRA: Task-Embedded Reasoning and Representation Architecture for Cross-Domain Applications — TERRA proposes a formal theory and testable hypothesis for cross-domain transfer of structured-state representations and predictors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.