Twins: Learn to Predict Unified Representations with Focal Loss

2026-07-24 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Twins proposes a unified continuous token space for multimodal models, integrating both understanding and generation capabilities. This space is created by channel-wise concatenating ViT and VAE features on the same token grid, maintaining sequence length and attention cost. When jointly modeled in a Diffusion Transformer, a significant optimization imbalance emerged, where the ViT component fit well but the VAE latent distribution struggled. This imbalance stems from frequency bias, intrinsic dimensionality, and condition-aligned vs. condition-independent uncertainty. To resolve this, Twins adapts a focal regression objective for flow matching, which upweights large-error VAE dimensions. This approach achieved up to 10.57 gFID gain on ImageNet over naive MSE loss, performs competitively on multimodal understanding benchmarks, and improves reconstruction fidelity.

Key takeaway

For machine learning engineers developing unified multimodal models or Diffusion Transformers, if you encounter optimization imbalances between understanding and generation components, consider adapting a focal regression objective for flow matching. This technique can effectively rebalance training by upweighting struggling dimensions, potentially yielding significant gFID gains and improving both understanding and reconstruction fidelity in your models.

Key insights

Twins unifies ViT and VAE features into a continuous token space, using focal regression to balance optimization for improved multimodal performance.

Principles

• Multimodal models face optimization imbalance from feature heterogeneity.
• Frequency bias, intrinsic dimensionality, and uncertainty cause imbalance.
• Focal regression can rebalance optimization across disparate components.

Method

Channel-wise concatenate ViT and VAE features on a shared token grid. Apply a focal regression objective for flow matching, upweighting large-error VAE dimensions to balance optimization.

In practice

• Achieve up to 10.57 gFID gain on ImageNet.
• Improve multimodal understanding benchmark performance.
• Enhance reconstruction fidelity in generative models.

Topics

• Unified Multimodal Models
• Diffusion Transformers
• Focal Loss
• ViT
• VAE
• Flow Matching
• Image Generation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Twins: Learn to Predict Unified Representations with Focal Loss — Channel-wise concatenation of ViT and VAE features creates a unified representation, balanced by focal loss to overcome optimization imbalance.
• Support-Conditioned Flow Matching Is Kernel Smoothing — Cross-attention conditioning in generative models is kernel smoothing, with predictable failure modes and learned corrections.
• Improving Visual Representation Alignment Generation with GRPO — VRPO uses adaptive, reward-guided representation alignment to dynamically optimize diffusion transformer training for better fidelity and speed.
• TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation — TrioPose natively integrates pose as an independent modality within diffusion transformers to resolve multi-person image generation distortions.
• Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing — The framework unifies data- and algorithm-level balancing to improve imbalanced regression performance.
• Focal Loss Explained: When It Helps, When It Hurts, and How to Think About It in Production ML — Focal Loss reweights training examples to prioritize hard-to-classify instances, mitigating the dominance of easy negatives in imbalanced datasets.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.