Learning to Dequantise with Truncated Flows

2022-04-08 · Source: when trees fall... · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

The article, published on April 8, 2022, introduces Truncated Flows (TRUFL) as a method for learning to dequantize in autoregressive language modeling, addressing the posterior collapse problem encountered with stochastic embeddings. TRUFL is presented as a trade-off between Categorical Normalizing Flows (CatNF) and Argmax Flow, aiming to enable deterministic decoding while allowing the model to learn the support for each category. Unlike Argmax Flow, which uses pre-specified binary encoding for partitioning, TRUFL learns the support dynamically. The method leverages the reparameterization trick with truncated uniform sampling and inverse CDFs, specifically using the Logistic distribution for its tractable cumulative and inverse cumulative functions. Experimental animations demonstrate TRUFL's ability to learn optimal truncations, leading to nearly deterministic decoding and better KL term minimization compared to CatNF under high reconstruction loss weighting.

Key takeaway

For research scientists developing autoregressive language models with stochastic embeddings, consider integrating Truncated Flows (TRUFL) to mitigate posterior collapse. This approach allows for learning category support and achieving nearly deterministic decoding, potentially improving model efficiency and interpretability. Your team should evaluate TRUFL as an alternative to CatNF or Argmax Flow, especially when balancing reconstruction accuracy and KL divergence.

Key insights

TRUFL enables learning dequantization with truncated flows, addressing posterior collapse in stochastic embeddings for language models.

Principles

• Optimal decoding encourages distinct categories.
• Tractable inverse CDFs allow distribution transformation.

Method

TRUFL uses truncated uniform sampling on an interval $(a(x), b(x))$ within $(0,1)$, transforming it via a differentiable inverse CDF (e.g., Logistic distribution's logit function) to parameterize distributions with learnable, limited support.

In practice

• Use Logistic distribution for tractable CDF/inverse CDF.
• Add two parameters for truncation boundaries.

Topics

• Truncated Flows
• Normalizing Flows
• Dequantisation
• Stochastic Embeddings
• Posterior Collapse

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models — DRIFT adapts VLMs for precise continuous outputs by combining a coarse base prediction with flow matching for iterative residual refinement.
• Uncertainty Quantification for Large Language Diffusion Models — Lightweight, zero-shot uncertainty quantification for LLDMs can achieve high performance with significantly reduced computational cost.
• TFTF: Training-Free Targeted Flow for Conditional Sampling — TFTF enables training-free conditional sampling in flow matching by combining importance sampling with stochastic flow and SMC resampling.
• PTL-Diffusion: Manifold-Aware Diffusion with Periodic Terminal Laws — PTL-Diffusion improves manifold data generation by using periodic, structured terminal laws in its forward noising process.
• DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders — DecQ uses detail-condensing queries to balance reconstruction and generation in RAEs, improving both with minimal overhead.
• Sink-Aware Pruning for Diffusion Language Models — DLM attention sinks are transient, unlike stable AR LLM sinks, enabling targeted pruning for efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by when trees fall....