The Quadrilateral Loss: Additivity as a Measurable Behavior of Dense Neural Networks

2026-07-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The paper introduces the quadrilateral loss, a differentiable penalty designed for dense neural networks that treats additivity as a measurable behavior. This loss is defined as a second-order mixed difference on pairs of training points, vanishing when a coordinate carries no interaction, and is informative for piecewise-linear networks. It quantifies per-coordinate interaction mass, analogous to the interventional Shapley-GAM. The quadrilateral loss allows additivity to be a tunable parameter, demonstrating that most learned interactions can be removed with minimal cost, and on small datasets, a moderate penalty simultaneously improves accuracy and additivity. Furthermore, its per-feature surrender curves reveal that pre-regularization interaction magnitude poorly predicts what a regularized model retains, challenging post-hoc interaction rankings. The work also compares various routes to exact additivity, concluding that constraining behavior before structure often outperforms weight-space constraints.

Key takeaway

For Machine Learning Engineers building interpretable models, consider integrating the quadrilateral loss into your dense neural network training. This differentiable penalty allows you to dial in additivity, potentially improving accuracy on smaller datasets while reducing feature interactions. You can also use its per-feature surrender curves to assess how regularization impacts interaction retention, guiding your choice of interpretability techniques beyond post-hoc rankings.

Key insights

The quadrilateral loss quantifies feature interaction in dense neural networks, enabling tunable additivity and improved interpretability.

Principles

• Additivity can be a measurable behavior, not solely an architectural constraint.
• Most learned feature interactions are removable with minimal cost.
• Behavioral constraints often outperform weight-space constraints for additivity.

Method

The quadrilateral loss is a differentiable penalty based on a second-order mixed difference on training points, vanishing when a coordinate carries no interaction.

In practice

• Apply quadrilateral loss to tune additivity in dense neural networks.
• Use per-feature surrender curves to evaluate interaction retention post-regularization.
• Prioritize behavioral penalties over structural masks for additivity.

Topics

• Quadrilateral Loss
• Neural Network Interpretability
• Feature Interaction
• Additive Models
• Differentiable Penalties
• Model Regularization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Quadrilateral Loss: Additivity as a Measurable Behavior of Dense Neural Networks — The Quadrilateral Loss measures and controls feature additivity in neural networks behaviorally, improving interpretability and accuracy.
• Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision — Directional confusions and Rate-Distortion geometry reveal distinct inductive biases in human and machine vision.
• Loss-Shift Transfer via Bayes Quotients — Loss shift, a distinct failure mode from distribution shift, occurs when the loss function changes, necessitating different data representations.
• Engineering Verifiable Modularity in Transformers via Per-Layer Supervision — Architectural interventions and per-layer supervision can engineer verifiable modularity in Transformers, overcoming the "Hydra effect."
• Effects of sparsity and superposition on loss in simple autoencoders — Superposition mathematically explains polysemanticity, enabling efficient data compression in sparse autoencoders.
• Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic — Neural networks on modular arithmetic tasks form task-intrinsic cyclic geometries, deviating from neural collapse due to a structured trade-off.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.