What Do Your Logits Know? (The Answer May Surprise You!)

2026-04-20 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

A recent study by Masha Fedzechkina, Eleonora Gualdoni, Rita Ramos, and Sinead Williamson investigates information leakage from large language models, specifically focusing on vision-language models. The research systematically compares information retention across different "representational levels" as data is compressed from the residual stream. This includes low-dimensional projections via tuned lens and the final top-$k$k logits. The authors demonstrate that even easily accessible bottlenecks, such as the model's top logit values, can inadvertently leak task-irrelevant information from image-based queries. In some instances, these logit values reveal as much information as direct projections from the full residual stream, highlighting a significant risk of unintentional or malicious data exposure that model owners might assume is inaccessible.

Key takeaway

For research scientists and CTOs concerned with model security and data privacy, this research indicates that even seemingly innocuous model outputs like top logits can expose sensitive, task-irrelevant information. You should implement robust information leakage assessments beyond just the full residual stream, focusing on all accessible model outputs to mitigate potential data exposure risks.

Key insights

Model logits can leak significant task-irrelevant information, posing a risk of unintended data exposure.

Principles

• Information leakage persists through model bottlenecks.
• Top logit values can reveal sensitive data.

Method

The study systematically compares information retention in vision-language models across residual stream projections and top-$k$k logits to quantify leakage at different representational levels.

In practice

• Audit logit values for sensitive data.
• Implement stricter output filtering.

Topics

• Information Leakage
• Model Probing
• Vision-Language Models
• Logit Values
• Residual Stream

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Security Engineer

Related on AIssential

• "Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms — Evaluating AI lie detectors critically depends on verifiable model beliefs, which current methods often fail to establish.
• The Statistics of Token Selection: Logits, Temperature, and Top-P Walkthrough — LLM output generation relies on a statistical pipeline involving logits, temperature, and top-p for next-token prediction.
• TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction — AI agents face an inherent trade-off between using private data for tasks and preventing its leakage, which prompt-based defenses cannot fully resolve.
• Pretraining Data Filtering for Open-Weight AI Safety — Pretraining data filtering effectively builds tamper-resistant safeguards into open-weight LLMs by preventing undesirable knowledge acquisition.
• A review of “Investigating the consequences of accidentally grading CoT during RL” — Accidental CoT exposure during RL training may subtly degrade model monitorability, even if direct impacts are hard to measure.
• Did the Model See the Benchmark During Training? Detecting LLM Contamination — A lightweight method estimates LLM benchmark contamination by analyzing model behavior when training data is unknown.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.