Using Topological Data Analysis to Characterize the Layers of Language Models Before and After Word Substitution Attacks

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

A new topological data analysis (TDA) framework has been introduced to investigate the structural effects of adversarial word substitution attacks on large language models' internal representations. This framework evaluates small encoder-based architectures like BERT, RoBERTa, and DistilBERT, fine-tuned for binary classification on the IMDb review dataset and attacked using TextFooler. The method converts attention maps into distance matrices, applies TDA to extract topological features, and compares these features using Wasserstein distances between original and perturbed states. A non-TDA baseline using per-head L1 distances was also computed. Findings indicate that adversarial perturbations induce systematic and statistically significant topological changes across model layers, with the largest deviations observed in late layers and notable effects in early layers. These patterns are consistent across models and validated by non-parametric (Kruskal–Wallis, Dunn) and parametric (one-way ANOVA, Tukey) tests. The TDA framework provides more distinct layer-wise separation and a robust, interpretable method compared to the L1 baseline.

Key takeaway

For AI Security Engineers evaluating language model robustness, this TDA framework offers a powerful new lens. You should consider integrating topological analysis to gain deeper, layer-by-layer insights into how adversarial attacks structurally alter internal model representations, beyond just output behavior. This approach provides a more robust and interpretable understanding of perturbation propagation, enabling more targeted defense strategies against synonym-based word substitutions.

Key insights

Topological Data Analysis reveals systematic structural changes in LLM internal representations due to adversarial word substitutions.

Principles

• Adversarial perturbations induce systematic topological changes.
• Late layers show largest deviations in internal structure.
• TDA offers robust, interpretable analysis of model structure.

Method

Convert attention maps into distance matrices, apply TDA to extract topological features, then compare original and perturbed features using Wasserstein distances.

In practice

• Apply TDA to analyze adversarial influence on attention maps.
• Compare TDA with L1 distances for structural insights.
• Evaluate layer-by-layer changes in model representations.

Topics

• Topological Data Analysis
• Adversarial Attacks
• Language Models
• Attention Maps
• BERT
• Model Robustness

Code references

• angelinatsai04/mitll_clinic

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models — The survey unifies disparate adversarial diffusion research across modalities, providing a common framework for attacks, defenses, and evaluation.
• MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models — Alignment selectively reorganizes LLM internal geometry, with measurable torsion shifts concentrated in specific layers and concepts.
• LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training — Data contamination in RL post-trained LLMs can be reliably detected by analyzing layer-wise representation deviations.
• Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometry — NLLB-200's internal geometry reflects language-universal conceptual structures, akin to human multilingual cognition.
• Localizing Anchoring Pathways in Language Models — Anchoring bias in LLMs is localized to sparse, shared internal pathways, but these pathways shift with instruction tuning.
• I Built a System That Detects AI Manipulation Attacks — Here Is What the Results Revealed — Deep learning models like BERT significantly outperform classical NLP in detecting sophisticated prompt injection attacks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.