Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models

2026-07-21 · Source: Artificial Intelligence · Field: Science & Research — Life Sciences & Biology, Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Expert, quick

Summary

A new computational framework integrates sparse dictionary learning with causal intervention to enhance interpretability in genomic language models. This method extracts, validates, and causally tests interpretable features within models like Nucleotide Transformer (using 6-mer tokenization) and DNABERT-2 (using byte-pair encoding). Researchers recovered thousands of monosemantic features corresponding to transcription-factor (TF) sequence motifs. The framework addresses a critical issue where naive validation against position weight matrices is confounded by GC composition and repetitive elements, generating spurious "TF features." By employing a composition-matched, binding-resolved protocol, these confounds are eliminated. Crucially, the approach moves beyond correlation, using causal ablation of dictionary directions during the forward pass to demonstrate that specific features are causally utilized for cell-type-specific TF binding, not merely motif presence. This was reproducibly validated across CTCF, GATA1, and REST, with 7--14 of 15 tested features per condition showing signal, unlike negative controls.

Key takeaway

For AI Scientists and Research Scientists developing or interpreting genomic language models, this framework offers a robust standard for feature validation. If you are assessing whether your model's internal representations genuinely reflect biological concepts like TF binding, you should adopt causal intervention methods. This approach moves beyond correlation, ensuring that identified features are causally utilized by the model, thereby preventing misinterpretation due to sequence composition artifacts. Implement this purely computational, public-data-driven protocol to establish verifiable interpretability claims.

Key insights

Causal dictionary learning validates interpretable features in genomic language models, distinguishing true concepts from compositional artifacts.

Principles

• Genomic LM feature validation requires accounting for sequence composition confounds.
• Causal intervention confirms active feature use, moving beyond mere correlation.
• Sparse dictionary learning extracts monosemantic features like TF motifs.

Method

Top-k sparse autoencoders learn dictionary features from hidden activations. Causal intervention ablates individual dictionary directions during the forward pass, measuring induced shifts in predictive distribution, validated via a composition-matched, binding-resolved protocol.

In practice

• Extract TF sequence motifs using sparse dictionary learning on genomic LMs.
• Employ composition-matched validation to prevent spurious feature identification.
• Use causal ablation to verify features' active role in model predictions.

Topics

• Genomic Language Models
• Model Interpretability
• Causal Intervention
• Sparse Dictionary Learning
• Transcription Factors
• TF Binding Motifs

Best for: AI Scientist, Research Scientist

Related on AIssential

• A Causal Foundation Model for Structure and Outcome Prediction — TabPFN-CFM is a causal foundation model predicting structure and outcomes, supporting Pearl's Causal Hierarchy queries.
• Automated Synthesis and Adversarial Validation of Executable Causal Research Pipelines — Automated causal research systems should prioritize surfacing invalid assumptions over merely producing numerical answers.
• Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers — Transformers' learned stack representations are causally necessary for their performance on counter languages.
• Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learning — Causal interventions can effectively manipulate continuous variables like verb bias in language model steering vectors.
• Predicting Causal Effects from Natural Language Queries using Structured Representations — Finetuning and structured representations significantly enhance LLM performance in predicting causal effects from natural language.
• CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts — CausalMoE uses pattern-routed heterogeneous experts and multimodal priors for robust Granger Causal Discovery in complex, dynamic time series.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.