TritonSigmoid: A fast, padding-aware sigmoid attention kernel for GPUs [R]

2026-05-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Life Sciences & Biology · Depth: Expert, quick

Summary

TritonSigmoid is an open-source, fast, and padding-aware sigmoid attention kernel designed for GPUs, specifically developed for single-cell foundation models. Unlike softmax, which forces competition among tokens, sigmoid attention allows models to attend strongly to multiple genes (tokens) simultaneously, crucial for sequences where cells express 200 to 16,000+ genes. The kernel natively handles variable-length padding, avoiding wasted compute on empty positions. Experiments show TritonSigmoid achieves up to 515 TFLOPS on H100 GPUs, outperforming FlashAttention-2 (361 TFLOPS) and FlashSigmoid (440 TFLOPS). It also demonstrated lower validation loss across six datasets, 25% better cell-type separation, and stable training where softmax attention diverged catastrophically.

Key takeaway

For AI Engineers developing models with variable sequence lengths, especially in genomics or other domains where multiple features can be simultaneously relevant, TritonSigmoid offers significant performance and stability advantages over traditional softmax attention. Its native padding awareness and `torch.compile` integration streamline development and improve training outcomes, even with a potential memory overhead compared to packed approaches. Consider integrating this kernel to enhance model accuracy and training robustness.

Key insights

TritonSigmoid offers superior performance and stability for variable-length sequence attention, especially in biological modeling.

Principles

• Sigmoid attention enables multi-gene focus.
• Padding-aware kernels optimize variable-length sequences.

Method

The kernel uses blockwise compute, similar to FlashAttention, and handles variable lengths by padding to max length and skipping fully padded blocks to maximize `torch.compile` integration.

In practice

• Use TritonSigmoid for single-cell genomics.
• Apply to any variable sequence length data.

Topics

• TritonSigmoid
• Sigmoid Attention
• GPU Kernel
• Padding-Aware Attention
• Single-Cell Foundation Models

Code references

• MSDLLCpapers/triton-sigmoid

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels — CuBridge adapts expert CUDA attention kernels using LLMs and a structured lift-transfer-lower workflow for high performance.
• Dispatch-Aware Ragged Attention for Pruned Vision Transformers — Dispatch overhead, not compute, bottlenecks pruned ViT attention at short sequence lengths.
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels — Factored norms and fused kernels significantly optimize high-rank DoRA, reducing memory and improving speed.
• Attention-Based Prototype Calibration for Multi-Rater Few-Shot Medical Image Segmentation — Prototype calibration with attention effectively models multi-rater variability in few-shot medical image segmentation.
• [Hands-On] GPT-OSS 바닥부터 구현하기 (3/5)- Self-Attention Mechanism — Implementing Self-Attention with GQA, Attention Sinks, and RoPE significantly optimizes Transformer model efficiency and contextual understanding.
• HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization — Head-level functional heterogeneity within attention layers provides a principled granularity for hybridizing attention signals.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.