Knowledge Graph and Hypergraph Transformers with Repository-Attention and Journey-Based Role Transport

2026-03-05 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A new architecture is proposed for jointly training language models on sentences and structured data, specifically knowledge graphs and hypergraphs, while maintaining a clear separation between knowledge and language representations. This model encodes structured data into a key-value repository that a language transformer can attend over. The attention mechanism is conditioned by "journey-based role transport," which unifies edge-labeled KG traversal, hyperedge traversal, and sentence structure. The architecture features a dual-stream design, hierarchical layer groups for instance-local, neighborhood, and global mixing attention, and retrieval over a separate repository. It supports multi-task objectives including masked language modeling, link prediction, and role-consistency denoising, enabling explicit, inspectable separation between linguistic context and structured knowledge through cross-attention.

Key takeaway

For research scientists developing advanced NLP models, this architecture offers a method to integrate structured knowledge graphs and hypergraphs with language models more effectively. Your models can achieve explicit separation of knowledge and language, making knowledge more inspectable and updateable. Consider implementing journey-based role transport to unify diverse data structures and improve reasoning over complex, multi-modal data.

Key insights

A novel architecture separates structured knowledge from language in transformers using a key-value repository and journey-based role transport.

Principles

• Separate knowledge from language representations.
• Unify diverse data structures via role-conditioned attention.
• Enable modular knowledge updates without retraining.

Method

Structured instances are encoded into a key-value repository. A language transformer attends to this repository using journey-based role transport, which generalizes positional embeddings to arbitrary roles and instances, enabling joint training via cross-attention.

In practice

• Implement dual-stream processing for language and structured data.
• Utilize hierarchical attention for varied receptive fields.
• Employ retrieval-augmented generation (RAG) for knowledge access.

Topics

• Knowledge Graph Transformers
• Hypergraph Transformers
• Repository-Attention
• Journey-Based Role Transport
• Dual-Stream Architecture

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• From Relational Database to Knowledge Graph: A Three-Pass Migration That Actually Works — Existing relational databases and data catalogs are foundational drafts for knowledge graph development.
• Implicit Semantic-Aware Communication Based on Hypergraph Reasoning — HISR uses hypergraphs and semantic subspaces to improve semantic-aware communication by capturing higher-order relationships and mitigating over-smoothing.
• LLMs+Graphs: Toward Graph-Native, Synergistic AI Systems — Integrating LLMs with graph computation creates synergistic AI systems that overcome LLM reasoning limitations.
• The Gap That Runs Both Ways — RDF 1.2 reification enables formal systems to model speech acts, bridging the expressivity gap with natural language by adding precision and provenance.
• Knowledge Graph Re-engineering Along the Ontological Continuum (extended version) — The "ontological continuum" offers a principled framework to describe and transform diverse knowledge graph modeling practices.
• Generating Graph-Like Logical Rules for Knowledge Graph Reasoning via Diffusion Models — Graph-like logical rules for Knowledge Graph reasoning can be efficiently generated by combining discrete diffusion models with reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.