Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification

2026-07-10 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Advanced, quick

Summary

A novel method integrates Large Language Models (LLMs) and Graph Convolutional Networks (GCNs) to enhance semi-supervised image classification, addressing the challenge of constructing effective graphs for visual data. Unlike traditional approaches that rely on kNN or reciprocal kNN algorithms applied to feature vectors from deep learning backbones, this technique employs a Vision Language Model (VLM) to generate textual descriptions for images. Subsequently, an LLM processes these descriptions to estimate semantic similarity scores between connected images. These scores are then utilized to prune edges in the initial kNN and reciprocal kNN graphs, effectively filtering out semantically irrelevant neighbors. Experimental results demonstrate that this LLM-guided graph refinement significantly improves classification accuracy, particularly when applied to kNN graphs and specific deep learning backbones. The source code is publicly available.

Key takeaway

For Machine Learning Engineers developing semi-supervised image classification systems with Graph Convolutional Networks, this research indicates a valuable strategy. If your current GCN performance is limited by graph construction, consider integrating Vision Language Models to generate image descriptions and then using Large Language Models to semantically prune kNN or reciprocal kNN graphs. This approach can significantly improve classification accuracy by filtering out irrelevant connections, offering a novel path to enhance model robustness with unlabeled data.

Key insights

Integrating LLMs with VLMs to semantically prune GCN image graphs improves semi-supervised classification accuracy.

Principles

• LLMs capture high-level image semantics.
• Semantic scores refine graph connections.
• Pruning irrelevant graph edges boosts GCNs.

Method

A VLM generates image descriptions, which an LLM processes to estimate semantic similarity scores. These scores then guide the pruning of kNN and reciprocal kNN graph edges, filtering semantically irrelevant neighbors.

In practice

• Generate image descriptions via VLM.
• Use LLMs for semantic similarity scoring.
• Prune kNN graphs with LLM-derived scores.

Topics

• Large Language Models
• Graph Convolutional Networks
• Semi-Supervised Learning
• Image Classification
• Vision Language Models
• Graph Pruning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval — Integrating graphs with LLMs enhances reasoning, retrieval, and decision-making by combining structured knowledge with semantic understanding.
• Convolution for Large Language Models — Depthwise convolutions enhance LLM locality with minimal parameters, improving context sensitivity.
• Graph Neural Networks for Semi-Supervised Image Classification with Multi-Feature Aggregation — Integrating diverse feature and graph representations with GNNs significantly boosts semi-supervised image classification accuracy.
• GCCLA: Graph-Conditioned Cross-Lingual Adaptation of Large Language Models Under Extreme Data Scarcity (A Case Study in Tigrigna) — Graph-conditioned cross-lingual adaptation using multilingual knowledge graphs enables data-efficient LLM transfer to extremely low-resource languages.
• Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models — Frozen LLMs can form a trainable feedforward graph via linear projections in a shared latent space.
• Semi-Supervised Learning on Graphs using Graph Neural Networks — GNN performance in semi-supervised learning is bounded by approximation, stochastic, and optimization errors, influenced by label scarcity and graph topology.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.