Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch

2023-10-26 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Mathswitch, an open-source project, connects mathematical concepts across diverse resources like Wikidata, Wikipedia, and MathWorld. It addresses data noise from Wikidata's collaboratively edited graph by employing a voting ensemble of three locally-hosted LLM judges (DeepSeek-R1 14B, Gemma-3 12B, Qwen-2.5 14B) to filter non-mathematical or ambiguous items. Evaluated on a 1000-item MathWorld-backed sample from an April 2026 Wikidata import, the ensemble achieved 98.2% apparent accuracy. The system processes 16,385 items, using SPARQL queries, Wikipedia API for article text, and spaCy for keyword extraction. Disagreements with MathWorld labels were categorized into degenerate descriptions, narrow scope bias (Gemma-3), and editorial-scope mismatches, informing pipeline improvements.

Key takeaway

For Research Scientists or ML Engineers building knowledge graphs from diverse, collaboratively edited sources, this work demonstrates a robust method for data cleansing. You should consider implementing a heterogeneous LLM voting ensemble to filter noisy or ambiguous entries, especially when ground truth is scarce. Analyzing model disagreements can reveal critical upstream data quality issues, such as degenerate descriptions, guiding targeted data enrichment or pre-filtering strategies to enhance overall system reliability.

Key insights

LLM voting ensembles effectively filter noisy mathematical concept data from collaborative knowledge graphs like Wikidata.

Principles

• Heterogeneous LLM ensembles reduce single-model biases.
• Contextual metadata improves categorization accuracy.
• Feedback loops refine data exclusion lists.

Method

Mathswitch uses SPARQL queries, fetches Wikipedia articles, extracts keywords with spaCy, then sends item metadata to a configurable LLM ensemble for yes/no classification with confidence scores. Majority votes update SPARQL exclusion lists.

In practice

• Implement LLM ensembles for data filtering in noisy datasets.
• Use confidence scores for precision/recall trade-offs.
• Analyze LLM disagreements to identify data quality issues.

Topics

• LLM Ensembles
• Mathematical Concepts
• Wikidata
• Knowledge Graphs
• Data Filtering
• Natural Language Processing

Code references

• UniMath/agda-unimath
• katjabercic/mathswitch

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• CauTion: Knowing When to Trust LLMs for Ensemble Causal Discovery — CauTion integrates LLM domain knowledge with statistical ensembles using trust calibration to improve causal discovery accuracy.
• Rethinking Semantic–Collaborative Integration in Recommenders, Efficient Token-Aware Clustering for Multivector Retrieval, and More! — Effective information retrieval systems benefit from managing complementarity, optimizing indexing, and distilling LLM utility.
• Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations — Internal LLM representations offer more robust uncertainty quantification than surface statistics, especially under distribution shift.
• A better method for identifying overconfident large language models — Cross-model disagreement combined with self-consistency more reliably identifies LLM overconfidence and hallucinations.
• Graph-Augmented LLMs for Swiss MP Ideology Prediction — Graph-augmented LLMs can model complex political ideologies from text and relational data.
• When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference — Delegation-based aggregation using semantic entropy and reasoning geometry significantly improves LLM multi-sample inference over majority voting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.