AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The AMATI team submitted a neuro-symbolic system to the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German. This system integrates automatically learned symbolic rules via Inductive Logic Programming with the Mistral-large language model. The primary objective was to enhance overall grading performance, leverage symbolic rules for explainability, and utilize the LLM for robustness, particularly for unseen answers. The combined approach demonstrated improved performance in the 3-way task, securing 5th place out of 8 competitors. Notably, neither the symbolic system nor Mistral-large alone would have achieved a rank higher than 6th in this challenge. However, the inclusion of symbolic rules did not improve upon Mistral's performance in the 2-way test, where the team placed 6th out of 9.

Key takeaway

For Machine Learning Engineers developing automated grading systems, this research suggests that combining symbolic logic with large language models can yield superior performance in multi-class scoring challenges. If you are evaluating architectures for rubric-based short answer grading, consider integrating Inductive Logic Programming with an LLM like Mistral-large to improve overall accuracy and potentially enhance explainability, especially for German language tasks. However, be aware that symbolic rules might not always boost LLM performance in simpler binary classification scenarios.

Key insights

Combining Inductive Logic Programming with a large language model can enhance short answer grading performance for multi-class tasks.

Principles

• Neuro-symbolic integration can improve performance over individual components.
• Symbolic rules can offer explainability in hybrid AI systems.
• LLMs contribute robustness to neuro-symbolic grading systems.

Method

The system uses Inductive Logic Programming to automatically learn symbolic rules, which are then combined with the Mistral-large language model for rubric-based short answer scoring in German.

In practice

• Explore neuro-symbolic architectures for complex grading tasks.
• Evaluate hybrid systems for explainability and robustness.

Topics

• Automatic Short Answer Grading
• Neuro-symbolic AI
• Inductive Logic Programming
• Large Language Models
• Mistral-large
• Educational Applications
• German NLP

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Neuro-symbolic Approaches for Rubric-Based Automatic Essay Evaluation of ENEM Essays — Neuro-symbolic methods can enhance automated essay scoring with interpretability and fine-grained feedback.
• Symbolic Machine Learning & NLP — Symbolic ML methods provide interpretable, rule-based approaches crucial for structured data and low-resource languages.
• ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark — LLMs struggle with symbolic math perturbations, indicating pattern memorization, though frontier models show surprising robustness.
• Towards Fully Automated Exam Grading: Fairness-Aware Recognition of Handwritten Answers with Foundation Models — VLMs enable highly accurate and fair automated grading of handwritten exams, overcoming prior limitations.
• Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering — Distilling ASP rules from LLMs simplifies adapting logic-based VQA reasoning theories with minimal examples.
• SLMJury: Can Small Language Models Judge as Well as Large Ones? — Small Language Models can effectively judge model outputs, but their performance is highly dependent on the task domain and judging paradigm.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.