RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

2026-07-01 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Natural Language Processing · Depth: Advanced, quick

Summary

RuleChef is an open-source framework, released under Apache 2.0, designed to generate human-editable, executable rules for various Natural Language Processing (NLP) tasks, including text classification, Named Entity Recognition (NER), and relation extraction. This system leverages large language models (LLMs) to synthesize rules from task descriptions and labeled examples. The generated rules undergo iterative refinement, incorporating additional examples and human feedback to enhance their accuracy. RuleChef also offers the capability to bootstrap rules by observing input-output pairs from existing models. Crucially, LLMs are exclusively utilized during the learning phase for rule generation and patching based on failures on a held-out dataset, resulting in a fast, deterministic, and inspectable rule system. Preliminary evaluations have been conducted on both classification and NER tasks.

Key takeaway

For NLP engineers developing explainable or high-performance rule-based systems, RuleChef offers a compelling approach to automate rule generation. You can leverage LLMs at learning time to synthesize and refine executable rules for tasks like classification or NER, ensuring deterministic and inspectable outputs. Consider integrating RuleChef to bootstrap new rule systems or enhance existing ones, reducing manual effort while maintaining transparency and control over your NLP logic.

Key insights

RuleChef uses LLMs to generate and iteratively refine human-editable, executable rules for NLP tasks, ensuring fast, deterministic, and inspectable systems.

Principles

• LLMs synthesize executable NLP rules.
• Iterative refinement improves rule accuracy.
• Human feedback enhances rule systems.

Method

RuleChef generates rules from task descriptions and labeled examples, then iteratively patches them using additional examples and human feedback, with LLMs active only during learning.

In practice

• Generate rules for text classification.
• Create NER rules from examples.
• Bootstrap rules from existing models.

Topics

• RuleChef
• Large Language Models
• Natural Language Processing
• Text Classification
• Named Entity Recognition
• Explainable AI
• Rule-based Systems

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval — An LLM-based agent can self-evolve query rewriting rules to enhance BM25 for legal case retrieval, leveraging an automatic evaluation environment without parameter training.
• Revisiting the Systematicity in Negation in the Era of In-Context Learning — LLMs partially grasp negation via in-context learning, but struggle with robust scope recognition and function vector composition for it.
• FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs — Domain-specific fine-tuning of LLMs significantly enhances knowledge graph construction quality and efficiency for complex, jargon-heavy legal texts.
• A Larger Annotated Corpus of Portuguese Coreference — LLMs can annotate Portuguese coreference, but a new LLM-based system did not outperform existing rule-based methods.
• Forget LLM: MIT's New RLM (Phase Shift in AI) — MIT's Recursive Language Model (RLM) uses a neurosymbolic approach to overcome LLM context rot and enhance reasoning.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.