SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

2026-07-13 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, extended

Summary

SLBench, an 86-case executable benchmark, evaluates how LLM agents follow logical relations within skill files, which are reusable instruction sets. Developed using the SkillLogic framework, this benchmark analyzes over 5,000 public skills, revealing that 70% contain at least one critical logical relation. Evaluations of Codex and Claude Code across six LLM backbones show high unsafe rates, reaching up to 70%, with violations leading to privacy leaks and unsafe configuration changes. The study attributes failures to both agent capability gaps and low-salience skill text. A lightweight inference-time scaffold, SLGuard, is shown to reduce these violations by 63% on targeted cases, highlighting logical-relation following as a distinct reliability challenge for skill-guided agents.

Key takeaway

For AI Engineers deploying LLM agents with external skills, you must prioritize evaluating and mitigating failures in logical-relation following. Your current agents may complete tasks but still introduce privacy leaks or unsafe configurations by missing critical preconditions or postconditions. Consider integrating relation-aware scaffolds like SLGuard and designing skills with explicit, salient logical structures to enhance agent reliability and safety in production environments.

Key insights

LLM agents struggle with logical relations in skill files, leading to unsafe outcomes despite task completion.

Principles

• Skill safety depends on resolving inter-instruction dependencies.
• Logical relations are a distinct reliability challenge.
• Skill text salience impacts agent following.

Method

SkillLogic analyzes logical relations in skill files, constructs executable tests with realistic prompts and controlled repositories, and uses artifact-first deterministic grading to detect violations.

In practice

• Implement SLGuard for relation-aware execution.
• Audit skill clarity to improve agent performance.
• Prioritize unsafe-first grading for behavioral failures.

Topics

• LLM Agents
• Skill Files
• Logical Relations
• Agent Safety
• SLBench
• SLGuard
• Benchmark Evaluation

Best for: AI Architect, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills — LLM agents using third-party skills face significant, unmitigated safety risks from latent vulnerabilities.
• 🥇Top AI Papers of the Week — LLM agents are evolving to reason compositionally, learn reusable skills, and operate more efficiently through code and structured memory.
• Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On — Trust in LLM-based agent networks must be architected into the system's core design, not retrofitted.
• SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills — LLM-as-Judge effectively vets open-source LLM agent skills for multi-dimensional security risks, surpassing traditional code-layer scanners.
• The Scaling Laws of Skills in LLM Agent Systems — LLM agent performance depends on skill library structure, granularity, and exposure policy, not just model capability.
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks — Agent Skills significantly boost LLM performance, but effectiveness varies, and self-generation is currently ineffective.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.