Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work

2026-05-20 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Advanced, quick

Summary

A new course-based practice, QuestBench, teaches AI education through benchmark construction, focusing on students' ability to test AI and judge machine-produced knowledge. Students develop verifiable, expert-level questions from disciplinary knowledge, review designs for clarity, and evaluate AI systems. The resulting QuestBench benchmark comprises 256 questions across 14 humanities and social-science domains. Evaluations reveal significant failures in current deep research systems; across thirteen systems, the mean question-level pass rate is only 16.85%. Even the best-performing system, GPT-5.5, achieved only a 57.58% pass rate. These failures are valuable for education, demonstrating how fluent, source-backed AI answers can still miss critical query, source, term, or evidence standards. This approach aims to help students become responsible knowledge actors as AI integrates into learning and professional work. The dataset is available at https://huggingface.co/datasets/PKUAIWeb/QuestBench/tree/main.

Key takeaway

For AI educators designing curricula, integrating benchmark construction like QuestBench can fundamentally shift how students engage with AI. You should empower students to create and evaluate AI tasks, fostering critical judgment over mere prompt engineering. This helps students understand AI's limitations, especially its struggles with evidence standards. It prepares them to be accountable knowledge actors in professional settings.

Key insights

The QuestBench practice teaches AI accountability by having students construct benchmarks that expose AI system failures.

Principles

• AI education needs testing and judgment skills.
• Benchmark construction reveals AI system limitations.
• Professional knowledge defines AI output quality.

Method

Students turn disciplinary knowledge into expert-level questions, review designs for ambiguity, and evaluate AI systems on the resulting tasks.

In practice

• Use QuestBench dataset for AI evaluation.
• Implement course-based benchmark construction.
• Design tasks revealing AI's evidence gaps.

Topics

• AI Education
• Benchmark Construction
• QuestBench
• AI Evaluation
• Accountable Knowledge Work

Best for: Research Scientist, AI Student, AI Scientist

Related on AIssential

• AI Literacy at Scale: K-to-Career Access That Delivers Real Student Outcomes — AI integration in education requires balancing student engagement with responsible use, curriculum adaptation, and equitable access.
• Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends — Automating embodied benchmark construction shifts costs to validation and governance, necessitating diagnosable and auditable pipelines for reliable evaluation.
• The Role of Instructional Guidance in Generative AI-Assisted Learning: Empirical Evidence from Construction Engineering Education — Instructional guidance, like a five-step prompting framework, significantly enhances generative AI's effectiveness for deeper learning tasks.
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks — Agent Skills significantly boost LLM performance, but effectiveness varies, and self-generation is currently ineffective.
• The engineer who learned by building: How Rilton Franzone became a legal AI specialist — Learning by building and prioritizing usefulness in high-stakes domains fosters effective, reliable engineering.
• From Exams to Escape Rooms: How We Learned to Test AI — AI evaluation is a dynamic process, constantly adapting to measure increasingly complex model capabilities beyond narrow task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.