Learning Critical Testing Literacy Through Puzzles: an Experience Report

2024-02-01 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Skill Development & Professional Training · Depth: Intermediate, extended

Summary

This experience report details findings from 13 workshops utilizing puzzles to teach Critical Testing Literacy (CTL), based on the P4TEST pedagogical framework. Six domain-agnostic puzzles, designed to evoke testing-related cognitive moves like hypothesis formation and systematic exploration, were used with students, testers, teachers, and primary school pupils. Initial observations from eleven workshops led to two additional workshops incorporating workbooks and think-aloud sessions for deeper data collection. Participants consistently perceived themselves as experimenting. Students tended to converge on solutions, while professionals explored solution spaces more. Emotions were visible but hard to articulate in writing. The study also led to an open-source web application for customizing workshops and collecting analytics.

Key takeaway

For educators or training managers designing software testing curricula, integrate puzzle-based activities with a strong emphasis on structured debriefing and reflection. Ensure your workshop environments support collaboration and externalized thinking, as the full learning sequence, not just puzzle solving, drives critical testing literacy development. Consider hybrid digital/paper tools to reduce friction in data collection and enhance analytical insights.

Key insights

Effective Critical Testing Literacy development requires puzzle-based activities integrated with structured debriefing and reflection.

Principles

• The entire sequence of solving, debriefing, and reflecting is the intervention.
• Debriefing is crucial for consolidating experience into learning and making connections.
• Group dynamics and physical environment significantly influence participant engagement.

Method

Workshops involve puzzle solving, followed by a debrief discussing solutions, assumptions, and biases, then reflection. Data collection can be enhanced via workbooks and think-aloud sessions.

In practice

• Provide and explicitly encourage using pen and paper to externalize thinking.
• Vary puzzle categories to avoid conceptual priming and enhance learning effects.
• Use puzzles to surface responses to uncertainty, creating reflection opportunities.

Topics

• Critical Testing Literacy
• Puzzle-based Learning
• Software Testing Education
• P4TEST Framework
• Experiential Learning
• Workshop Design

Best for: Software Engineer, Research Scientist

Related on AIssential

• Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work — The QuestBench practice teaches AI accountability by having students construct benchmarks that expose AI system failures.
• Teaching testing seriously in academia — Testing education should shift from a rationalist to an empirical, inquiry-driven paradigm to foster critical reasoning.
• How to Learn to Solve Problems | Problem Solving Lifecycle | Soft Skills in Tech — Effective problem-solving prioritizes deep understanding and structured preparation over immediate tool application.
• Beyond Tool Adoption: A Practical Five-Stage Developmental Continuum for AI Literacy in Higher Education — A five-stage AI Literacy Continuum offers a diagnostic and developmental pathway for responsible AI engagement in higher education.
• Training LLMs at test time for Scientific Discovery — Test-time training with an Entropic objective enables LLMs to self-improve and discover novel solutions by prioritizing high-reward outliers.
• From Real-World Projects to Research-Oriented Learning: Continuous Improvement of a Master-Level Course in Software Engineering Education — Continuously evolving a master-level course towards research-oriented learning can maintain positive student perceptions with deliberate design.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.