EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

2026-05-19 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

An evaluation of code embedding models for Scratch programs was conducted, comparing four large language models (LLMs) and five dedicated embedding approaches, including ASTNN, code2seq, code2vec, GGNN, and NeuralCodeSum. Researchers empirically assessed these models on a "sprite naming" task, multilingual datasets, and two program classification tasks (project category and remixes). Additionally, the study investigated the embeddings' ability to predict program correctness and student exercise progress. Findings indicate that models incorporating more structural information, such as GGNN, generally outperform flat token-based LLMs for sprite naming. However, for whole-project classification and educational analytics, pre-trained embedding LLMs, like Qwen 3 Embedding, achieved comparable or superior performance to dedicated Scratch models, often without requiring further task-specific fine-tuning. This research demonstrates the feasibility of transferring code embeddings to the Scratch educational environment, capturing relevant structural and semantic information.

Key takeaway

For AI Scientists and Machine Learning Engineers developing educational tools for Scratch, prioritize dedicated models like GGNN for tasks requiring deep structural code insights, such as bug detection or next-step hint generation, especially in resource-constrained environments. For higher-level analytics like student progress tracking or project categorization, leverage pre-trained embedding LLMs (e.g., Qwen 3 Embedding) due to their comparable performance and robustness, even without extensive fine-tuning. Ensure your tokenization strategies account for Scratch's multilingual user base to maintain accuracy across diverse languages.

Key insights

Code embeddings, especially those capturing structural information, are effective for Scratch learning analytics, even without task-specific fine-tuning.

Principles

• Structural code information improves Scratch embedding performance.
• LLMs and dedicated models have complementary roles.
• Pre-trained embeddings transfer effectively to new tasks.

Method

Evaluated four LLMs and five dedicated code embedding models on Scratch programs. Used a "sprite naming" task for pre-training, then assessed transferability to program classification, correctness prediction, and student progress visualization.

In practice

• Use GGNN for structural Scratch code insights.
• Employ embedding LLMs for high-level project analysis.
• Pre-train on large datasets for classroom use.

Topics

• Scratch Programming Language
• Code Embeddings
• Learning Analytics
• Large Language Models
• Program Classification
• Educational Technology

Code references

• se2p/LitterBox
• pemistahl/lingua-rs

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Königsberg at SemEval-2026 Task 13: Beyond Language Models: A Low-Resource Feature-Driven and Data-Flow Embedding Approach for Machine-Generated Code Detection — A hybrid framework efficiently detects machine-generated code using software metrics and frozen GraphCodeBERT embeddings, avoiding LLM fine-tuning overhead.
• Solving the Embedding Step in NLP Projects — Effective embeddings are crucial for NLP model generalization and reliable downstream task performance.
• CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval — Multimodal code retrieval unifies natural language, code, and images to improve code search and generation.
• Anchor Detection for RAG: Parallel Detectors, Then One LLM Call at the End — Enterprise RAG retrieval combines parallel keyword and embedding detectors, aggregating results for a single, auditable LLM ranking.
• Graph Construction and Matching for Imperative Programs using Neural and Structural Methods — Converting programs to typed, attributed graphs using neural and structural methods enables scalable reuse of verification artifacts.
• multiple linear regression in scratch [P] — Implementing advanced algorithms like multiple linear regression in Scratch is feasible.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.