Vector-Bench: Can Models Surgically Edit SVG Code?

2026-07-21 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Vector-Bench is a new, compact, and challenging benchmark designed to evaluate instruction-based SVG code editing models, focusing on both requested changes and the preservation of unrequested elements. It comprises 40 SVG repair tasks, each featuring a corrupted SVG program, an author-written visual instruction, a hidden target program, an average of 5.05 annotated repairs, and 60.55 protected objects. Instructions describe visible defects without exposing technical identifiers or data. The benchmark employs a deterministic binary specification reward, requiring requested repairs to meet attribute-aware perceptual tolerances while ensuring unrequested rendering- or application-relevant structure remains semantically unchanged and the output is a valid SVG. An evaluation of 34 model endpoints, including 25 open-weight, 5 control, and 4 frontier closed models, across 1360 requests revealed that the strongest endpoint achieved only 15.0% full specification success, despite a 43.7% mean repair progress, indicating a significant challenge in achieving precise, specification-faithful SVG editing. All prompts, outputs, scoring code, and reports are publicly released.

Key takeaway

For Machine Learning Engineers developing or evaluating code generation models for vector graphics, you should recognize that current models significantly underperform in precise, "surgical" SVG editing. Your evaluation metrics must extend beyond apparent visual repair progress to include strict semantic preservation of unrequested elements and valid SVG output. This benchmark highlights that achieving 15.0% full specification success is a major hurdle, indicating a need for new architectural approaches or training methodologies focused on code fidelity.

Key insights

Surgically editing SVG code requires precise changes while preserving untouched elements, a challenge current models largely fail.

Principles

• Instruction-based vector editing needs both change and preservation.
• Output judgment as raster images can mask unintended changes.
• Semantic preservation of unrequested structure is critical.

Method

Vector-Bench defines a binary specification reward: requested repairs meet perceptual tolerances, unrequested structure remains semantically unchanged, and output must be a valid SVG.

In practice

• Evaluate models on both repair progress and unintended changes.
• Use attribute-aware perceptual tolerances for SVG edits.
• Focus on semantic preservation of untouched code.

Topics

• SVG Editing
• Vector Graphics
• Code Generation Models
• Benchmark Evaluation
• Semantic Preservation
• Instruction Following

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• 1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World — Iterative UI code generation with visual feedback improves performance, but advanced training methods need refinement.
• Claude’s Hidden Art Skill: Making Illustrations With Code — Claude generates editable, scalable vector graphics (SVG) from text prompts, offering unique iterative refinement capabilities.
• SACE: Concept Erasure at the Semantic Singularity in Visual Autoregressive Models — SACE enables surgical concept erasure in VAR models by intervening only at the semantic singularity (Scale-0).
• Consistent-Inversion: Reverse Consistency Guidance for Structure-Preserving Visual Editing — Consistent-Inversion uses reverse consistency guidance to correct denoising trajectories, preserving structure in text-guided visual editing.
• The Benchmark No One Asked For! — A novel benchmark evaluates AI models' capacity for visual code generation, specifically SVG, rather than direct image synthesis.
• VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents — A new benchmark, VISTA, evaluates LLM agents' end-to-end web-app generation, revealing decoupled visual fidelity and functional correctness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.