An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

2025-10-10 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

A new reference-free evaluation framework has been developed to monitor the quality of flowchart image-to-code generation in production environments, where ground-truth code is unavailable. This framework assesses output quality at inference time using only the input image and the generated code. It introduces two automated metrics: $\text{Recall}_{\text{OCR}}$, which estimates content coverage by extracting text from the input image via OCR as a proxy reference, and $\text{Precision}_{\text{VE}}$, which detects hallucinated elements through Visual Entailment against the original image. Their harmonic mean, $\text{F1}_{\text{OCR-VE}}$, provides a unified quality score. Validation on the FlowVQA dataset demonstrated strong agreement with ground-truth metrics, showing average Pearson's $r=0.97$ for Recall, \$0.91$ for Precision, and \$0.94$ for F1. Gemini 1.5 Pro achieved the highest OCR fidelity, while Gemini 2.5 Pro led on Visual Entailment, confirming the framework's reliability for continuous quality monitoring.

Key takeaway

For MLOps Engineers deploying Vision-Language Models in document processing pipelines, you can implement continuous quality monitoring for flowchart image-to-code generation without relying on ground-truth data. This framework allows you to detect missing content via $\text{Recall}_{\text{OCR}}$ and identify hallucinated elements through $\text{Precision}_{\text{VE}}$ at inference time, ensuring output quality and enabling targeted error triage. Prioritize selecting Visual Entailment models with low false positive rates, such as Gemini 2.5 Pro, to accurately identify critical hallucination errors.

Key insights

The framework enables reference-free, online quality monitoring for image-to-code generation using OCR and Visual Entailment.

Principles

• Production VLM evaluation needs reference-free metrics.
• Decouple generation and verification models.
• High-quality component models improve metric reliability.

Method

Decompose generated code into elements. Calculate $\text{Recall}_{\text{OCR}}$ using OCR-extracted text. Calculate $\text{Precision}_{\text{VE}}$ by verifying elements via Visual Entailment. Combine into $\text{F1}_{\text{OCR-VE}}$.

In practice

• Integrate as a quality gate in VLM pipelines.
• Use Gemini 1.5 Pro for OCR, Gemini 2.5 Pro for VE.
• Triage errors using lists of missed/hallucinated elements.

Topics

• Flowchart Image-to-Code
• Vision-Language Models
• Reference-Free Evaluation
• Optical Character Recognition
• Visual Entailment
• Production Monitoring

Best for: AI Scientist, Research Scientist, MLOps Engineer, AI Engineer, Machine Learning Engineer

Related on AIssential

• VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents — Image editing models require specialized benchmarks to accurately assess their performance on multilingual and densely textual visual documents.
• FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Mining — FreeStyle uses community LoRAs and a two-stage curriculum for robust style-content dual-reference image generation.
• 3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects — VLM-based 3D defect detection requires evaluating the entire pipeline, not just the model, for reliable automation.
• VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment — Integrating external knowledge graphs into VLMs significantly enhances domain-specific image descriptions and reduces factual errors.
• FlowEval: Reference-Based Evaluation of Generated User Interfaces — FlowEval offers a scalable, trustworthy method for evaluating generated UIs by comparing interaction traces.
• Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation — A hardened VLM-as-3D-judge protocol can specialize 3D generators, but exceeding strong baselines requires more than lightweight PEFT.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.