My AI Pipeline Scored 0.97

2026-07-06 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, quick

Summary

The "medium-agent-factory", a 25-node LangGraph pipeline, was developed to write, evaluate, and revise Medium posts. It processed 250 files, including 40 test files and package manifests, to grasp the full architecture. Initially, the pipeline excelled at mimicking a human writing style, achieving high voice scores with varied sentence rhythm and correct contractions, making its output indistinguishable from human authors. However, despite this success, a significant flaw was discovered: roughly every third paragraph contained claims lacking attribution or citing non-existent sources. For example, run #33 identified eight "unattributed_claim" violations in a single draft, demonstrating that while the pipeline passed the "human voice" test, it critically failed the "is this true" test.

Key takeaway

For AI Engineers developing content generation pipelines, prioritize factual accuracy and attribution alongside stylistic quality. Your pipeline might pass a "human voice" test but still produce significant hallucinations, as seen with 51 unattributed words per post. Implement explicit attribution checks and quality gates that specifically flag unsubstantiated claims. This ensures your AI-generated content is not only engaging but also trustworthy, preventing critical failures in factual integrity.

Key insights

AI pipelines can achieve human-like voice but still hallucinate, failing attribution despite high quality scores.

Principles

• Human-like voice does not guarantee factual accuracy.
• Quality metrics can miss critical factual errors.
• Attribution is a distinct challenge from stylistic mimicry.

Method

The article describes building a 25-node LangGraph pipeline ("medium-agent-factory") that writes, evaluates, and revises posts by scanning repo files. It uses a quality gate to flag "unattributed_claim" violations.

In practice

• Implement specific attribution checks in AI pipelines.
• Design quality gates for factual accuracy, not just style.
• Validate AI-generated claims against external sources.

Topics

• LangGraph
• AI Pipelines
• Content Generation
• Hallucination Detection
• Attribution
• Quality Gates

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• KPMG pulls report on AI usage due to apparent hallucinations — AI-generated content requires rigorous human oversight to prevent factual inaccuracies and reputational damage.
• AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps — Automating test case generation from natural language requirements with AI faces significant challenges, particularly in quality dimensions like hallucination and traceability.
• The Internet Was Always Bad at Writing. Now It's Just More Obvious. — AI amplifies existing writing quality issues, making human voice, judgment, and responsibility more critical than ever.
• AI Gets Rewarded For Lying To You — AI systems based on next token prediction inherently "hallucinate" all outputs due to a lack of meta-cognition.
• Every AI output is a hallucination — Large Language Models inherently produce fluent, confident outputs that are often wrong yet indistinguishable from truth, posing significant reliability issues.
• Fellini cameo in Juliet — AI models can generate plausible but factually incorrect information, known as hallucinations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.