LAI #135: The Useful Part of Graph Engineering Is Not the Graph

2026-07-23 · Source: Learn AI Together · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

LAI #135, a daily intelligence brief, discusses "Graph Engineering" as a method for orchestrating multiple AI agent loops, noting its conceptual overlap with Anthropic's 2024 work but emphasizing its value in process formalization. The brief also highlights a critical issue with structured outputs from LLMs: while achieving a 100% parse rate, requiring rigid schemas can degrade answer quality, particularly for complex reasoning tasks. Other topics include a multi-agent support triage system adaptable for Llama3 or GPT-4o-mini, a pure NumPy implementation of Direct Preference Optimization (DPO) without a reward model, and an orchestrator pattern with a free demo mode. It also covers setting up a self-hosted LLMOps stack using Docker Compose for prompt versioning, tracing, and LLM-as-judge evaluations.

Key takeaway

For AI Engineers implementing LLM-powered applications, you must prioritize answer quality over strict schema adherence. If your application requires structured outputs, first evaluate the model's free-form response quality, then introduce the simplest possible schema. If accuracy drops, simplify further or package the free-form answer in code. This ensures your users receive correct information, not just perfectly formatted but potentially flawed data.

Key insights

Structured outputs can degrade LLM answer quality despite perfect parsing, requiring separate quality evaluation.

Principles

• Evaluate LLM answer quality and schema validity separately.
• Simplify schemas to preserve model reasoning capacity.
• Orchestration formalizes multi-agent system design.

Method

The orchestrator pattern decomposes tasks, runs parallel specialist workers over a shared blackboard, and uses a critic for capped retries on low-confidence answers.

In practice

• Test multi-agent systems in a free demo mode.
• Implement DPO from scratch using NumPy.
• Use Docker Compose for self-hosted LLMOps.

Topics

• LLM Orchestration
• Structured Outputs
• Direct Preference Optimization
• Multi-Agent Systems
• LLMOps Stack
• Statistical Hypothesis Testing

Code references

• mahmoudmoe84/review-toolkit

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• LAI #135: The Useful Part of Graph Engineering Is Not the Graph — Forcing structured outputs can degrade LLM reasoning; evaluate answer quality and schema validity separately.
• You Probably Don’t Need an Agent Framework — Explicit LLM workflows offer more reliability and control than autonomous agents for many applications.
• 🥇Top AI Papers of the Week — Agent systems benefit from structured memory, skill evolution, and robust evaluation beyond simple precision.
• Does the Structured Output schema become part of the prompt? — Structured Outputs ensure LLM responses conform to a predefined schema through dynamic constrained decoding, not by modifying the prompt text.
• Are Large Language Models Suitable for Graph Computation? Progress and Prospects — LLMs show promise for simple graph tasks but struggle with large-scale, exact graph computation, necessitating role-based integration.
• LLM Consortium for Software Design Refinement: A Controlled Experiment on Multi-Agent Collaboration Topologies — Structural adversarial and cross-model review topologies significantly enhance multi-agent LLM software design.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Learn AI Together.