Evaluating Deep Agents using LangSmith on AWS

2026-05-28 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

This post, co-authored with LangChain, details a practical guide for evaluating deep AI agents using LangSmith on AWS, specifically focusing on a text-to-SQL agent with Amazon Bedrock and Amazon Nova 2 Lite. It outlines five evaluation patterns: custom test logic per datapoint, single-step evaluations, full agent turns, multi-turn conversations, and safety/state checks. The article also describes three types of graders—code-based, model-based (LLM-as-judge), and human—and how to combine them. It demonstrates building offline evaluations using Pytest and LangSmith, and configuring online monitoring for production with LangSmith's online evaluators. The example utilizes Amazon Nova 2 Lite, a fast, cost-effective reasoning model in Amazon Bedrock, supporting a 1 million-token context window.

Key takeaway

For AI Engineers validating deep agent behavior, you should implement a robust evaluation framework combining offline and online strategies. Integrate LangSmith's Pytest integration for development-phase testing, utilizing code-based, LLM-as-judge, and human graders across single-step, full-turn, and multi-turn scenarios. For production, configure LangSmith's online evaluators for continuous monitoring of safety and quality, ensuring agent reliability and catching issues early.

Key insights

Evaluating deep AI agents requires a multi-faceted approach combining diverse grading methods and evaluation patterns across the development lifecycle.

Principles

• Agent evaluation needs multiple trials due to non-determinism.
• Evaluate trajectory, final response, and other state artifacts.
• Combine deterministic, LLM-based, and human graders for robustness.

Method

Apply five evaluation patterns: custom logic per datapoint, single-step, full agent turns, multi-turn, and safety/state checks, using LangSmith's Pytest integration for offline testing and online evaluators for production.

In practice

• Use "pytest.mark.langsmith" for automatic trace logging.
• Configure online evaluators for production monitoring.
• Calibrate LLM-as-judge with human expert feedback.

Topics

• AI Agent Evaluation
• LangSmith
• Amazon Bedrock
• Amazon Nova 2 Lite
• LLM-as-Judge
• Pytest Integration
• MLOps

Code references

• langchain-ai/deepagents
• lerocha/chinook-database
• aws-samples/sample-text2sql-deep-agent-evalulation

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Harrison Chase of LangChain on Deep Agents, LangSmith, and Earning Trust | NVIDIA AI Podcast Ep. 297 — Deep Agents provide a general-purpose, open-source harness for building autonomous, event-driven LLM applications with enhanced control and observability.
• Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize — Formal evaluations are crucial for AI agents, moving beyond subjective "vibe checks" to ensure reliability and performance.
• You Shipped an AI Agent to Production Without Testing It. So Did I — Layered testing and evaluation gates are crucial for safe AI agent deployment in high-liability workflows.
• Evaluating AI Agents: A Practical Guide with Microsoft Foundry — Early and continuous evaluation is crucial for AI agents to catch regressions and ensure quality.
• Langfuse vs LangSmith: Two Competing AI Observability Platforms Compared — Effective AI agent debugging requires deep observability into internal decision processes, not just final outputs.
• Build custom code-based evaluators in Amazon Bedrock AgentCore — Custom code-based evaluators in Amazon Bedrock AgentCore enable deterministic, domain-specific quality checks for agentic applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.