Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted)

2026-05-25 · Source: PyImageSearch · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

This lesson, the second in a 3-part series on LLM observability, details manual tracing, evaluation, and scoring with Langfuse (Self-Hosted). It contrasts decorator-based tracing with manual control, highlighting the latter's necessity for complex RAG pipelines and agent workflows requiring custom spans, metadata, and step-level visibility. The content also introduces integrating custom quality metrics, like "answer_length" and "latency" thresholds, into Langfuse traces for real-time evaluation. Finally, it covers vLLM health checks, a diagnostic tool to ensure the underlying LLM server is operational, models are loaded, and text generation functions correctly before any tracing or scoring begins.

Key takeaway

For AI Engineers building complex RAG or agent-based LLM systems, adopting Langfuse's manual tracing and evaluation capabilities is crucial. This approach provides explicit control over trace structure, custom metadata, and quality scoring, enabling precise debugging and performance monitoring. Implement vLLM health checks first to ensure a stable environment, then integrate manual spans and custom metrics to gain actionable insights into model behavior and detect degradation.

Key insights

Manual tracing and custom evaluation in Langfuse provide granular control over LLM observability and performance analysis.

Principles

• Manual tracing offers precision for dynamic LLM pipelines.
• Hybrid tracing combines decorators for structure, manual for detail.
• Evaluation metrics enable real-time model degradation detection.

Method

Create a Langfuse trace, add generation spans, manually update with LLM outputs, token usage, latency, and custom metadata, then attach a numerical quality score.

In practice

• Use `langfuse.trace()` for root objects.
• Employ `trace.generation()` for LLM calls.
• Implement `langfuse_context.score_current_observation()` for quality.

Topics

• Langfuse
• LLM Observability
• Manual Tracing
• LLM Evaluation
• vLLM
• RAG Pipelines
• Agent Workflows

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Building a Self-Improving AI Support Agent with Langfuse — Langfuse enables robust LLM production systems through comprehensive tracing, metrics, and user feedback mechanisms.
• Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents — Insights Generator automates corpus-level diagnosis of LLM agent failures, providing evidence-backed behavioral patterns.
• How Do We Know AI Isn’t Lying? The Art of Evaluating LLMs in RAG Systems — Evaluating RAG systems requires multi-dimensional assessment of retrieval, understanding, grounding, and generation quality.
• langfuse / langfuse — Langfuse provides an open-source platform for LLM engineering, offering tools for development, monitoring, evaluation, and debugging.
• Livecoding: ICE and the Factored Cognition Primer by Ought — Iterated decomposition and execution tracing significantly enhance language model reliability and debugging.
• How Tool Chaining Fails in Production LLM Agents and How to Fix It — Tool chaining in LLM agents often fails silently in production due to cascading errors and context loss.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by PyImageSearch.