Building a high-volume metrics pipeline with OpenTelemetry and vmagent

2026-04-07 · Source: The Airbnb Tech Blog - Medium · Field: Technology & Digital — Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

A large-scale metrics pipeline was successfully migrated from a StatsD-based system to one utilizing OpenTelemetry Protocol (OTLP) and Prometheus. This transition involved a dual-write strategy, allowing services to emit both StatsD and OTLP metrics, which significantly reduced CPU time spent on metrics processing from 10% to under 1% and improved reliability. The team adopted the OpenTelemetry Collector and a Prometheus-based backend like Grafana Mimir. A key challenge with high-volume OTLP metrics, causing memory pressure, was addressed by selectively implementing delta temporality. For streaming aggregation and cost control, the team chose vmagent, deploying it in a two-layer sharded architecture that scaled to hundreds of aggregators, ingesting over 100 million samples per second and reducing costs by an order of magnitude. Finally, a "zero injection" technique was developed within the vmagent aggregation tier to resolve undercounting issues with sparse Prometheus counters, ensuring data accuracy.

Key takeaway

For MLOps or DevOps Engineers migrating high-volume metrics to OpenTelemetry and Prometheus, prioritize a dual-write strategy to manage transition friction. You should adopt OTLP for its performance benefits, but be prepared to use delta temporality for extremely high-cardinality services to prevent memory issues. Implement a sharded streaming aggregator like vmagent to control costs and ensure accurate sparse counter reporting via zero injection. This approach provides a robust, scalable, and cost-efficient observability pipeline.

Key insights

Migrating high-volume metrics to OpenTelemetry and Prometheus requires careful instrumentation, scalable aggregation, and addressing counter semantics.

Principles

• Dual-write strategies ease large-scale migrations.
• OTLP offers superior performance and reliability over StatsD.
• Centralized aggregation reduces costs and enables transformations.

Method

Implement a dual-write strategy for OTLP and StatsD. Use OpenTelemetry Collector for collection. Deploy sharded vmagent for streaming aggregation. Inject zeros at the aggregation tier for sparse counter accuracy.

In practice

• Use delta temporality for high-volume OTLP metrics.
• Configure vmagent routers for consistent sharding.
• Implement zero injection for accurate sparse counters.

Topics

• OpenTelemetry
• Prometheus
• vmagent
• Metrics Migration
• Streaming Aggregation
• Observability Pipelines

Code references

• statsd/statsd
• stripe/veneur
• open-telemetry/opentelemetry-collector-contrib
• VictoriaMetrics/VictoriaMetrics

Best for: MLOps Engineer, DevOps Engineer, Software Engineer

Related on AIssential

• Amazon CloudWatch Introduces OpenTelemetry Metrics Support in Preview — CloudWatch now natively ingests OpenTelemetry metrics, completing its observability suite and simplifying metric collection.
• Building a Production RAG Pipeline: Webhooks, Deduplication, and 40k Documents — Production RAG pipelines are primarily data engineering challenges, requiring robust solutions for continuous updates and data integrity.
• Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops — Effective GenAI operations require robust observability, cost management, and flexible prompt handling to ensure reliable and cost-effective LLM applications.
• Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus — NCCL Inspector's Prometheus Mode enables real-time GPU communication monitoring for AI workloads, improving performance debugging.
• When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI — GPU utilization alone is insufficient; I/O bottlenecks and resource fragmentation significantly reduce GenAI system productivity and increase costs.
• Rethinking Data Movement: A First Principles Approach — Modern data movement prioritizes incremental, observable, and extensible pipelines to address current data complexity, latency, and cost challenges.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Airbnb Tech Blog - Medium.