Query Profiling: See Where a Slow Query Spends Its Time
Summary
Weaviate has introduced a new query profiling feature, generally available since v1.38, to directly diagnose slow queries. This opt-in, per-query flag allows users to obtain a detailed timing breakdown for a specific search directly within the response, eliminating the need for server restarts or temporary configuration changes. Unlike the previous slow query log, which required environment variables like QUERY_SLOW_LOG_ENABLED=true and QUERY_SLOW_LOG_THRESHOLD=2s and provided per-node data, the new profiling aggregates timings across all participating shards and nodes into a single, comprehensive view. It covers vector search, keyword scoring, and filter evaluation, providing insights into stages like object hydration ("objects_took"), filter resolution ("filters_build_allow_list_took"), and vector index traversal ("vector_search_took"). This tool is designed for debugging and optimization, not for production hot paths, incurring microsecond-level overhead when active.
Key takeaway
For MLOps Engineers troubleshooting slow Weaviate queries, you should adopt the new query profiling feature. This allows you to precisely identify performance bottlenecks—whether it's disk-bound object hydration, an inefficient filter, or expensive vector traversal—without restarting nodes or sifting through fragmented logs. Integrate this opt-in flag into your debugging workflow to quickly pinpoint the exact stage requiring optimization, saving significant time and reducing guesswork in performance tuning. Remember it's for debugging, not production hot paths.
Key insights
Weaviate's query profiling provides granular, real-time performance breakdowns for complex database searches, consolidating distributed timings into a single view.
Principles
- Accurate query performance diagnosis requires granular, stage-level timing data.
- Distributed query timings must be aggregated for a complete, cluster-wide picture.
- Debugging tools should minimize interference with the system being measured.
Method
Enable query profiling via a per-query flag in client requests (e.g., "return_metadata=MetadataQuery(query_profile=True)"). The coordinating node gathers and returns timing breakdowns from all participating shards and nodes.
In practice
- Pinpoint disk-bound object hydration by analyzing "objects_took" values.
- Identify overly broad filters by high "filters_build_allow_list_took" and "filters_ids_matched".
- Differentiate between compute-intensive vector traversal and disk-latency during rescoring.
Topics
- Weaviate
- Query Profiling
- Vector Search
- Performance Tuning
- Distributed Queries
- Database Optimization
Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by Weaviate Blog.