Three Design Decisions in LLM Observatory and Why I Made Them

2026-05-11 · Source: Naturallanguageprocessing on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The "LLM Observatory" platform employs three distinct design decisions to balance functionality, cost, and developer experience. First, it uses a dual authentication model: JWTs for human operators requiring role-based access and session management, and service API keys for machine-to-machine integrations needing scoped, revocable credentials. Second, it prioritizes heuristic evaluation before LLM-as-judge, enabling dependency-free demonstrations, robust schema validation for evaluation versioning, and explicit cost control. Third, the platform adopts a SQLite-first, PostgreSQL-forward storage strategy, facilitating rapid local development and schema validation while providing a clear, documented path to a production-grade PostgreSQL deployment. These choices introduce some complexity and trade-offs in precision or local-production parity, but ensure each component is fit for its specific purpose.

Key takeaway

For AI Architects or MLOps Engineers designing internal LLM platforms, consider adopting a dual authentication model to cleanly separate human and service access requirements. Prioritize heuristic evaluation for initial platform demonstrability and cost control, deferring more expensive LLM-as-judge methods. Additionally, starting with SQLite for local development before transitioning to PostgreSQL for production can significantly accelerate schema validation and developer onboarding, even with documented local-production gaps.

Key insights

Dual authentication, heuristic-first evaluation, and SQLite-first storage optimize LLM platform development and operation.

Principles

• Match authentication to caller type.
• Control evaluation cost and dependencies.
• Prioritize local development velocity.

Method

Implement JWTs for human operators and API keys for services. Build heuristic evaluation first for demonstrability and cost control. Start with SQLite for local development, then migrate to PostgreSQL for production.

In practice

• Use `POST /api/v1/auth/register` for human JWTs.
• Use `POST /api/v1/api-keys` for service API keys.
• Validate schema with Alembic against SQLite first.

Topics

• LLM Observatory
• Authentication Design
• Evaluation Sequencing
• Heuristic Evaluation
• LLM-as-Judge

Best for: MLOps Engineer, AI Engineer, AI Architect

Related on AIssential

• Security in an LLM Observability Platform: Controls, Tradeoffs, and What Is Still Open — LLM observability platforms require integrated security controls for sensitive prompt and completion data.
• How I Choose Between GPT, Claude, Gemini and Open Source Models for Every Task — Optimizing LLM usage requires a decision framework matching tasks to models for cost and quality.
• LLMs: A Journey Through Time and Architecture — LLM evolution prioritizes data quality, multi-stage pre-training, and architectural efficiency over sheer model size.
• When the Judge Is Wrong — LLM-as-judge is unreliable for structured tasks, even with ground truth or source documents.
• Legare Kerrison and Cedric Clyburn on LLM Performance and Evaluations — Effective LLM deployment requires balancing model quality, responsiveness, and cost through tailored evaluation and optimization.
• Presentation: Accelerating LLM-Driven Developer Productivity at Zoox — Zoox built an internal AI platform, Cortex, to enhance developer productivity through RAG, multi-modal LLMs, and agent APIs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Naturallanguageprocessing on Medium.