Building production-ready LLM-powered applications @ Scale By the Bay

2023-10-31 · Source: The Full Stack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

Scale By the Bay is hosting a 10th-anniversary conference in San Francisco, featuring a workshop on building production-ready LLM-powered applications. Josh Tobin, CEO of Gantry and co-creator of Full Stack Deep Learning (FSDL) courses, will lead the workshop. The FSDL LLM boot camp, which previously sold out, focuses on pragmatic, real-world application development rather than just theoretical fundamentals. The course aims to equip product engineering teams, not just AI specialists, with the mindset and tools to move LLM prototypes to high-quality, user-centric products. A key methodology taught is test-driven development for AI applications, emphasizing continuous measurement and evaluation to ensure consistent performance and user value, moving beyond initial "good enough" versions to truly effective solutions.

Key takeaway

For AI Engineers and product teams aiming to deploy LLM applications, prioritize moving beyond initial prototypes by adopting a product-focused, test-driven development approach. Your focus should be on continuous evaluation and iteration to ensure the application consistently solves user problems, rather than just achieving initial functionality. This shift will help you build robust, high-quality products that deliver sustained user value and avoid early churn.

Key insights

Transitioning LLM prototypes to production requires a pragmatic, product-focused approach centered on continuous evaluation and user outcomes.

Principles

• Focus on user outcomes, not just technical functionality.
• Iterate rapidly beyond the first version.
• Evaluation is a contract between developers and users.

Method

Implement test-driven development for LLM applications by building an incremental test suite of example inputs and conversations, automating evaluation, and quantifying performance to guide iterative improvements.

In practice

• Start with UI-based prototyping (e.g., ChatGPT, OpenAI Playground).
• Immediately move to a platform for running prompts on multiple inputs.
• Encode opinions on good/bad outputs into quantitative metrics.

Topics

• Production LLMs
• LLM Test-Driven Development
• AI Model Evaluation
• AI Safety and Regulation
• Gantry

Best for: Machine Learning Engineer, AI Engineer, Software Engineer

Related on AIssential

• [AINews] WTF Happened in December 2025? — AI engineering seeks a "standard model" to guide development, moving beyond basic wrappers to structured, scalable applications.
• You can vibe code a demo, but what about a product? — Moving generative AI from demo to product requires overcoming significant non-model challenges in infrastructure, compliance, and evaluation.
• Online Evals Done Right: Runtime Scoring and Review Queues for Production LLM Systems — Effective online LLM evaluation prioritizes deterministic checks and structured escalation over early tooling adoption.
• Towards Structured Data: LLMs from Prototype to Production — Deploy LLMs beyond chatbots for structured data, moving NLP projects from prototype to production.
• Your LLM Eval Set Needs a Manifest — LLM evaluation examples require comprehensive metadata to provide context beyond file names for effective testing.
• When Claude changed, everything changed: Managing AI blast radius in production — LLM upgrades can introduce unpredictable "infinite blast radius" failures, necessitating robust evaluation as the true system specification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Full Stack.