Concurrency Foundations For FastHTML

2024-10-11 · Source: Hamel Husain's Blog · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

This article, published October 11, 2024, explores various concurrency approaches for Starlette-based Python web applications like FastAPI and FastHTML, particularly for handling non-blocking background tasks such as LLM API calls. It demonstrates using a SQL database (via `fastlite`) as a queue for task management, including `enqueue` and `dequeue` operations with a 5-minute expiry mechanism. The post then details how to use Python's `concurrent.futures.ThreadPoolExecutor` to run queue processing in background threads. It also compares synchronous and asynchronous processing for OpenAI API calls, showing async processing can achieve a 2.29x speedup, and introduces `asyncio.Semaphore` for limiting concurrent async tasks. Finally, it illustrates integrating async tasks into a FastHTML application and briefly touches on `fastcore` utilities like `@threaded` and `startthread` for managing threads and processes.

Key takeaway

For Machine Learning Engineers building Starlette or FastAPI applications that integrate with external APIs like LLMs, prioritize asynchronous programming with `asyncio` to prevent blocking the main server process. Consider using a SQL database as a simple task queue for persistent background jobs, and leverage `ThreadPoolExecutor` for I/O-bound tasks or `fastcore`'s `@threaded` decorator for simplified thread management to ensure responsive user experiences.

Key insights

Implement concurrency in Starlette apps using database queues, threads, or async for non-blocking background tasks.

Principles

• Databases can serve as simple task queues.
• Threads suit I/O-bound tasks; processes suit CPU-bound tasks.
• Async processing excels for waiting on external APIs.

Method

To implement a database queue, define a table with `id`, `data`, and `expire` fields. Use `UPDATE SET expire = future_time WHERE id = item.id AND expire = 0` for atomic dequeueing.

In practice

• Use `ThreadPoolExecutor` for background thread processing.
• Employ `asyncio.create_task` to run async functions in the background.
• Limit concurrent async tasks with `asyncio.Semaphore`.

Topics

• Concurrency
• Asynchronous Programming
• FastAPI
• Background Tasks
• LLM Integration

Code references

• hamelsmu/hamel-site
• AnswerDotAI/fastlite

Best for: Software Engineer, Machine Learning Engineer, AI Engineer

Related on AIssential

• Every Python Concept a Generative AI Developer Actually Needs to Know — Mastering Python's concurrency, memory, and structural patterns is vital for building scalable and robust GenAI applications.
• Why is synchronous IO slow? — Synchronous I/O blocks the main process, while asynchronous I/O offloads blocking operations to maintain responsiveness.
• AI for Systems: Using LLMs to Optimize Database Query Execution — LLMs can semantically optimize database query plans by correcting cardinality misestimates without engine modification.
• Stochastic convergence of parallel asynchronous adaptive first-order methods — Asynchronous adaptive first-order methods achieve O(1/sqrt{t}) convergence on non-convex functions in stochastic settings.
• Why Your LLM App Works in Notebooks But Fails in Production — Decoupling long-running LLM API calls from the web server's request-response cycle prevents production failures.
• Python Concurrency: The Tricky Bits — Python concurrency requires understanding threads, processes, and coroutines to optimize for CPU-bound versus I/O-bound tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hamel Husain's Blog.