Hot French startup ZML releases free product to speed inference across lots of AI chips

2026-07-08 · Source: AI News & Artificial Intelligence | TechCrunch · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Emerging Technologies & Innovation · Depth: Intermediate, short

Summary

French AI startup ZML has launched ZML/LLMD, a free LLM inference server designed to accelerate large language model performance across a diverse range of AI chips, including those from Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc. Endorsed by Yann LeCun, ZML aims to dismantle existing hardware silos, enabling enterprises and cloud providers to utilize mixed chip environments for maximum efficiency and reduced AI-related costs. Founder Steeve Morin highlights the software's potential to support novel AI chipmakers like Axelera and Fractile. While competing with established players such as Baseten, Inferact (vLLM), and RadixArk (SGLang), ZML distinguishes itself with ambitions for silicon co-design. The company, a lean team of 20, secured \$20 million in funding from investors including 20VC and Kima Ventures, and plans to gather usage data before potentially monetizing ZML/LLMD.

Key takeaway

For MLOps Engineers managing LLM deployments, ZML/LLMD offers a compelling solution to diversify your inference hardware strategy. You can now run open-source LLMs efficiently across a mix of Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc chips, potentially reducing operational costs and energy consumption. Evaluate ZML/LLMD to mitigate vendor lock-in and optimize your infrastructure for greater flexibility and efficiency.

Key insights

ZML/LLMD offers cross-platform LLM inference optimization to break hardware silos and reduce AI operational costs.

Principles

• Inference optimization is critical for AI dissemination and efficiency.
• Software can mitigate hardware vendor lock-in across diverse chip architectures.
• Free product launches enable usage data collection before monetization.

In practice

• Deploy ZML/LLMD for LLM inference across mixed chip environments.
• Evaluate software solutions for reducing AI inference costs and energy use.

Topics

• LLM Inference
• Multi-chip Support
• Hardware Acceleration
• AI Cost Optimization
• Vendor Lock-in
• ZML/LLMD

Code references

• zml/zml

Best for: AI Architect, Machine Learning Engineer, NLP Engineer, AI Engineer, MLOps Engineer, Director of AI/ML

Related on AIssential

• 3 Agents. 3 LLMs. 1 Aging GPU: Engineering Parallel Inference on Bare Metal — A C++ daemon enables parallel LLM inference on limited GPU VRAM by centralizing resource management and KV-cache swapping.
• The Strangest Bottleneck in Modern LLMs — TiDAR unifies autoregressive and diffusion models to achieve parallel LLM inference, significantly boosting speed without sacrificing accuracy.
• llama.cpp: Fast Local LLM Inference, Hardware Choices & Tuning — llama.cpp is the dominant open-source C/C++ framework enabling fast local large language model (LLM) inference, leveraging hardware advancements like NVIDIA's RTX 5090 and Apple's M4 Ultra for privacy…
• LLM‑D Explained: Building Next‑Gen AI with LLMs, RAG & Kubernetes — LLM-D optimizes LLM inference by intelligently routing requests and disaggregating prefill/decode phases on Kubernetes.
• not much happened today — AI development emphasizes rapid model expansion, agentic orchestration, and hardware-optimized inference for diverse applications.
• What is your current local LLM setup? — The community actively explores diverse local LLM setups, balancing hardware, software, and model choices for varied applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI News & Artificial Intelligence | TechCrunch.