ZML Ships a Free LLM Server That Runs on Any Chip

A 20-person startup from Paris just shipped a LLM inference server that runs on Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc - the same binary, no separate deployment configs per chip vendor. ZML's new LLMD launched today as a free product, putting itself directly against vLLM, SGLang, and Baseten in the inference serving market.

The company has been building Zig-based ML infrastructure since 2024. LLMD is the first product aimed at operations teams rather than researchers - a drop-in inference server with an OpenAI-compatible API that you can swap underneath a production workload without touching application code.

What LLMD Does Differently

Any Chip, One Server

Most inference servers pick a lane. vLLM is a strong Nvidia-first system that added ROCm support later, but Nvidia-specific optimizations dominate its roadmap. SGLang follows a similar pattern. Baseten's managed service abstracts hardware from the user's perspective, but you're paying Baseten to do that for you.

LLMD compiles directly to each target's native stack: CUDA for Nvidia, ROCm for AMD, XLA for Google TPU. In theory that means hardware-specific performance without separate deployment configs per chip type. The underlying ZML framework uses OpenXLA and MLIR as the compilation backend, so ZML doesn't need to maintain hand-tuned kernels for every architecture. The compiler handles lowering.

The API Layer

The server exposes an OpenAI-compatible REST API. In practice, swapping an existing OpenAI client to a locally-hosted Llama requires one environment variable change:

OPENAI_BASE_URL=http://localhost:8080/v1 \
OPENAI_API_KEY=dummy \
python my_existing_llm_app.py

LLMD handles continuous batching and paged attention out of the box. Those two features separate production inference servers from naive single-request implementations. Tensor sharding across multiple cards is also supported. Cross-vendor sharding - splitting a model across one Nvidia card and one AMD card - isn't documented in the current release.

Under the Hood

Zig and OpenXLA, Not Python

The ZML framework is written in Zig, a systems language designed for explicit memory management with no hidden allocations. No Python runtime overhead, no GIL, and far fewer surprise memory spikes under concurrent batches. The tradeoff: contributing to ZML's core requires Zig familiarity, which limits the community contributor pool compared to a Python-first project.

The compilation stack is OpenXLA and MLIR, both originating from Google's AI infrastructure group. This lets ZML express tensor operations at a high level and lower them to hardware-native code per target, similar to how JAX works internally. The key difference from JAX is that ZML targets inference, not training. That constraint makes the compilation path more predictable and the runtime behavior more deterministic under production load.

ZML's framework on GitHub: 3.5K stars, built with Zig, OpenXLA, MLIR, and Bazel. Source: github.com

The Distributed Inference Angle

An earlier ZML demo showed pipeline parallelism across physically separate machines running different chip types - a Nvidia GPU in one location, an AMD card in another, and a Google Cloud TPU instance, collaborating on a single Llama 2 inference. Network latency was present but manageable for non-interactive workloads.

LLMD doesn't lead with this feature, which is sensible engineering discipline. Cross-vendor distributed inference with acceptable latency requires specific network topology and isn't a common enterprise deployment pattern yet. But the demo uncovers that ZML's architecture wasn't designed around a single chip vendor from the start. That design decision is now a product advantage as mixed-hardware data centers become more common.

Who's Behind It

Steeve Morin's Background

Founder Steeve Morin spent years as VP of Engineering at Zenly, a location-sharing app that Snapchat bought in 2017 for nine figures. Zenly scaled to tens of millions of users before Snap shut it down in 2023. That background in low-latency mobile infrastructure at scale doesn't map directly to ML systems, but the underlying concerns are similar: predictable latency, efficient memory management, and high concurrency under unpredictable workloads.

ZML was founded in 2023, roughly six months after Zenly's shutdown. The team is 20 people.

A Cap Table Worth Reading

ZML raised $20M from institutional investors including 20VC (Harry Stebbings), Kima Ventures (Xavier Niel), Kindred Capital, LocalGlobe, and >commit. The angel list is more interesting for what it signals about where the problem sits in the ecosystem.

Solomon Hykes, who built Docker and then Dagger, is on the cap table. Clément Delangue and Julien Chaumond, the founders of Hugging Face, are in. So is Yann LeCun, now Chief AI Scientist at AMI Labs, who has argued publicly for hardware diversity in AI infrastructure rather than Nvidia consolidation.

Docker became the standard container runtime because it abstracted away OS and kernel differences for application developers. ZML is betting on a similar abstraction at the chip level - and has assembled backers who understand what that looks like when it works at scale.

The Market ZML Is Targeting

Inference serving is a real commercial problem. Baseten recently closed a $1.5B funding round, which implies substantial enterprise revenue in the space. Most of that market runs on Nvidia today, because CUDA's tooling depth and community kernel library are truly hard to compete with on Nvidia hardware.

The problem for teams with mixed hardware - AMD Instinct cards, Google Cloud TPU allocations, Intel Arc Pro B70 nodes - is managing separate inference stacks per chip type, each with its own configuration, monitoring, and model loading code. LLMD's pitch is a single stack that replaces all of them.

That matters most to organizations that picked up AMD hardware through large-scale deals, or to sovereign AI deployments in countries that won't route workloads through US cloud providers. Kima Ventures, backed by Xavier Niel, understands that market well.

ZML's landing page: "Any model. Any hardware. Zero compromise." Source: zml.ai

For context on how inference serving is evolving alongside orchestration, the llm-d project recently moved to CNCF as a Kubernetes-native alternative for teams that want inference integrated with existing container infrastructure.

Where It Falls Short

LLMD isn't open source. ZML's framework is - Zig, GitHub, Apache 2.0, 3.5K stars - but the inference server itself is proprietary. ZML's stated reason is to learn from production usage before opening the code. That's a reasonable engineering position, but it means taking a dependency on a 20-person startup's continued commitment to the free tier. There's no guarantee LLMD stays free as ZML finds its commercial footing.

The model support list - Llama, Gemma, Mistral, Ministral, Qwen - covers the most widely launched open-weight families but excludes newer models with significant developer adoption. DeepSeek V4 and Kimi K2, which handle a meaningful and growing share of developer token traffic, aren't listed. Support for those models will determine whether LLMD can compete where workloads are actually shifting.

There are no independent benchmark numbers available. ZML claims peak hardware performance across chip targets. But claims without reproducible results are marketing until proven otherwise. The relevant numbers for an inference server are tokens per second at various batch sizes and concurrency levels, measured on specific hardware configurations with specific model sizes - not general claims about "maximum available speed." The HuggingFace test drive published by the community lists the current maximum batch size as 16, which suggests the technical preview isn't yet optimized for high-throughput production workloads.

The competition isn't standing still. vLLM's FlashAttention 4 integration and elastic parallelism in 0.17.0 shows an active development community that improves Nvidia performance on a fast cadence. SGLang has similar momentum. LLMD has a genuinely different architecture, and that matters, but "different" needs to translate into benchmark wins before it becomes a migration argument for teams already running well on vLLM.

Sources:

• Hot French startup ZML releases free product to speed inference across lots of AI chips (TechCrunch)
• ZML's Hardware-Agnostic Distributed Inference (Chipstrat)
• Test-Driving the LLMD Inference Engine (HuggingFace Blog)
• ZML - Model to Metal (zml.ai)
• ZML GitHub Repository
• ZML/LLMD on Docker Hub