Self-Hosted LLM Inference

Building and Operating Production AI Systems from a Single GPU to Multi-Node Frontier Models

Getting a large language model to load on your own hardware takes an afternoon. Keeping it serving real traffic, at a cost you can predict, is a different job, and it is the one this book covers. It does so by building one system and growing it across all fifteen chapters.

You begin with an 8B model on a single card, the kind you may already own or can rent for well under a dollar an hour. From there the book adds what real traffic needs: a choice of inference engine, a control plane that provisions GPUs and tears them down again, a router and queue that decide what runs next, and cache-aware routing that stops a conversation being re-read from scratch on every turn. Part III moves to models that do not fit on one card, placing a 70B deployment across GPUs chosen for the bandwidth between them as much as for their memory, and buying that capacity from several providers at once.

In the final part you serve a 753-billion-parameter sparse model on eight rented H200s, measure what each session costs as concurrency rises, and find the point where a larger batch stops making sessions cheaper. That result explains how commercial inference providers price what they sell, and when self-hosting is cheaper than their APIs.

Costs come from real rented-GPU runs, and the book says which numbers are measurements and which are predictions. The control plane is open source and written in Go, so you can run every step yourself.

You Will:

  • Build a working inference control plane in Go that provisions GPUs across providers, deploys engines onto them, routes and queues requests, and reports what it is doing.
  • Learn how inference uses a GPU, covering KV cache growth, quantization formats and what they cost, tensor and expert parallelism, and why a fast interconnect matters more than total memory.
  • Work out per-session cost from real runs as batch size and context length grow, and find the point where self-hosting stops beating a commercial API.

This book is for:

Backend, systems and platform engineers who want to run LLMs on infrastructure they control. Also useful to infrastructure and DevOps engineers building internal AI platforms, and to technical founders with data-residency or cost constraints that rule out third-party APIs. No machine learning background is assumed. Readers should be comfortable with a systems language such as Go or Python, with containers, and with HTTP APIs.

April 2027, ca. 256 Seiten, Englisch
Springer EN
979-8-8688-3544-5

Weitere Titel zum Thema