Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
-
Updated
Nov 9, 2026 - Markdown
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
A curated list of awesome inference engineering tools, papers and resources.
A curated, continuously verified map of inference engineering: tools, techniques, hardware, and knowledge for serving AI models.
Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.
A practical handbook for software engineers to learn AI, Large Language Models (LLMs), and Inference Engineering—from fundamentals to production systems.
KI ohne Cloud: a hands-on inference engineering workshop. Run, measure, optimize and operate open models with vLLM on your own infrastructure (nextacademy.io)
it's me as a repository
CPU-only conversational voice assistant with VAD-based speech segmentation, streaming LLM response generation overlapped with concurrent TTS synthesis via producer-consumer queues. Batch STT (Whisper). Fully instrumented with per-stage latency benchmarking.
LLM inference benchmarking dashboard: Python FastAPI backend with async orchestration, WebSocket live TTFT/TBT/throughput comparison across configs (512/128 to 4096/1024 tokens), Grafana + Docker Compose stack, GitHub Actions CI; 21/21 pytest passing.
Evidence-gated benchmark, comparison and promotion control plane for LLM inference serving changes.
Private AI on NeoClouds - Large Open Weight models need a lot of VERY large GPUs (aka massive infra)
An interactive playground for learning inference engineering—explore LLM serving concepts, tune the stack, and graduate to production incidents.
Trace-Aware Serving Controller: eval-gated inference policy optimization
Deterministic simulator for KV-cache admission, placement, movement, eviction, and recomputation policies
Offline, evidence-first bottleneck hypotheses for LLM inference traces
Deep Agents and SvelteKit harness for authoring verifier-gated Bonsai workflow packs.
Interactive inference-engineering lessons and a bilingual AI mentor, built with Next.js and the Vercel AI SDK.
A hands-on lab for how AI models get served: GPU memory, batching, the scheduler, parallelism and fleet economics. Drag sliders instead of reading the book.
Executable field guide and deterministic labs for LLM inference engineering across kernels, scheduling, KV cache, placement, and control.
To associate your repository with the inference-engineering topic, visit your repo's landing page and select "manage topics."