11 min readApplied
Top 10 Inference Engineering Interview Questions for 2026
Ten inference engineering interview questions grouped by the physical constraint they test, each with an answer skeleton and the common trap.
Inference, Serving & Cost
GPU economics — latency, throughput, and the bill.
15 posts
7 min readDeep dive
Tensor vs Pipeline Parallelism: Splitting Models Across GPUs
Tensor parallelism splits the math inside every layer; pipeline parallelism splits the layers themselves. The cost each pays, and when engines combine both.
10 min readPrimer
What Is Inference Engineering? A First-Principles Map
Inference engineering makes a trained model fast and cheap in production under two physical constraints. Here is the map from constraint to lever to deep dive.
7 min readDeep dive
Test-Time Compute: The Model Decides Your Bill
Reasoning models spend output tokens to think, and they pick how many. Why test-time compute breaks every serving optimization that assumed fixed output.
6 min readDeep dive
Mixture of Experts: Cheap to Compute, Expensive to Hold
Mixture-of-experts models activate a fraction of parameters per token, but every expert still sits in GPU memory. Mechanism, real numbers, and the payoff math.
6 min readPrimer
Inference Engineering for CEOs: Train or Rent?
Should you train and serve your own AI model, or pay per token via an API? A GPU cost breakdown, a breakeven formula, and the exact ICP for each path.
10 min readDeep dive
Disaggregated Inference: When KV Cache Crosses the Network
Splitting prefill and decode onto separate GPU pools is now production, not research. It also turns the KV cache into a network problem — here's the tradeoff.
8 min readDeep dive
Inference Engineering: TTFT, TPOT, and Two Clocks
TTFT and TPOT measure two different things in LLM inference. Optimize the wrong one for your workload and you fix nothing. Here's the engineer's map.
10 min readDeep dive
The KV Cache Is Your Concurrency Ceiling
Model weights are a fixed cost. The KV cache is the variable one, and its arithmetic decides how many users one GPU can serve at the same time.
13 min readDeep dive
Prefill and Decode: Why One Model Has Two Bottlenecks
LLM inference runs in two phases with opposite limits — prefill is compute-bound, decode is memory-bound. Where they collide is where your latency spikes.
14 min readDeep dive
Continuous Batching Is How One GPU Serves a Crowd
Static batching leaves your GPU idle waiting for the slowest request. Continuous batching swaps requests in and out every step — up to 23x more throughput.
14 min readDeep dive
Speculative Decoding: Faster Tokens, Identical Answers
A small draft model guesses tokens, a big model verifies them — 2-3x faster LLM generation with mathematically identical output. Here's the catch.
9 min readDeep dive
The Serving Layer Sets Your RAG Latency Floor
vLLM, TensorRT-LLM, and SGLang all fix the same problem — GPU throughput. None of them fix a RAG system that answers fast from the wrong context.
16 min readDeep dive
Quantization Is the Cheapest Way to Cut Your Inference Bill
Quantization shrinks model weights to cut LLM inference cost and memory 2-4x with little quality loss. Here is the tradeoff — and the one problem it can't fix.
14 min readDeep dive
Prompt Caching Cuts Cost. Semantic Caching Cuts Corners.
Prompt/prefix caching is a bankable inference discount on stable prefixes. Semantic caching promises bigger savings by reusing answers — and quietly serves stale ones. Here's the cost math for both.
Inherent on Substack
Keep yourself updated on the latest in AI news and trends.
Everything you need to know about AI, delivered to your inbox. Every week.
Powered by Substack. Unsubscribe anytime.