Inference and infrastructure articles
- How to calculate LLM inference cost per successful request: compare serving configurations using costs, latency requirements, and accepted outputs.
- When does GPU scale-to-zero save money?: work through idle gaps, restart costs, and first-request latency.
- Lazy loading GPU containers: why startup is not readiness: distinguish image availability, model readiness, and the first useful response.
- Understanding multi-GPU communication and NVIDIA NCCL: learn about communication during distributed training and investigate watchdog timeouts.
Inference how-to guides
- Reduce LLM inference cost
- Serve a quantized model with vLLM
- Run vLLM on AMD GPUs
- Serve LLMs with AWS Neuron
- Run vLLM on Google TPUs
- Serve a local model with MLX on Apple silicon
- Run CPU inference with llama.cpp
- Compare Groq, Cerebras, and SambaNova APIs