Skip to main content
Updated September 6, 2026 · Worked example, not a measured benchmark A serving configuration can generate more tokens per second and still cost more per useful request. That happens when its requests miss the latency target, need retries, or produce outputs the application cannot use. To compare inference cost, divide the full cost of serving a workload by the number of requests that meet the application’s acceptance criteria.

​
Define a successful request before benchmarking

For a streaming assistant, acceptance might require a correct response, a first token within two seconds, and an acceptable delay between subsequent tokens. For extraction, it might require valid JSON, correct fields, and completion before a deadline. The thresholds depend on the product. Keep quality and performance checks separate so a failure can be explained. A request can return HTTP 200 while failing either check. Establish a fixed evaluation set and keep the model, prompt templates, output limits, and correctness rubric constant when comparing configurations. Goodput measures throughput subject to specified latency objectives in vLLM’s serving benchmark. It helps distinguish raw throughput from timely responses. Application-level acceptance adds another requirement: the response must also be useful. A latency benchmark alone does not measure answer quality.

​
Use the full cost numerator

For a chosen measurement window:
Include billed compute during startup and idle periods, retained cluster infrastructure, storage, network charges, and applicable platform charges. Include the compute consumed by failed attempts and retries. Count a user request once when a retry eventually succeeds; counting every attempt as a separate success makes retries look productive. For self-hosted inference, record the AWS bill components attributable to the service. For a hosted API, record the actual billed input, cached input, and output usage under its pricing rules. Do not compare an all-inclusive API charge with only the GPU portion of a self-hosted bill.

​
A worked example: faster tokens, higher cost

Assume two configurations process the same 10,000 logical requests in one hour. The numbers below are hypothetical arithmetic, not Tensorfuse measurements or provider prices. Quality and latency requirements are identical. A is slightly cheaper per accepted request, but it fails one in five requests under the stated requirements. If the product requires at least 95% acceptance, A is not an eligible configuration. Choose among configurations that satisfy the product requirement first, then compare their costs. Now suppose B costs 13forthesameacceptedworkload.Itscostbecomesapproximately13 for the same accepted workload. Its cost becomes approximately 0.00133 per accepted request. The conclusion changes because the measured numerator changes. A token-throughput headline cannot settle either comparison.

​
Keep token metrics useful

Tokens per second still helps diagnose engine capacity. Cost per million output tokens can also be useful within a fixed workload. Report input lengths, generated output lengths, concurrency, model precision, and cache state alongside it. A system generating unnecessarily long answers can look efficient per token while being expensive per completed task. Separate warm steady-state traffic from startup traffic. Then combine them using the observed production mix. A short benchmark that excludes startup cannot establish the economics of an endpoint that repeatedly returns from zero. Use the inference benchmarking guide to collect latency, throughput, and failures. Compare vLLM and SGLang with the same acceptance rubric. If idle GPU hours dominate the numerator, work through the scale-to-zero calculation before changing engines.

​
Put this into practice

Apply the calculation with the step-by-step cost-reduction workflow, then test vLLM quantization or a compatible alternate-silicon runtime.