> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark LLM inference latency, throughput, and cost

> Benchmark a self-hosted LLM endpoint with vLLM: measure TTFT, inter-token latency, throughput, failures, and cost under realistic request rates on AWS.

A tokens-per-second number cannot tell you whether an endpoint will keep up with your users. Benchmark LLM inference at several request rates, recording time to first token, streaming latency, completed requests, and failures. Compare configurations at the same quality and latency targets before translating resource cost into cost per successful request.

## Record a reproducible configuration

Save the engine version and image digest, model and tokenizer revisions, GPU type/count, driver version, precision, context limit, server flags, replica count, and cache state. Keep an application correctness set alongside the load test.

| Metric              | What it measures                                             | What to watch                              |
| ------------------- | ------------------------------------------------------------ | ------------------------------------------ |
| TTFT                | Request start to first generated token                       | Queueing and prefill delay                 |
| Inter-token latency | Time between generated tokens during streaming               | Pauses while reading an answer             |
| End-to-end latency  | Request start to completion                                  | Total wait for the result                  |
| Output throughput   | Generated output tokens per second across requests           | Aggregate capacity at the tested load      |
| Failure rate        | Failed requests divided by attempted requests                | Timeouts, overload, and invalid responses  |
| Goodput             | Work completed within a defined service target per unit time | Capacity your application can actually use |

Time per output token averages decode time within a request; it does not capture every streaming stall. Name the metric and percentile when publishing a result.

## Run a baseline against your endpoint

Use a separate load-generator environment with the vLLM CLI installed. The example follows the [vLLM benchmark CLI](https://docs.vllm.ai/en/stable/cli/bench/serve/). Check `vllm bench serve --help` in your pinned version for available flags.

Set `INFERENCE_URL` to the HTTPS endpoint without `/v1`, `MODEL_ID` to the served model name, and `INFERENCE_API_KEY` to its API key. Set `TOKENIZER_ID` to the corresponding Hugging Face tokenizer repository if the served name is an alias.

```bash theme={null}
: "${INFERENCE_URL:?Set the endpoint URL without /v1}"
: "${MODEL_ID:?Set the served model name}"
: "${TOKENIZER_ID:?Set the model tokenizer repository}"
: "${INFERENCE_API_KEY:?Set the endpoint API key}"

OPENAI_API_KEY="$INFERENCE_API_KEY" vllm bench serve \
  --backend openai-chat \
  --base-url "${INFERENCE_URL%/}" \
  --endpoint /v1/chat/completions \
  --model "$MODEL_ID" \
  --tokenizer "$TOKENIZER_ID" \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 200 \
  --request-rate 1 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --save-result
```

The benchmark client reads `OPENAI_API_KEY` for authentication. Keeping the key out of `--header` also keeps it out of the parsed arguments printed by vLLM 0.28.0; see the [client authentication implementation](https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/benchmarks/lib/endpoint_request_func.py).

These lengths, prompt count, and request rate are illustrative inputs, not a capacity recommendation. Early stopping and model-specific behavior can change actual output lengths; retain the measured token counts. Synthetic prompts also do not establish answer quality or realistic prefix reuse.

## Increase load and replay realistic traffic

First hold the deployment at one warm replica to measure engine capacity. Increase the request rate in steps, running long enough at each level for stable results. Repeat runs and inspect client CPU and network use so that the load generator does not become the bottleneck.

Then enable your intended [autoscaling configuration](/docs/concepts/configuration) and test bursts, scale-out, and a request after scale-to-zero. Include representative prompts, long inputs, long outputs, and [repeated prefixes](/docs/guides/inference/prefix-caching). Keep startup latency and steady-state latency in separate results.

## Convert performance into cost

Choose a measurement window and sum its applicable compute, storage, network, and platform costs. Include idle replicas and failed work in the numerator. For example, a hypothetical $12 window with 6,000 successful requests costs $0.002 per successful request. If only 4,000 meet the latency target, the cost is \$0.003 per qualifying request.

For token reporting, state whether the denominator is input tokens, output tokens, or both. Tokenizers differ, so cost per completed application task is often more useful when comparing models. Use actual billed rates for the AWS region and purchase option you test.

## Related guides

<CardGroup cols={2}>
  <Card title="Compare vLLM and SGLang" href="/docs/guides/inference/vllm-vs-sglang">Use the same workload and resource budget for each engine.</Card>
  <Card title="Investigate startup delays" href="/docs/blogs/reducing_gpu_cold_start">Break down model loading and engine initialization.</Card>
</CardGroup>

## Inference economics

Read [cost per successful request](/docs/blogs/inference-cost-per-request) for a worked cost comparison and [GPU scale-to-zero economics](/docs/blogs/scale-to-zero-gpu-inference) for the idle-versus-restart calculation.