> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM vs SGLang: choose an LLM inference engine

> Compare vLLM and SGLang for self-hosted LLM inference using model support, prefix caching, latency, throughput, and operational requirements on AWS.

Switching inference engines can change latency, memory use, and application behavior. Choose between vLLM and SGLang by testing the same checkpoint, GPU allocation, request mix, and latency target. Both offer OpenAI-compatible serving and prefix reuse; the useful comparison is how each performs with the features your application actually uses.

## What should you compare?

| Decision            | vLLM                                       | SGLang                                     | Verify in your application                              |
| ------------------- | ------------------------------------------ | ------------------------------------------ | ------------------------------------------------------- |
| Client integration  | OpenAI-compatible server                   | OpenAI-compatible APIs                     | Streaming, errors, tool calls, and structured responses |
| Repeated prompts    | Automatic prefix caching                   | RadixAttention and prefix caching          | Reuse with your actual prompt ordering and routing      |
| Model deployment    | Model and hardware support vary by release | Model and hardware support vary by release | Exact checkpoint, quantization, and GPU architecture    |
| Multi-GPU execution | Parallel serving configurations            | Parallel serving configurations            | Topology, memory headroom, and per-request latency      |

See the upstream [vLLM documentation](https://docs.vllm.ai/en/stable/) and [SGLang documentation](https://docs.sglang.io/) for supported configurations. Treat an OpenAI-compatible endpoint as an integration starting point: model-specific chat templates and tool parsers still affect behavior.

Hardware backends can have different feature coverage within the same engine. Use the [inference hardware guide](/docs/guides/inference/hardware/overview) to check the execution stack for each accelerator.

## Establish a fair baseline

Record the model revision, container digest, GPU count and type, driver, precision, context limit, and engine arguments. Give both engines the same resource budget and output limits. Comparing one engine with more GPUs or a shorter output budget does not isolate engine performance.

Begin with a small correctness set from your application. Check answer quality, valid JSON where required, tool names and arguments, stop conditions, and streaming completion. Then use the [benchmarking guide](/docs/guides/inference/benchmarking) to sweep request rates.

Run both cold-cache and warm-cache workloads. Random prompts help establish a baseline, while realistic repeated prefixes reveal behavior for agents and conversations. Keep [prefix caching](/docs/guides/inference/prefix-caching) settings explicit in the results.

## How do you choose a winner?

Compare completed requests that satisfy your latency target, alongside failures and cost. An engine can produce more aggregate tokens while individual users wait longer. Report p95 time to first token, p95 inter-token latency, and end-to-end latency at the same offered request rate.

If your current vLLM deployment already meets the target, evaluate SGLang when a required model feature or measured bottleneck provides a reason to switch. If you are starting fresh, benchmark both after confirming that each supports the features you need. There is no workload-independent speed ranking in this guide.

## What changes on Tensorfuse?

The engine runs inside your container. Tensorfuse's [deployment configuration](/docs/concepts/configuration) supplies the infrastructure settings: GPUs, secrets, ports, readiness, and scaling. Changing the engine also requires checking its startup command, authentication, and health endpoint.

The existing [GPT-OSS example](/docs/guides/modality/text/openai_oss) demonstrates the vLLM path. This comparison does not establish a tested SGLang deployment recipe or managed support for distributed caches and prefill/decode disaggregation.

## Related guides

<CardGroup cols={2}>
  <Card title="Benchmark both engines" href="/docs/guides/inference/benchmarking">Compare latency and cost under the same traffic.</Card>
  <Card title="Choose an open model" href="/docs/guides/inference/open-models">Check recent releases against your deployment requirements.</Card>
</CardGroup>