> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run serverless LLM inference on AWS

> Plan self-hosted LLM inference on AWS: choose a model and serving engine, benchmark latency and cost, and configure GPU autoscaling with Tensorfuse.

An LLM that responds quickly to one request can slow down when traffic arrives in bursts. To run serverless inference on AWS, choose a model and serving engine, measure their behavior under load, then configure GPU replicas around your latency target. Tensorfuse deploys these containerized services inside your AWS account.

## Start with the workload

Write down the input length, expected output length, arrival rate, and acceptable response time. A coding agent with repeated tool definitions, an interactive chat application, and an overnight extraction job put different demands on the same model.

| Workload                         | Measure first                                        | Next step                                                               |
| -------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------- |
| Interactive chat                 | Time to first token and gaps between streamed tokens | [Benchmark inference](/docs/guides/inference/benchmarking)                   |
| Agents with repeated context     | Prefix reuse and latency across turns                | [Understand prefix caching](/docs/guides/inference/prefix-caching)           |
| Long generated answers           | Decode latency at target concurrency                 | [Evaluate speculative decoding](/docs/guides/inference/speculative-decoding) |
| Offline extraction or evaluation | Completed jobs, failures, and total cost             | [Use job queues](/docs/concepts/job_queues)                                  |

## Choose a model and engine together

Check the exact checkpoint, quantization, GPU architecture, chat template, and required tool-calling or multimodal support. A model appearing in a supported-model list does not establish its memory requirements or performance for your workload.

Use the [open-model evaluation guide](/docs/guides/inference/open-models) for recent upstream changes, then compare [vLLM and SGLang](/docs/guides/inference/vllm-vs-sglang). Start with one reproducible configuration and change one variable at a time.

## Evaluate inference hardware

If GPU availability, memory, or cost limits the workload, compare [NVIDIA alternatives for inference](/docs/guides/inference/hardware/overview). The explainers cover AMD GPUs, AWS Neuron, Google TPUs, specialized accelerators, and local execution. Match hardware to your model and serving software before comparing performance.

## Deploy a working endpoint

Follow [Getting started](/docs/concepts/getting_started_tensorkube) to configure Tensorfuse. The [GPT-OSS deployment guide](/docs/guides/modality/text/openai_oss) connects a model server to a deployment configuration, authentication, and a readiness check.

Keep three settings aligned: the port the server listens on, the deployment's `port`, and the readiness probe's port. Store credentials in [secrets](/docs/concepts/secrets), and configure [TLS](/docs/concepts/custom_domains_with_tls) for clients that access the service over a network.

## Set autoscaling from measured capacity

In the [deployment configuration](/docs/concepts/configuration), `min_scale` controls the minimum replica count and `max_scale` bounds scale-out. Setting `min_scale: 0` permits scale-to-zero; a new request can then wait for startup. Keeping a warm replica trades ongoing resource cost for faster availability.

The deployment's `concurrency` setting controls the autoscaler's view of request capacity. Engine settings such as vLLM's token budget control scheduling inside each replica. Measure both together: increasing the replica's request allowance can increase queueing before another replica is added.

For cold starts, separate node provisioning, image loading, model loading, and engine initialization. Use the existing [vLLM cold-start case study](/docs/blogs/reducing_gpu_cold_start) to investigate each stage, and rerun measurements with your own image and GPU.

## Related guides

<CardGroup cols={2}>
  <Card title="Benchmark latency and cost" href="/docs/guides/inference/benchmarking">Find the request rate that meets your application's latency target.</Card>
  <Card title="Configure your deployment" href="/docs/concepts/configuration">Set GPU resources, ports, secrets, and replica limits.</Card>
</CardGroup>

## Inference economics

Read [cost per successful request](/docs/blogs/inference-cost-per-request) for a worked cost comparison and [GPU scale-to-zero economics](/docs/blogs/scale-to-zero-gpu-inference) for the idle-versus-restart calculation.

## Put this into practice

Follow the [cost-reduction workflow](/docs/guides/inference/how-to/reduce-cost) for an ordered set of experiments, with commands for quantization, caching, and alternate-silicon serving.