Skip to main content
An LLM that responds quickly to one request can slow down when traffic arrives in bursts. To run serverless inference on AWS, choose a model and serving engine, measure their behavior under load, then configure GPU replicas around your latency target. Tensorfuse deploys these containerized services inside your AWS account.

​
Start with the workload

Write down the input length, expected output length, arrival rate, and acceptable response time. A coding agent with repeated tool definitions, an interactive chat application, and an overnight extraction job put different demands on the same model.

​
Choose a model and engine together

Check the exact checkpoint, quantization, GPU architecture, chat template, and required tool-calling or multimodal support. A model appearing in a supported-model list does not establish its memory requirements or performance for your workload. Use the open-model evaluation guide for recent upstream changes, then compare vLLM and SGLang. Start with one reproducible configuration and change one variable at a time.

​
Evaluate inference hardware

If GPU availability, memory, or cost limits the workload, compare NVIDIA alternatives for inference. The explainers cover AMD GPUs, AWS Neuron, Google TPUs, specialized accelerators, and local execution. Match hardware to your model and serving software before comparing performance.

​
Deploy a working endpoint

Follow Getting started to configure Tensorfuse. The GPT-OSS deployment guide connects a model server to a deployment configuration, authentication, and a readiness check. Keep three settings aligned: the port the server listens on, the deployment’s port, and the readiness probe’s port. Store credentials in secrets, and configure TLS for clients that access the service over a network.

​
Set autoscaling from measured capacity

In the deployment configuration, min_scale controls the minimum replica count and max_scale bounds scale-out. Setting min_scale: 0 permits scale-to-zero; a new request can then wait for startup. Keeping a warm replica trades ongoing resource cost for faster availability. The deployment’s concurrency setting controls the autoscaler’s view of request capacity. Engine settings such as vLLM’s token budget control scheduling inside each replica. Measure both together: increasing the replica’s request allowance can increase queueing before another replica is added. For cold starts, separate node provisioning, image loading, model loading, and engine initialization. Use the existing vLLM cold-start case study to investigate each stage, and rerun measurements with your own image and GPU.

Benchmark latency and cost

Find the request rate that meets your application’s latency target.

Configure your deployment

Set GPU resources, ports, secrets, and replica limits.

​
Inference economics

Read cost per successful request for a worked cost comparison and GPU scale-to-zero economics for the idle-versus-restart calculation.

​
Put this into practice

Follow the cost-reduction workflow for an ordered set of experiments, with commands for quantization, caching, and alternate-silicon serving.