How to serve LLMs on AWS Inferentia and Trainium with vLLM

By Samagra Sharma · Reviewed September 6, 2026 An inference service can keep its OpenAI-compatible API when moving to AWS Inferentia or Trainium, but it needs a different execution stack. Start with a supported Neuron container and a small model, verify compilation and generation, then measure the workload that determines your bill.

This upstream EC2 recipe uses vLLM 0.16.0 with Neuron SDK 2.30.0 and NxD Inference on Inf2 or Trn1. It has not been executed on Neuron hardware by Tensorfuse. Check the Tensorfuse configuration reference for documented platform resource options.

Which Neuron serving path should you use?

AWS currently documents two distinct integrations:

Target	Serving path
Inferentia2 / Inf2 and Trainium1 / Trn1	vLLM with NxD Inference; the pinned example below uses this path
Trainium2 / Trn2 and Trainium3 / Trn3	Also evaluate the newer vLLM Neuron plugin, introduced with SDK 2.31

AWS’s container support table lists the instance types for each image. The new plugin setup guide explicitly targets Trn2 and Trn3. Keep its installation and flags separate from this NxD recipe. Inf1 uses a different software path and is outside this example.

1. Prepare the EC2 host

Use an Inf2 or Trn1 instance with Docker, SSH access, internet access for the container and model downloads, and a Neuron driver compatible with SDK 2.30.0. Follow the Neuron 2.30 environment instructions when selecting the Neuron DLAMI. Record its AMI ID and your instance type. On the host, check:

neuron-ls
ls -l /dev/neuron0
docker version

Reserve the first device for this test. The command below exposes only /dev/neuron0 and uses two tensor-parallel ranks. An Inferentia2 device and a Trn1 device each contain two NeuronCores. Verify those cores are available before launching a second model process.

2. Pull the pinned serving image

NEURON_IMAGE='public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.16.0-neuronx-py312-sdk2.30.0-ubuntu24.04'
docker pull "$NEURON_IMAGE"
NEURON_IMAGE=$(docker image inspect "$NEURON_IMAGE" \
  --format '{{index .RepoDigests 0}}')
printf '%s\n' "$NEURON_IMAGE"

Save this digest with your evaluation results. The explicit version follows AWS’s TinyLlama DLC quickstart; it is a reproducible starting point, not a claim that SDK 2.30 is the newest release.

3. Launch TinyLlama

docker run --name neuron-vllm \
  --device=/dev/neuron0 \
  --publish 127.0.0.1:8080:8080 \
  --volume neuron-hf-cache:/root/.cache/huggingface \
  --entrypoint vllm "$NEURON_IMAGE" serve \
  --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --host 0.0.0.0 --port 8080 \
  --tensor-parallel-size 2 \
  --max-model-len 128 \
  --max-num-seqs 4 \
  --block-size 32 \
  --num-gpu-blocks-override 16 \
  --additional-config '{"override_neuron_config":{"enable_bucketing":false}}'

This compact configuration is an installation check. Its 128-token limit includes the formatted input and generated output. The block settings and Neuron override follow the upstream example; benchmark production settings separately once this check works. The process stays in the foreground so compilation and load errors remain visible. In a second terminal, docker logs --follow neuron-vllm shows the same logs. Wait for model initialization and API startup before sending traffic. Docker publishes the endpoint only on the EC2 host’s loopback interface, so run the test there or use an SSH tunnel. Configure authenticated TLS access before making a remote service available to clients.

4. Verify a real completion

curl --fail --silent --show-error http://127.0.0.1:8080/health
curl --fail --silent --show-error http://127.0.0.1:8080/v1/models
curl --fail --silent --show-error http://127.0.0.1:8080/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{"role": "user", "content": "Name the capital of France."}],
    "temperature": 0,
    "max_tokens": 32
  }'

Expect a successful HTTP response with a nonempty assistant message and the TinyLlama model ID. This verifies the serving path, not your application’s accuracy. Move to a supported application model and your own correctness tests before drawing a hardware conclusion.

What usually needs fixing?

Symptom	Check
`/dev/neuron0` is missing	Instance family, host driver installation, and `neuron-ls`
Insufficient NeuronCores	Device exposure and other processes already using the two cores
Unrecognized Neuron argument	Image digest and whether the recipe belongs to NxD Inference or the newer plugin
Context-length rejection	Prompt, chat-template tokens, and output budget must fit within 128 tokens
First startup appears slow	Compilation progress and the first actual error in the logs

Separate first compilation, subsequent model loading, and steady-state requests in your measurements. AWS documents Neuron’s compilation cache; verify which artifacts your serving integration reuses before assuming a fresh replica starts warm.

How do you compare Neuron with GPU inference?

Use the same benchmark workload, model quality requirement, and latency limits on both endpoints. Include the complete EC2 instance, startup and idle time, storage, and network charges. Exposing one device from a larger instance still leaves the entire instance bill in the numerator. Follow the cost reduction workflow and the Inferentia versus Trainium explainer before selecting production capacity. Stop the evaluation with docker stop neuron-vllm, remove it with docker rm neuron-vllm when its logs and artifacts are no longer needed, and release unused EC2 capacity. Container cleanup alone does not end instance charges.

Get Started

LLM Inference

Inference How-to Guides

Inference Hardware

Concepts

Operations

Troubleshooting

Enterprise Setup

Architecture

How to serve LLMs on AWS Inferentia and Trainium with vLLM

Which Neuron serving path should you use?

1. Prepare the EC2 host

2. Pull the pinned serving image

3. Launch TinyLlama

4. Verify a real completion

What usually needs fixing?

How do you compare Neuron with GPU inference?

​Which Neuron serving path should you use?

​1. Prepare the EC2 host

​2. Pull the pinned serving image

​3. Launch TinyLlama

​4. Verify a real completion

​What usually needs fixing?

​How do you compare Neuron with GPU inference?

Which Neuron serving path should you use?

1. Prepare the EC2 host

2. Pull the pinned serving image

3. Launch TinyLlama

4. Verify a real completion

What usually needs fixing?

How do you compare Neuron with GPU inference?