How to run vLLM on AMD GPUs with ROCm

By Samagra Sharma · Reviewed September 6, 2026 An AMD GPU quote can look cheaper than your current NVIDIA instance, but the useful comparison starts with a working model endpoint. Run a small Qwen model in a ROCm-compatible vLLM container, confirm that inference reaches the GPU, then repeat your real workload at the same quality and latency target.

This is an upstream Linux host recipe. The commands were checked against the linked documentation but have not been executed on AMD hardware by Tensorfuse. Tensorfuse’s documented resource options are in the configuration reference.

1. Check the host and choose a compatible image

Use a Linux host with Docker and an AMD GPU supported by your selected vLLM release. This example targets an Instinct GPU such as MI300X; Radeon and Ryzen installations have separate hardware and OS requirements. Match the host driver, OS, GPU architecture, and container runtime using AMD’s compatibility matrix. Check the host before downloading weights:

rocminfo
ls -l /dev/kfd /dev/dri
docker version

rocminfo must identify the GPU, and the device files must exist. A container includes userspace libraries but still needs the host’s working AMD driver. Choose a release tag from the official ROCm image tags that matches the vLLM installation requirements. Enter that tag below in Bash. The commands resolve it to a digest so later runs use the same image:

read -r -p 'Compatible vLLM ROCm release tag: ' ROCM_TAG
test -n "$ROCM_TAG" || exit 1
ROCM_IMAGE="vllm/vllm-openai-rocm:$ROCM_TAG"
docker pull "$ROCM_IMAGE"
ROCM_IMAGE=$(docker image inspect "$ROCM_IMAGE" \
  --format '{{index .RepoDigests 0}}')
printf '%s\n' "$ROCM_IMAGE"

Save the digest with the driver version and GPU model. Keep this shell open for the next commands.

2. Verify PyTorch can access the AMD GPU

docker run --rm \
  --device=/dev/kfd --device=/dev/dri --group-add=video \
  --entrypoint python "$ROCM_IMAGE" -c \
  'import torch; print("HIP:", torch.version.hip); assert torch.version.hip and torch.cuda.is_available(); print(torch.cuda.get_device_name(0))'

Expect a HIP version and an AMD device name. PyTorch intentionally uses torch.cuda APIs for its HIP backend, so that namespace does not imply NVIDIA execution. See PyTorch’s HIP semantics. If this fails, fix driver compatibility or device access before changing model settings. Installing a different model cannot repair a missing GPU runtime.

3. Start a local chat endpoint

docker run --name amd-vllm --rm \
  --device=/dev/kfd --device=/dev/dri --group-add=video \
  --ipc=host \
  --publish 127.0.0.1:8000:8000 \
  --volume amd-hf-cache:/root/.cache/huggingface \
  "$ROCM_IMAGE" \
  --model Qwen/Qwen3-0.6B \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --max-num-seqs 8

The upstream Docker guide uses Qwen3-0.6B as its ROCm serving example. This small model checks installation and API behavior; choose your application model after the check passes. The server listens on the container interface while Docker publishes it only on the host’s loopback address. Run the following requests on that host, or connect through an SSH tunnel. Add authentication and TLS through your serving gateway before exposing an endpoint beyond the evaluation host. The 4,096-token context limit covers input and output together. Eight sequences is an initial scheduling limit. Neither is a recommendation for maximum throughput. Read the engine arguments before increasing memory or concurrency settings.

4. Check readiness and generate an answer

In another terminal on the same host, wait for the startup logs to show the server is ready, then run:

curl --fail --silent --show-error http://127.0.0.1:8000/health
curl --fail --silent --show-error http://127.0.0.1:8000/v1/models
curl --fail --silent --show-error http://127.0.0.1:8000/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Name the capital of France."}],
    "max_tokens": 64,
    "temperature": 0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Confirm a successful HTTP response, the expected model ID, and a nonempty choices[0].message.content. The request disables Qwen’s thinking mode for this short check. A health check alone does not establish that generation works.

What should you check when startup fails?

Symptom	First check
No HIP device in the preflight	Host driver, device permissions, and the selected image’s ROCm requirements
Missing kernel or unsupported GPU architecture	Exact GPU architecture and image support; use a matching upstream build
Out of memory	Other processes, model weight size, context length, and concurrency
Request exceeds context	Tokenized prompt plus output budget, including the chat template
Connection refused	Startup completion and the loopback port mapping

Keep the working image intact while testing a change. ROCm-specific attention and quantization paths can depend on both the GPU generation and the vLLM build; see the AMD architecture explainer before moving to a larger or quantized model.

How do you tell whether AMD inference is cheaper?

Repeat the benchmarking procedure with your actual model, request lengths, quality checks, and latency requirements. Compare total billed host cost per accepted request, including idle time. A one-GPU process on an eight-GPU host can still incur the entire host bill. Use the cost reduction guide to decide which constraint to optimize next. Finish the evaluation with docker stop amd-vllm, then release rented capacity you no longer need; stopping a container does not stop cloud billing.

Get Started

LLM Inference

Inference How-to Guides

Inference Hardware

Concepts

Operations

Troubleshooting

Enterprise Setup

Architecture

How to run vLLM on AMD GPUs with ROCm

1. Check the host and choose a compatible image

2. Verify PyTorch can access the AMD GPU

3. Start a local chat endpoint

4. Check readiness and generate an answer

What should you check when startup fails?

How do you tell whether AMD inference is cheaper?

​1. Check the host and choose a compatible image

​2. Verify PyTorch can access the AMD GPU

​3. Start a local chat endpoint

​4. Check readiness and generate an answer

​What should you check when startup fails?

​How do you tell whether AMD inference is cheaper?

1. Check the host and choose a compatible image

2. Verify PyTorch can access the AMD GPU

3. Start a local chat endpoint

4. Check readiness and generate an answer

What should you check when startup fails?

How do you tell whether AMD inference is cheaper?