> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to run vLLM on AMD GPUs with ROCm

> Serve a Qwen model on an AMD GPU using a ROCm vLLM container, verify the API, troubleshoot device errors, and measure inference cost fairly.

By [Samagra Sharma](/docs/authors/samagra-sharma) · Reviewed September 6, 2026

An AMD GPU quote can look cheaper than your current NVIDIA instance, but the useful comparison starts with a working model endpoint. Run a small Qwen model in a ROCm-compatible vLLM container, confirm that inference reaches the GPU, then repeat your real workload at the same quality and latency target.

<Note>
  This is an upstream Linux host recipe. The commands were checked against the linked documentation but have not been executed on AMD hardware by Tensorfuse. Tensorfuse's documented resource options are in the [configuration reference](/docs/concepts/configuration).
</Note>

## 1. Check the host and choose a compatible image

Use a Linux host with Docker and an AMD GPU supported by your selected vLLM release. This example targets an Instinct GPU such as MI300X; Radeon and Ryzen installations have separate hardware and OS requirements. Match the host driver, OS, GPU architecture, and container runtime using [AMD's compatibility matrix](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html).

Check the host before downloading weights:

```bash theme={null}
rocminfo
ls -l /dev/kfd /dev/dri
docker version
```

`rocminfo` must identify the GPU, and the device files must exist. A container includes userspace libraries but still needs the host's working AMD driver.

Choose a release tag from the [official ROCm image tags](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags) that matches the [vLLM installation requirements](https://docs.vllm.ai/en/stable/getting_started/installation/gpu/). Enter that tag below in Bash. The commands resolve it to a digest so later runs use the same image:

```bash theme={null}
read -r -p 'Compatible vLLM ROCm release tag: ' ROCM_TAG
test -n "$ROCM_TAG" || exit 1
ROCM_IMAGE="vllm/vllm-openai-rocm:$ROCM_TAG"
docker pull "$ROCM_IMAGE"
ROCM_IMAGE=$(docker image inspect "$ROCM_IMAGE" \
  --format '{{index .RepoDigests 0}}')
printf '%s\n' "$ROCM_IMAGE"
```

Save the digest with the driver version and GPU model. Keep this shell open for the next commands.

## 2. Verify PyTorch can access the AMD GPU

```bash theme={null}
docker run --rm \
  --device=/dev/kfd --device=/dev/dri --group-add=video \
  --entrypoint python "$ROCM_IMAGE" -c \
  'import torch; print("HIP:", torch.version.hip); assert torch.version.hip and torch.cuda.is_available(); print(torch.cuda.get_device_name(0))'
```

Expect a HIP version and an AMD device name. PyTorch intentionally uses `torch.cuda` APIs for its HIP backend, so that namespace does not imply NVIDIA execution. See [PyTorch's HIP semantics](https://docs.pytorch.org/docs/stable/notes/hip.html).

If this fails, fix driver compatibility or device access before changing model settings. Installing a different model cannot repair a missing GPU runtime.

## 3. Start a local chat endpoint

```bash theme={null}
docker run --name amd-vllm --rm \
  --device=/dev/kfd --device=/dev/dri --group-add=video \
  --ipc=host \
  --publish 127.0.0.1:8000:8000 \
  --volume amd-hf-cache:/root/.cache/huggingface \
  "$ROCM_IMAGE" \
  --model Qwen/Qwen3-0.6B \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --max-num-seqs 8
```

The [upstream Docker guide](https://docs.vllm.ai/en/stable/deployment/docker/) uses Qwen3-0.6B as its ROCm serving example. This small model checks installation and API behavior; choose your application model after the check passes.

The server listens on the container interface while Docker publishes it only on the host's loopback address. Run the following requests on that host, or connect through an SSH tunnel. Add authentication and TLS through your serving gateway before exposing an endpoint beyond the evaluation host.

The 4,096-token context limit covers input and output together. Eight sequences is an initial scheduling limit. Neither is a recommendation for maximum throughput. Read the [engine arguments](https://docs.vllm.ai/en/stable/configuration/engine_args/) before increasing memory or concurrency settings.

## 4. Check readiness and generate an answer

In another terminal on the same host, wait for the startup logs to show the server is ready, then run:

```bash theme={null}
curl --fail --silent --show-error http://127.0.0.1:8000/health
curl --fail --silent --show-error http://127.0.0.1:8000/v1/models
curl --fail --silent --show-error http://127.0.0.1:8000/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Name the capital of France."}],
    "max_tokens": 64,
    "temperature": 0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

Confirm a successful HTTP response, the expected model ID, and a nonempty `choices[0].message.content`. The request disables [Qwen's thinking mode](https://huggingface.co/Qwen/Qwen3-0.6B#switching-between-thinking-and-non-thinking-mode) for this short check. A health check alone does not establish that generation works.

## What should you check when startup fails?

| Symptom                                        | First check                                                                 |
| ---------------------------------------------- | --------------------------------------------------------------------------- |
| No HIP device in the preflight                 | Host driver, device permissions, and the selected image's ROCm requirements |
| Missing kernel or unsupported GPU architecture | Exact GPU architecture and image support; use a matching upstream build     |
| Out of memory                                  | Other processes, model weight size, context length, and concurrency         |
| Request exceeds context                        | Tokenized prompt plus output budget, including the chat template            |
| Connection refused                             | Startup completion and the loopback port mapping                            |

Keep the working image intact while testing a change. ROCm-specific attention and quantization paths can depend on both the GPU generation and the vLLM build; see the [AMD architecture explainer](/docs/guides/inference/hardware/amd-rocm) before moving to a larger or quantized model.

## How do you tell whether AMD inference is cheaper?

Repeat the [benchmarking procedure](/docs/guides/inference/benchmarking) with your actual model, request lengths, quality checks, and latency requirements. Compare total billed host cost per accepted request, including idle time. A one-GPU process on an eight-GPU host can still incur the entire host bill.

Use the [cost reduction guide](/docs/guides/inference/how-to/reduce-cost) to decide which constraint to optimize next. Finish the evaluation with `docker stop amd-vllm`, then release rented capacity you no longer need; stopping a container does not stop cloud billing.