Skip to main content
If your inference workload already runs on AWS, Inferentia and Trainium provide accelerator options beyond NVIDIA GPUs. Both participate in AWS’s Neuron ecosystem, and Trainium can serve inference as well as train models. Choose by the specific instance generation, supported model path, compilation requirements, and measured serving behavior rather than the product name alone.

​
What is the Neuron software path?

Neuron provides the compiler, runtime, libraries, and framework integrations for AWS’s ML accelerators. The serving framework hands supported model computation to that stack. AWS documents current integrations and the relationship between Trainium, Inferentia, and inference in the Neuron overview. This changes the model-execution path below your application. CUDA extensions and NVIDIA-specific images need replacement or adaptation. A familiar vLLM API does not establish that every vLLM model, optimization, or engine version works on every Neuron device.

​
Inferentia or Trainium?

Do not assume that a guide for one generation transfers to all inf and trn instances. For example, the current vLLM Neuron setup guide carries device applicability and environment-specific installation instructions. Follow those alongside the supported-model documentation for the chosen release.

​
Why does compilation matter for serving?

A compiled execution path prepares device code for a supported model configuration. Different execution shapes or configurations can require additional artifacts, depending on the integration. A deployment that performs well after compilation can still have unacceptable startup behavior if replicas repeatedly rebuild what they need. AWS’s Neuron persistent cache documentation explains reuse of compiled artifacts. This cache serves a different purpose from a transformer’s runtime KV cache: one avoids compilation work; the other avoids recomputing attention state. Test a fresh start, a restart with reusable compilation artifacts, and a request mix spanning the intended lengths and batch sizes. Record which phase incurs compilation, what gets cached, and when the service becomes ready.

​
Build the evaluation around your AWS workload

Start with a supported upstream example, then substitute your real prompts and correctness checks. Measure latency, failures, and cost at the intended request rate. Include the host, storage, startup, idle capacity, and engineering work required to maintain the new path. Being available on AWS does not automatically make an accelerator available through Tensorfuse. These are architectural explainers; consult the Tensorfuse configuration reference for its documented resource types. Use the benchmarking guide to compare an independently configured Neuron endpoint with your existing service.

Understand another compiled accelerator

Explore TPU topology and serving software.

Plan inference on AWS

Connect measured capacity to your deployment requirements.

​
Put this into practice

Follow the Neuron serving how-to for a pinned Inf2/Trn1 recipe and the distinction from the newer Trn2/Trn3 plugin.