> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS Inferentia vs Trainium for LLM inference

> Understand inference on AWS Inferentia and Trainium with Neuron: model compatibility, compilation, caching, request shapes, and serving tradeoffs.

If your inference workload already runs on AWS, Inferentia and Trainium provide accelerator options beyond NVIDIA GPUs. Both participate in AWS's Neuron ecosystem, and Trainium can serve inference as well as train models. Choose by the specific instance generation, supported model path, compilation requirements, and measured serving behavior rather than the product name alone.

## What is the Neuron software path?

Neuron provides the compiler, runtime, libraries, and framework integrations for AWS's ML accelerators. The serving framework hands supported model computation to that stack. AWS documents current integrations and the relationship between Trainium, Inferentia, and inference in the [Neuron overview](https://aws.amazon.com/ai/machine-learning/neuron/).

This changes the model-execution path below your application. CUDA extensions and NVIDIA-specific images need replacement or adaptation. A familiar vLLM API does not establish that every vLLM model, optimization, or engine version works on every Neuron device.

## Inferentia or Trainium?

| Question                                 | What to check                                                             |
| ---------------------------------------- | ------------------------------------------------------------------------- |
| Which model implementation is available? | The integration's exact checkpoint and architecture support               |
| Which device can run it?                 | Instance generation and Neuron release compatibility                      |
| How is it partitioned?                   | NeuronCore allocation, model parallelism, and communication               |
| Can it handle your traffic?              | Context limits, batch sizes, compilation behavior, and latency under load |

Do not assume that a guide for one generation transfers to all `inf` and `trn` instances. For example, the [current vLLM Neuron setup guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/vllm-neuron/docs/getting-started/setup-guide.html) carries device applicability and environment-specific installation instructions. Follow those alongside the supported-model documentation for the chosen release.

## Why does compilation matter for serving?

A compiled execution path prepares device code for a supported model configuration. Different execution shapes or configurations can require additional artifacts, depending on the integration. A deployment that performs well after compilation can still have unacceptable startup behavior if replicas repeatedly rebuild what they need.

AWS's [Neuron persistent cache documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-features/neuron-caching.html) explains reuse of compiled artifacts. This cache serves a different purpose from a transformer's [runtime KV cache](/docs/guides/inference/prefix-caching): one avoids compilation work; the other avoids recomputing attention state.

Test a fresh start, a restart with reusable compilation artifacts, and a request mix spanning the intended lengths and batch sizes. Record which phase incurs compilation, what gets cached, and when the service becomes ready.

## Build the evaluation around your AWS workload

Start with a supported upstream example, then substitute your real prompts and correctness checks. Measure latency, failures, and cost at the intended request rate. Include the host, storage, startup, idle capacity, and engineering work required to maintain the new path.

Being available on AWS does not automatically make an accelerator available through Tensorfuse. These are architectural explainers; consult the [Tensorfuse configuration reference](/docs/concepts/configuration) for its documented resource types. Use the [benchmarking guide](/docs/guides/inference/benchmarking) to compare an independently configured Neuron endpoint with your existing service.

## Related guides

<CardGroup cols={2}>
  <Card title="Understand another compiled accelerator" href="/docs/guides/inference/hardware/google-tpu">Explore TPU topology and serving software.</Card>
  <Card title="Plan inference on AWS" href="/docs/guides/inference/overview">Connect measured capacity to your deployment requirements.</Card>
</CardGroup>

## Put this into practice

Follow the [Neuron serving how-to](/docs/guides/inference/how-to/aws-neuron-vllm) for a pinned Inf2/Trn1 recipe and the distinction from the newer Trn2/Trn3 plugin.