> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to run LLM inference on a CPU with llama.cpp

> Build a CPU-only llama.cpp server, download a small Qwen GGUF model, call its local chat API, and compare useful throughput and cost with GPU inference.

By [Samagra Sharma](/docs/authors/samagra-sharma) · Reviewed September 6, 2026

A small extraction or classification workload may fit on spare CPU capacity. To find out, build llama.cpp with GPU backends disabled, load a quantized model, and measure how many correct requests finish within your deadline. This guide serves Qwen2.5-1.5B-Instruct Q4\_K\_M through a local OpenAI-compatible API on Linux or macOS.

The model is a small starting point for validating the serving path. Its ability to answer a smoke test does not establish quality for a production workload.

<Note>
  Reviewed September 6, 2026 against the pinned llama.cpp source and official Qwen model files. This guide has not been benchmarked on CPU hardware. It describes a standalone llama.cpp setup and does not imply additional Tensorfuse hardware support.
</Note>

## 1. Build llama.cpp for CPU execution

Install Git, curl, CMake, and a C/C++ compiler. On Ubuntu, install `build-essential`, `cmake`, `git`, and `curl` with your package manager. On macOS, install the Xcode Command Line Tools and CMake. Then create a fresh checkout:

```bash theme={null}
git clone --depth 1 --branch v0.4.0 \
  https://github.com/ggml-org/llama.cpp.git llama-cpp-cpu
cd llama-cpp-cpu

cmake -S . -B build-cpu \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=OFF \
  -DGGML_CUDA=OFF \
  -DGGML_HIP=OFF \
  -DGGML_VULKAN=OFF \
  -DGGML_SYCL=OFF \
  -DLLAMA_OPENSSL=OFF
cmake --build build-cpu --config Release --target llama-server -j 4
./build-cpu/bin/llama-server --version
```

This pins the [v0.4.0 release](https://github.com/ggml-org/llama.cpp/releases/tag/v0.4.0). A fresh build directory matters because CMake remembers options from previous builds. Metal defaults on for supported Macs; disabling it makes this a CPU baseline. Other accelerator backends remain off in this fresh default configuration. See the [upstream build instructions](https://github.com/ggml-org/llama.cpp/blob/v0.4.0/docs/build.md) and [backend options](https://github.com/ggml-org/llama.cpp/blob/v0.4.0/ggml/CMakeLists.txt).

OpenSSL support inside llama.cpp is disabled here because curl downloads the model separately and the server listens locally over HTTP.

## 2. Download one GGUF file

Run from the checkout directory:

```bash theme={null}
mkdir -p models
curl --fail --location --retry 3 \
  'https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/91cad51170dc346986eccefdc2dd33a9da36ead9/qwen2.5-1.5b-instruct-q4_k_m.gguf' \
  --output models/qwen2.5-1.5b-instruct-q4_k_m.gguf
```

This is the approximately 1.12 GB Q4\_K\_M file in [Qwen's official GGUF repository](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF). The commit in the URL fixes the model revision. Quantization reduces weight storage; total RAM usage also includes the KV cache, compute buffers, the operating system, and other processes. Leave several GB of available memory for this initial short-context experiment and check actual usage after loading.

## 3. Start the server and make a request

```bash theme={null}
./build-cpu/bin/llama-server \
  --model models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --alias qwen-cpu \
  --n-gpu-layers 0 \
  --ctx-size 2048 \
  --parallel 1 \
  --threads 4 \
  --host 127.0.0.1 \
  --port 8080
```

Use four threads as an initial setting on a machine with at least four available CPU cores; lower it on smaller hosts. The 2,048-token context must accommodate the formatted input and generated output. One server slot makes the initial latency measurement easier to interpret.

In a second terminal:

```bash theme={null}
curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8080/v1/models

curl --fail --show-error http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-cpu",
    "messages": [{
      "role": "user",
      "content": "Classify this message as billing or technical. Reply with one label: I was charged twice."
    }],
    "temperature": 0,
    "max_tokens": 16
  }'
```

Expect a successful health response after loading, `qwen-cpu` in the models response, and text inside `choices[0].message.content`. Check the label itself for correctness. The alias and endpoints follow the [pinned server API](https://github.com/ggml-org/llama.cpp/blob/v0.4.0/tools/server/README.md). Keep the unauthenticated example on loopback and stop it with Ctrl+C when finished.

## 4. Find the cost and latency boundary

Replay representative requests, including longer inputs and failure cases. Compare warm and first-request latency, accepted requests per minute, and peak memory. Increase thread count one setting at a time while keeping the model, prompts, and output limits fixed. More threads can lose performance through memory contention or competition with other services.

Compare CPU and GPU using the same task deadline and quality threshold:

```text theme={null}
cost per accepted request =
  total serving cost during the test window
  / requests meeting both quality and latency requirements
```

Include the full VM charge for a dedicated cloud CPU instance. For shared capacity, document its cost allocation and the impact on neighboring workloads. For owned hardware, include electricity and hardware allocation. Count retries and fallback requests. A lower hourly price can still produce a higher cost per useful result when throughput is too low.

## What if the CPU run fails?

| Symptom                                    | Next check                                                                       |
| ------------------------------------------ | -------------------------------------------------------------------------------- |
| Build cannot find a compiler               | Install the platform's compiler tools and rerun configuration.                   |
| Server cannot load the model               | Check the curl exit status, file size, and exact file path.                      |
| Requests exceed context                    | Shorten the input or raise `--ctx-size` after checking available RAM.            |
| Added threads make requests slower         | Reduce threads and check competing load, memory bandwidth, and socket placement. |
| The small model misses your quality target | Compare a larger model or higher precision before accepting a cost result.       |

Use the [benchmarking guide](/docs/guides/inference/benchmarking) for a controlled comparison. On a Mac, test [MLX on the Apple GPU](/docs/guides/inference/how-to/apple-silicon-mlx) separately. Read the [CPU and Apple silicon explainer](/docs/guides/inference/hardware/cpu-apple-silicon) and [inference cost checklist](/docs/guides/inference/how-to/reduce-cost) before selecting the next backend.