> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to run a local LLM on Apple silicon with MLX

> Serve a 4-bit Qwen model on your Mac with MLX LM, call its local chat API, and measure whether local inference meets your latency and cost targets.

By [Samagra Sharma](/docs/authors/samagra-sharma) · Reviewed September 6, 2026

For an occasional extraction job or personal assistant, renting a GPU can cost more than using a Mac you already own. Start with a small quantized model in MLX LM, expose a local chat endpoint, and test your actual requests. This guide runs Qwen2.5-1.5B-Instruct in 4-bit MLX format on an Apple silicon GPU using unified memory.

The small model is a setup baseline. Use your task's quality results to decide whether it is sufficient before comparing costs.

<Note>
  Reviewed September 6, 2026 against the linked release, package metadata, and model card. These commands have not been executed on Apple silicon for this guide; no hardware performance result is claimed. This is a local MLX workflow, separate from Tensorfuse's cloud deployment configuration.
</Note>

## 1. Check your Mac and install MLX LM

Use an Apple silicon Mac with macOS 14 or later and native ARM Python 3.11. The [MLX installation requirements](https://ml-explore.github.io/mlx/build/html/install.html) explain why an Intel Python running through Rosetta does not work for this installation.

```bash theme={null}
python3.11 -c 'import platform; print(platform.machine())'
sw_vers -productVersion
python3.11 -m venv .venv-mlx
source .venv-mlx/bin/activate
python -m pip install 'mlx==0.31.2' 'mlx-lm==0.31.3'
python -m pip check
python -c 'import mlx.core as mx; print(mx.metal.is_available())'
```

Expect `arm64` from the architecture check and `True` for Metal availability. The guide pins [MLX LM 0.31.3](https://github.com/ml-explore/mlx-lm/releases/tag/v0.31.3) and its minimum MLX version; save the resolved environment with `python -m pip freeze > requirements-mlx.txt` after installation.

## 2. Download the quantized model

```bash theme={null}
python - <<'PY'
from pathlib import Path
from huggingface_hub import snapshot_download
from huggingface_hub.constants import HF_HUB_CACHE

# This server version scans the Hub cache when listing models.
Path(HF_HUB_CACHE).mkdir(parents=True, exist_ok=True)
snapshot_download(
    repo_id="mlx-community/Qwen2.5-1.5B-Instruct-4bit",
    revision="8b403126fc14f14cfc99bb4cfa72ecbc129ea677",
    local_dir="models/qwen2.5-1.5b-mlx",
)
PY
```

The [MLX community model card](https://huggingface.co/mlx-community/Qwen2.5-1.5B-Instruct-4bit) identifies the original Qwen model and quantization. This is a community conversion; evaluate it independently for your application. The download is approximately 869 MB, while runtime memory also includes the KV cache, temporary buffers, macOS, and other applications.

The [Hub download API](https://huggingface.co/docs/huggingface_hub/guides/download) supports a commit revision so a later model update does not silently change your baseline. Keep enough free disk space for both the environment and model files.

## 3. Start a local OpenAI-compatible endpoint

```bash theme={null}
mlx_lm.server \
  --model ./models/qwen2.5-1.5b-mlx \
  --host 127.0.0.1 \
  --port 8080
```

Leave this terminal running. In another terminal, check the server and send a short request:

```bash theme={null}
curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8080/v1/models

curl --fail --show-error http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default_model",
    "messages": [{
      "role": "user",
      "content": "Extract the order ID. Reply with only the ID: Order TF-1042 ships tomorrow."
    }],
    "temperature": 0,
    "max_tokens": 32
  }'
```

In this [pinned MLX server implementation](https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/server.py), `default_model` resolves to the model passed through `--model`. The models endpoint lists its absolute local path, which is also a valid model value. Expect a JSON response containing `choices[0].message.content`; check whether the content is the correct ID. A healthy HTTP server alone does not establish useful model output.

Keep this development endpoint on loopback. An OpenAI-compatible request format does not add authentication, TLS, or all OpenAI API features. Stop it with Ctrl+C when finished.

## 4. Measure whether local inference is cheaper

Run a representative set of short and long inputs, first with one request at a time. Record correctness, output length, full response time, and memory pressure. Separate the first request after launch from subsequent requests. If the application needs streaming, measure time to the first generated token separately from total completion time.

Use this cost boundary for the same measurement window:

```text theme={null}
local cost per accepted request =
  (allocated hardware cost + measured electricity + operating effort)
  / requests that pass quality and latency requirements
```

An already-owned Mac can have a low incremental cost, but include hardware allocation when comparing a new Mac purchase with a hosted service. Count retries and fallback API calls. Hold the task, acceptance criteria, and request set constant using the [inference benchmarking guide](/docs/guides/inference/benchmarking).

## What if it is slow or fails?

| Symptom                                    | Next check                                                                                                                 |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| No matching MLX distribution               | Check macOS, Python version, and native ARM architecture, then recreate the environment.                                   |
| Metal availability is false                | Verify the Apple silicon environment before treating the result as a Mac GPU baseline.                                     |
| Memory pressure rises during long requests | Close competing applications, shorten input, and test a smaller context workload before increasing concurrency.            |
| Output is incorrect or truncated           | Inspect the prompt and token limit, then compare a more capable model or higher precision against the same labeled inputs. |
| An application cannot select the model     | Use `default_model` or the absolute model path returned by `/v1/models`.                                                   |

For a comparison with CPU execution, follow [the llama.cpp CPU guide](/docs/guides/inference/how-to/cpu-llama-cpp). The [CPU and Apple silicon explainer](/docs/guides/inference/hardware/cpu-apple-silicon) covers the memory tradeoffs; [reducing inference cost](/docs/guides/inference/how-to/reduce-cost) provides the next decision when local capacity falls short.