How to run a local LLM on Apple silicon with MLX

By Samagra Sharma · Reviewed September 6, 2026 For an occasional extraction job or personal assistant, renting a GPU can cost more than using a Mac you already own. Start with a small quantized model in MLX LM, expose a local chat endpoint, and test your actual requests. This guide runs Qwen2.5-1.5B-Instruct in 4-bit MLX format on an Apple silicon GPU using unified memory. The small model is a setup baseline. Use your task’s quality results to decide whether it is sufficient before comparing costs.

Reviewed September 6, 2026 against the linked release, package metadata, and model card. These commands have not been executed on Apple silicon for this guide; no hardware performance result is claimed. This is a local MLX workflow, separate from Tensorfuse’s cloud deployment configuration.

1. Check your Mac and install MLX LM

Use an Apple silicon Mac with macOS 14 or later and native ARM Python 3.11. The MLX installation requirements explain why an Intel Python running through Rosetta does not work for this installation.

python3.11 -c 'import platform; print(platform.machine())'
sw_vers -productVersion
python3.11 -m venv .venv-mlx
source .venv-mlx/bin/activate
python -m pip install 'mlx==0.31.2' 'mlx-lm==0.31.3'
python -m pip check
python -c 'import mlx.core as mx; print(mx.metal.is_available())'

Expect arm64 from the architecture check and True for Metal availability. The guide pins MLX LM 0.31.3 and its minimum MLX version; save the resolved environment with python -m pip freeze > requirements-mlx.txt after installation.

2. Download the quantized model

python - <<'PY'
from pathlib import Path
from huggingface_hub import snapshot_download
from huggingface_hub.constants import HF_HUB_CACHE

# This server version scans the Hub cache when listing models.
Path(HF_HUB_CACHE).mkdir(parents=True, exist_ok=True)
snapshot_download(
    repo_id="mlx-community/Qwen2.5-1.5B-Instruct-4bit",
    revision="8b403126fc14f14cfc99bb4cfa72ecbc129ea677",
    local_dir="models/qwen2.5-1.5b-mlx",
)
PY

The MLX community model card identifies the original Qwen model and quantization. This is a community conversion; evaluate it independently for your application. The download is approximately 869 MB, while runtime memory also includes the KV cache, temporary buffers, macOS, and other applications. The Hub download API supports a commit revision so a later model update does not silently change your baseline. Keep enough free disk space for both the environment and model files.

3. Start a local OpenAI-compatible endpoint

mlx_lm.server \
  --model ./models/qwen2.5-1.5b-mlx \
  --host 127.0.0.1 \
  --port 8080

Leave this terminal running. In another terminal, check the server and send a short request:

curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8080/v1/models

curl --fail --show-error http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default_model",
    "messages": [{
      "role": "user",
      "content": "Extract the order ID. Reply with only the ID: Order TF-1042 ships tomorrow."
    }],
    "temperature": 0,
    "max_tokens": 32
  }'

In this pinned MLX server implementation, default_model resolves to the model passed through --model. The models endpoint lists its absolute local path, which is also a valid model value. Expect a JSON response containing choices[0].message.content; check whether the content is the correct ID. A healthy HTTP server alone does not establish useful model output. Keep this development endpoint on loopback. An OpenAI-compatible request format does not add authentication, TLS, or all OpenAI API features. Stop it with Ctrl+C when finished.

4. Measure whether local inference is cheaper

Run a representative set of short and long inputs, first with one request at a time. Record correctness, output length, full response time, and memory pressure. Separate the first request after launch from subsequent requests. If the application needs streaming, measure time to the first generated token separately from total completion time. Use this cost boundary for the same measurement window:

local cost per accepted request =
  (allocated hardware cost + measured electricity + operating effort)
  / requests that pass quality and latency requirements

An already-owned Mac can have a low incremental cost, but include hardware allocation when comparing a new Mac purchase with a hosted service. Count retries and fallback API calls. Hold the task, acceptance criteria, and request set constant using the inference benchmarking guide.

What if it is slow or fails?

Symptom	Next check
No matching MLX distribution	Check macOS, Python version, and native ARM architecture, then recreate the environment.
Metal availability is false	Verify the Apple silicon environment before treating the result as a Mac GPU baseline.
Memory pressure rises during long requests	Close competing applications, shorten input, and test a smaller context workload before increasing concurrency.
Output is incorrect or truncated	Inspect the prompt and token limit, then compare a more capable model or higher precision against the same labeled inputs.
An application cannot select the model	Use `default_model` or the absolute model path returned by `/v1/models`.

For a comparison with CPU execution, follow the llama.cpp CPU guide. The CPU and Apple silicon explainer covers the memory tradeoffs; reducing inference cost provides the next decision when local capacity falls short.

Get Started

LLM Inference

Inference How-to Guides

Inference Hardware

Concepts

Operations

Troubleshooting

Enterprise Setup

Architecture

How to run a local LLM on Apple silicon with MLX

1. Check your Mac and install MLX LM

2. Download the quantized model

3. Start a local OpenAI-compatible endpoint

4. Measure whether local inference is cheaper

What if it is slow or fails?

​1. Check your Mac and install MLX LM

​2. Download the quantized model

​3. Start a local OpenAI-compatible endpoint

​4. Measure whether local inference is cheaper

​What if it is slow or fails?

1. Check your Mac and install MLX LM

2. Download the quantized model

3. Start a local OpenAI-compatible endpoint

4. Measure whether local inference is cheaper

What if it is slow or fails?