Run large language models locally on a Mac Studio M3 Ultra (96GB) and connect coding agents over LAN.

Mistral Small 4 note: On Apple Silicon, MLX support is still incomplete. Prefer GGUF on llama.cpp / LM Studio / Ollama for local use, or vLLM for Mistral's official full-feature self-deployment path.

MacBook / Linux / WSL  ──── LAN ────>  Mac Studio M3 Ultra (96GB)
  Claude Code                            vllm-mlx (primary) :8000
  OpenCode                               mlx-openai-server :8000
  OpenClaw                               oMLX (multi-model) :8000
  Pi                                     vmlx (JANGTQ) :8000
                                         OpenAI + Anthropic API (+ Ollama for vmlx)

This repository is primarily an operations notebook + config bundle for the Mac Studio inference stack, not a single application codebase.

1. Read this README.md for the stack overview and server selection.
2. Read one server runbook in docs/server/<server>/summary.md.
3. Read configs/README.md for the matching client config templates.
4. Read docs/models/model-summary.md when choosing or adding models.
5. Read the relevant maintenance or patch docs only when upgrading or debugging.

• summary.md files are the main operational entry points.
• maintenance.md and jang-patch.md files are task-specific follow-ups.
• plans/ captures ideas, experiments, and pending investigations; it is not the live runbook layer.

Pick one — all serve on port 8000. Stop others first if switching.

# vllm-mlx — fastest, single model
nohup ~/vllm-mlx-env/bin/python ~/run_vllm_jang.py serve \
  ~/.omlx/models/JANGQ-AI--Qwen3.5-122B-A10B-JANG_2S \
  --served-model-name JANGQ-AI/Qwen3.5-122B-A10B-JANG_2S \
  --port 8000 --host 0.0.0.0 > /tmp/vllm-mlx.log 2>&1 &

# mlx-openai-server — multi-model, low overhead
JANG_PATCH_ENABLED=1 nohup ~/mlx-openai-server-env/bin/mlx-openai-server launch \
  --config ~/mlx-openai-server-multimodel.yaml --no-log-file \
  > /tmp/mlx-openai-server.log 2>&1 &

# oMLX — 9 models, hot-swap
/opt/homebrew/bin/brew services start omlx

# vmlx — JANGTQ CRACK (MLX Studio bundled Python, headless)
# Tool use + Qwen3 thinking require all three parser flags AND a one-time
# source patch (scripts/patch_vmlx_jangtq_mllm_tools.py). See
# docs/server/vmlx/maintenance.md#tool-use-and-reasoning-mllm-models.
BP=/Applications/vMLX.app/Contents/Resources/bundled-python/python
SNAP=~/.cache/huggingface/hub/models--dealignai--MiniMax-M2.7-JANGTQ-CRACK/snapshots/033d5537f48f2f836ce3dfbe392304a2b30f8536
nohup $BP/bin/python3 -m vmlx_engine.cli serve "$SNAP" \
  --host 0.0.0.0 --port 8000 \
  --enable-auto-tool-choice --tool-call-parser qwen3 --reasoning-parser qwen3 \
  > /tmp/vmlx.log 2>&1 &

pkill -f vllm-mlx                                                                # stop vllm-mlx
pkill -f mlx-openai-server                                                       # stop mlx-openai-server
/opt/homebrew/bin/brew services stop omlx                                        # stop oMLX
pkill -f vmlx_engine                                                             # stop vmlx

curl -s http://<MAC_STUDIO_IP>:8000/v1/models | python3 -m json.tool            # vllm-mlx / mlx-openai-server
curl -s http://<MAC_STUDIO_IP>:8000/v1/models \
  -H "Authorization: Bearer <YOUR_API_KEY>" | python3 -m json.tool               # oMLX (auth required)

open http://<MAC_STUDIO_IP>:8000/admin                                            # oMLX dashboard

tail -f /tmp/vllm-mlx.log                                                       # vllm-mlx logs
tail -f /tmp/mlx-openai-server.log                                              # mlx-openai-server logs
tail -f ~/.omlx/logs/server.log                                                 # oMLX logs
tail -f /tmp/vmlx.log                                                           # vmlx logs

Works on all servers — swap <MODEL_NAME> from /v1/models. Add auth header for oMLX.

curl -s http://<MAC_STUDIO_IP>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<MODEL_NAME>","messages":[{"role":"user","content":"Say hello"}],"max_tokens":50}' \
  | python3 -m json.tool

All servers support JANG mixed-precision models via patches: vllm-mlx · oMLX · mlx-openai-server · mlx-lm

Server maintenance: vllm-mlx · oMLX · mlx-openai-server · vmlx

Current mlx-openai-server roster: mlx-community/Qwen3.6-35B-A3B-6bit (single-model, Qwen3.6-only mode — switched 2026-04-18 for through-server benchmarking).

All models fit in 96GB unified memory.

Full specs and per-model details: Model Summary

Quantization key: JANG = adaptive mixed-precision (jangq.ai), MoE = Mixture of Experts (total/active params), nvfp4 = NVIDIA 4-bit float.

Qwen3-Coder-Next 6-bit (dense 60GB):

Gemma 4 26B-A4B 4-bit (MoE, multimodal — mlx-openai-server 1.7.1, Apr 2026):

Tokens include both reasoning (reasoning_content) and output (content) phases. 512 warm values shown (run 1 cold-start: 59.4 tok/s / 28 tok/s prefill / 18.7s TTFT).

Qwen3.6-35B-A3B 6-bit (Hybrid MoE, multimodal — mlx-openai-server 1.7.1, Apr 2026):

Tokens include both reasoning_content (always-on <think>) and content. Server-validated; standalone-only Apr 17 numbers in standalone benchmarks carry a VLM double-prefill artefact and are not directly comparable.

Hybrid Gated DeltaNet pays off at long context — Qwen3.6's 35.6 tok/s @ 128K is 31% faster than Gemma 4 at the same context, despite Qwen3.6 carrying a vision encoder.

Qwen3.5-35B-A3B JANG (MoE, primary architecture):

128K cross-model (long-context comparison — all measured through-server, Apr 2026):

Full results: Standalone · API Server · TurboQuant KV Cache

Server constraints:

• oMLX — No GGUF, no MXFP8, starlette 1.0 dashboard bug (#361). JANG+Nemotron-H matmul mismatch (details). Maintenance
• mlx-openai-server — No Anthropic API, single-request queue, 15% overhead at 64K context, tool arg string bug (patch). Maintenance
• vllm-mlx — Single model only, no dashboard, manual start, v0.2.6 return bug needs patch. Maintenance
• vmlx — JANGTQ only (MLX Studio DMG bundled Python), no GUI but overwritten on every DMG upgrade. MLLM path drops tools[], ignores tools= in chat template, and crashes on multi-turn tool replay — fix with scripts/patch_vmlx_jangtq_mllm_tools.py (detail). Requires --enable-auto-tool-choice --tool-call-parser qwen3 --reasoning-parser qwen3. Maintenance

Model compatibility:

• Nemotron family — Only works on vllm-mlx (chat template not packaged in MLX weights). Details
• Mistral Small 4 — Broken on current MLX servers here (missing native mistral4 MLA support in upstream mlx-lm). For Apple Silicon, the practical local path is GGUF on llama.cpp / LM Studio / Ollama; Mistral's official full-feature deployment guidance still points to vLLM. Details
• Qwen3.5-122B + OpenClaw — HTTP 500 with large system prompts (#42)

Maintenance:

• JANG fork overlay — brew upgrade omlx overwrites the fork; re-apply after every upgrade. Guide
• Hot cache patch — scripts/patch_omlx_cache.py must re-run after every brew upgrade omlx. Guide
• SSH timeouts — Fix: sudo pmset -a sleep 0 disksleep 0 displaysleep 10 on Mac Studio