ForgeLLM

Compile your LLMs, don't interpret them.

ForgeLLM is a Rust-native ahead-of-time (AOT) ML compiler for language models (1M-7B parameters). It compiles GGUF models into optimized, self-contained binaries with native Metal GPU acceleration — no runtime interpreter, no Python dependencies, no dynamic dispatch.

Faster than llama.cpp on Apple Silicon.

Documentation | Crates.io | forgellm.dev | Blog: How we beat llama.cpp

Performance

Benchmarks on Apple M5 Pro, 8-bit quantization, 64-token generation.

Generation Speed (tok/s)

Prefill Speed (tok/s, long prompt)

Prefill uses hardware matrix-multiply via simdgroup_matrix<float, 8, 8>. The large-tile MMA kernel (matmul_q8_mma32, 32×32 tile) hits ~12.1 TFLOPS sustained on Llama-3.2-1B at 1,501 tokens — ~93% of the M5 Pro FP32 peak. For 1B/3B a FP16-tile 4-simdgroup variant (matmul_q8_mma32_h4, 128 threads per TG, each simdgroup owning a 2×2 grid of 8×8 accumulators) doubles FLOP-per-simdgroup-load via A/B tile reuse.

Deploy Size

Binary size is constant across all models. Compare: llama.cpp ~15 MB, MLX ~500 MB Python runtime.

We beat MLX and llama.cpp on generation across all model sizes. On prefill, we lead for short prompts on 135M and catch up at long contexts on 1B (~3x our previous number), using simdgroup_matrix hardware matrix-multiply tiles. MLX's Accelerate BLAS still edges us for mid-length 1B prefill (~325 tokens).

See benchmarks/HISTORY.md and blog/beating-llama-cpp.md for details.

Quick Start

Install

# From crates.io (recommended)
cargo install forgellm-cli

# Or build from source
git clone https://github.com/sauravpanda/forge-llm.git
cd forge-llm && cargo build --release

Metal GPU (Apple Silicon)

# Compile model to Metal binary
forge compile --model model.gguf --output ./my-model --target metal
forge export-weights --model model.gguf --output ./my-model/weights.bin
cp tokenizer.json ./my-model/

# Build and run
cd my-model && cargo build --release
./target/release/my-model weights.bin tokenizer.json "The meaning of life is"

API Server

# Start OpenAI-compatible server
./target/release/my-model weights.bin tokenizer.json --serve --port 8080

# Query it
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}], "stream": true}'

CPU (cross-platform)

# Compile for CPU with NEON SIMD + Rayon parallelism
forge compile --model model.gguf --output ./my-model --target cpu --run

Why ForgeLLM is faster

Every existing LLM inference engine — llama.cpp, vLLM, MLX — loads model weights at runtime and executes a generic inference loop. This is like shipping a Python interpreter when you could ship a compiled binary.

ForgeLLM compiles models into hardware-specific code:

Compilation Targets

Supported Models

Supports GGUF quantization formats: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q2_K through Q8_K. Also supports SafeTensors and LoRA adapter merging at compile time.

Metal GPU Features

The Metal backend generates optimized Apple Silicon compute shaders:

• Hardware matrix-multiply prefill — simdgroup_matrix<float, 8, 8> MMA tiles dequantize Q8_0 into threadgroup memory and run 8×8×8 simdgroup_multiply_accumulate per tile, hitting ~8.7 TFLOPS sustained on 1B
• Simdgroup cooperative matmul — 32-lane SIMD reductions with shared memory vector caching (fast path for single-token decode)
• Native Q8_0/Q4_0 kernels — Dequantize on-the-fly during matmul, halving memory bandwidth
• Fused projections — QKV and gate+up concatenated into single matmul dispatches
• Single compute encoder — Entire forward pass in one encoder, zero transitions
• Double-buffered prefill — GPU overlaps with CPU encoding
• fast:: math — Hardware-accelerated rsqrt/exp in normalization and attention
• OpenAI-compatible API — --serve mode with SSE streaming

CLI Commands

# AOT compile to Metal GPU binary
forge compile --model model.gguf --output ./out --target metal

# AOT compile to CPU binary
forge compile --model model.gguf --output ./out --target cpu --run

# Export weights for compiled binary
forge export-weights --model model.gguf --output ./out/weights.bin

# Run interpreter (no compilation)
forge run --model model.gguf --tokenizer tokenizer.json --prompt "Hello"

# Interactive chat
forge chat --model model.gguf --tokenizer tokenizer.json

# Start API server (interpreter mode)
forge serve --model model.gguf --tokenizer tokenizer.json --port 8080

# Benchmark
forge bench --model model.gguf --tokenizer tokenizer.json --num-tokens 128

# Inspect model
forge info model.gguf

# ONNX export
forge export-onnx --model model.gguf --output model.onnx

# Speculative decoding
forge speculative --draft small.gguf --target-model large.gguf --output ./spec

Architecture

GGUF/SafeTensors → Frontend → IR Graph → Optimizer → Codegen → Binary
                     parse      build      fuse       emit     compile

8 crates: forgellm-frontend, forgellm-optimizer, forgellm-codegen-cpu, forgellm-codegen-wasm, forgellm-codegen-gpu, forgellm-codegen-metal, forgellm-runtime, forgellm-cli

Contributing

cargo test --workspace --exclude forgellm-python  # 258+ tests
cargo clippy --workspace -- -D warnings
cargo fmt --all -- --check

License

MIT