name: llama-cpp
description: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
version: 1.0.0
author: Orchestra Research
license: MIT
dependencies: [llama-cpp-python]
metadata:
hermes:
tags: [Inference Serving, Llama.cpp, CPU Inference, Apple Silicon, Edge Deployment, GGUF, Quantization, Non-NVIDIA, AMD GPUs, Intel GPUs, Embedded]
llama.cpp
Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.
When to use llama.cpp
Use llama.cpp when:
- - Running on CPU-only machines
- - Deploying on Apple Silicon (M1/M2/M3/M4)
- - Using AMD or Intel GPUs (no CUDA)
- - Edge deployment (Raspberry Pi, embedded systems)
- - Need simple deployment without Docker/Python
- - Have NVIDIA GPUs (A100/H100)
- - Need maximum throughput (100K+ tok/s)
- - Running in datacenter with CUDA
- - Have NVIDIA GPUs
- - Need Python-first API
- - Want PagedAttention
- - Llama 2 (7B, 13B, 70B)
- - Llama 3 (8B, 70B, 405B)
- - Code Llama
- - Mistral 7B
- - Mixtral 8x7B, 8x22B
- - Falcon, BLOOM, GPT-J
- - Phi-3, Gemma, Qwen
- - LLaVA (vision), Whisper (audio)
- - Quantization Guide - GGUF formats, conversion, quality comparison
- - Server Deployment - API endpoints, Docker, monitoring
- - Optimization - Performance tuning, hybrid CPU+GPU
- - GitHub: https://github.com/ggerganov/llama.cpp
- - Models: https://huggingface.co/models?library=gguf
- - Discord: https://discord.gg/llama-cpp
Use TensorRT-LLM instead when:
Use vLLM instead when:
Quick start
Installation
`bash
macOS/Linux
brew install llama.cpp
Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
With Metal (Apple Silicon)
make LLAMA_METAL=1
With CUDA (NVIDIA)
make LLAMA_CUDA=1
With ROCm (AMD)
make LLAMA_HIP=1
`
Download model
`bash
Download from HuggingFace (GGUF format)
huggingface-cli download \
TheBloke/Llama-2-7B-Chat-GGUF \
llama-2-7b-chat.Q4_K_M.gguf \
--local-dir models/
Or convert from HuggingFace
python convert_hf_to_gguf.py models/llama-2-7b-chat/
`
Run inference
`bash
Simple chat
./llama-cli \
-m models/llama-2-7b-chat.Q4_K_M.gguf \
-p "Explain quantum computing" \
-n 256 # Max tokens
Interactive chat
./llama-cli \
-m models/llama-2-7b-chat.Q4_K_M.gguf \
--interactive
`
Server mode
`bash
Start OpenAI-compatible server
./llama-server \
-m models/llama-2-7b-chat.Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 32 # Offload 32 layers to GPU
Client request
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-2-7b-chat",
"messages": [{"role": "user", "content": "Hello!"}],
"temperature": 0.7,
"max_tokens": 100
}'
`
Quantization formats
GGUF format overview
| Format | Bits | Size (7B) | Speed | Quality | Use Case |
| -------- | ------ | ----------- | ------- | --------- | ---------- |
| Q4_K_M | 4.5 | 4.1 GB | Fast | Good | Recommended default |
| Q4_K_S | 4.3 | 3.9 GB | Faster | Lower | Speed critical |
| Q5_K_M | 5.5 | 4.8 GB | Medium | Better | Quality critical |
| Q6_K | 6.5 | 5.5 GB | Slower | Best | Maximum quality |
| Q8_0 | 8.0 | 7.0 GB | Slow | Excellent | Minimal degradation |
| Q2_K | 2.5 | 2.7 GB | Fastest | Poor | Testing only |
| CPU | Threads | Speed | Cost | ||
| ----- | --------- | ------- | ------ | ||
| Apple M3 Max | 16 | 50 tok/s | $0 (local) | ||
| AMD Ryzen 9 7950X | 32 | 35 tok/s | $0.50/hour | ||
| Intel i9-13900K | 32 | 30 tok/s | $0.40/hour | ||
| AWS c7i.16xlarge | 64 | 40 tok/s | $2.88/hour | ||
| GPU | Speed | vs CPU | Cost | ||
| ----- | ------- | -------- | ------ | ||
| NVIDIA RTX 4090 | 120 tok/s | 3-4× | $0 (local) | ||
| NVIDIA A10 | 80 tok/s | 2-3× | $1.00/hour | ||
| AMD MI250 | 70 tok/s | 2× | $2.00/hour | ||
| Apple M3 Max (Metal) | 50 tok/s | ~Same | $0 (local) |
Supported models
LLaMA family:
Mistral family:
Other:
Find models: https://huggingface.co/models?library=gguf
References
Resources