Performance Optimization Guide

Maximize llama.cpp inference speed and efficiency.

CPU Optimization

Thread tuning

`bash

Set threads (default: physical cores)

./llama-cli -m model.gguf -t 8

For AMD Ryzen 9 7950X (16 cores, 32 threads)

-t 16 # Best: physical cores

Avoid hyperthreading (slower for matrix ops)

`

BLAS acceleration

`bash

OpenBLAS (faster matrix ops)

make LLAMA_OPENBLAS=1

BLAS gives 2-3× speedup

`

GPU Offloading

Layer offloading

`bash

Offload 35 layers to GPU (hybrid mode)

./llama-cli -m model.gguf -ngl 35

Offload all layers

./llama-cli -m model.gguf -ngl 999

Find optimal value:

Start with -ngl 999

If OOM, reduce by 5 until fits

`

Memory usage

`bash

Check VRAM usage

nvidia-smi dmon

Reduce context if needed

./llama-cli -m model.gguf -c 2048 # 2K context instead of 4K

`

Batch Processing

`bash

Increase batch size for throughput

./llama-cli -m model.gguf -b 512 # Default: 512

Physical batch (GPU)

--ubatch 128 # Process 128 tokens at once

`

Context Management

`bash

Default context (512 tokens)

-c 512

Longer context (slower, more memory)

-c 4096

Very long context (if model supports)

-c 32768

`

Benchmarks

CPU Performance (Llama 2-7B Q4_K_M)

GPU Offloading (RTX 4090)