llama.cpp
C/C++ runtime that made local GGUF models practical.
Fit.
Catalog snapshot
- Platforms
- Linux, Macos, Windows
- Operating systems
- Linux, macOS, Windows
- Ram floor
- 8 GB
- Gpu
- Optional · 6 GB VRAM helps
- Docker
- No
- Setup
- Intermediate
- Deploy as
- Local
- Commercial use
- Allowed
CPU inference works. It is slow on 7B+ without a GPU.
Project.
Fetched 1
About.
llama.cpp is the engine under a lot of local AI. It is a CLI and a library, not a product UI. If you need to squeeze a 7B model onto 8 GB, this is the stack to learn.
GGUF · CPU and GPU backends · server example · quantization
