Local AI Inference Speed: What to Expect (September 2026)
Real tokens per second from local LLMs on consumer hardware: how quant and context change speed, why memory bandwidth is the limit, and what helps.
Tag
#gguf
How to Safely Download and Verify Open-Source LLMs
Sourced steps to safely download and verify an open-source LLM: safetensors, GPG-signed commits, pinned revisions, and HF / Ollama integrity checks.
What Quantization Costs You in Quality (August 2026)
How much quality a smaller quant actually loses. Real MMLU and KL Divergence numbers across Q2_K through Q8_0, plus how to measure on your own workload.
How much VRAM does a local LLM actually need? (September 2026)
The published GGUF file size is the weights. VRAM use also includes the KV cache, framework overhead, and your context length. The math, walked through.
What It Takes to Run Qwen3.8-27B on Your Own Hardware
Five stock GGUF quants for Qwen3.8-27B span 9.01 GB to 29.05 GB. A file size is not a VRAM requirement, and a 262,144-token ceiling is not free to fill.
What Quantization Actually Costs You in Quality
8-bit measures close to lossless. 4-bit ranges from a 0.6% gain to a 59% drop on one benchmark, depending on the model. The figures, fully attributed.
GGUF vs AWQ vs GPTQ vs MLX: Which Quantization to Use
Quantization formats compared with real file sizes and bits per weight. Why Q4 does not halve a model, and why low quants generate faster.
MiniMax-M3 Lands in llama.cpp: Sparse Attention and Vision
Two back-to-back merges add MiniMax Sparse Attention and a Qwen2.5-VL style vision tower to llama.cpp, but every existing MiniMax-M3 GGUF must be regenerated.