llama.cpp v0.9.0 Adds OCP FP8 Quantization and Flash Attention Improvements

5 November 2026 1 min read

#batch-inference #consumer-gpu #flash-attention #inference-optimization #llama-cpp #quantization #release

llama.cpp v0.9.0 represents a significant advancement in quantization options and inference efficiency. OCP FP8 E4M3 quantization provides another avenue for reducing model memory footprint while maintaining reasonable accuracy, giving operators more flexibility in deploying models on resource-constrained hardware. The new flash_attn_ext_rows operation specifically improves the efficiency of unified KV caches for multi-sequence (batch) inference scenarios.

These improvements matter because they directly address real deployment challenges: fitting larger models into limited VRAM and handling concurrent inference requests efficiently. With support for four additional architectures and audio generation capabilities, this release broadens llama.cpp's applicability across different model families and use cases. For practitioners optimizing deployments on consumer GPUs, mobile devices, or server infrastructure, these additions provide immediate practical benefits.

Read the full article on llama.cpp release.

Source: llama.cpp release · Relevance: 9/10