Tagged "kv-cache-optimization"
21 articles tagged kv-cache-optimization, 14 March 2026 to 23 September 2026. Newest first.
-
vLLM Architecture, Memory and Benchmarks Deep Dive
An in-depth technical analysis of vLLM's architecture, memory management, and throughput characteristics, providing concrete benchmarks and optimization strategies for local LLM inference.
-
vLLM v0.30.0 Released With DeepSeek-V4.1 and Advanced Optimizations
vLLM v0.30.0 brings 762 commits including support for DeepSeek-V4.1-Flash with MXFP8 quantization and async prefetch optimizations for improved throughput on local hardware.
-
Running 104GB Qwen3.8-Flash-Next on 48GB Mac with Slotstream at ~12 tok/s
A breakthrough demonstration of running a 104GB model on a 48GB Mac using adaptive KV streaming techniques, achieving practical inference speeds of ~12 tokens/second. This showcases innovative memory optimization for consumer hardware.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM
A specialized llama.cpp implementation adds adaptive KV-cache streaming to run Qwen 3.8 27B with large context windows on 16GB GPUs, demonstrating significant memory optimization advances.
-
llama.cpp Optimizes DFlash Encoder with KV Cache Injection
Recent llama.cpp builds include performance improvements for DFlash models by fusing encoder operations into KV cache injection, reducing computational overhead for local inference.
-
K-EXAONE 2.0 Brings 262K Context to Frontier AI
K-EXAONE 2.0 introduces a 262K token context window, significantly expanding the capabilities of frontier-class models for local deployment and extended reasoning tasks. This represents a major advancement in practical context window management.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
Deep dive into KV cache management and practical strategies to prevent GPU out-of-memory errors when running local LLMs, a critical bottleneck for on-device inference.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
A comprehensive guide addressing one of the most critical bottlenecks in local LLM deployment: KV cache memory consumption. Learn practical strategies to manage GPU memory constraints when running LLMs on-device.
-
TriAttention Solves KV Cache Memory Bottleneck in Local LLM Inference
TriAttention presents a solution to the KV cache memory bottleneck that constrains local LLM inference speed and hardware requirements. This breakthrough addresses one of the most significant performance limitations in on-device language model deployment.
-
CacheWise Optimizes KVCache Reuse for LLM Coding Agents
CacheWise improves inference efficiency by optimizing KVCache reuse in language models used for coding tasks. This memory optimization technique reduces computational overhead and latency for agent-based LLM applications.
-
Prefill Once, Fan Out: KV Snapshot Sharing for Multi-Agent LLM Pipelines
Towards Data Science published research on KV snapshot sharing optimization that enables efficient multi-agent LLM pipelines by reusing computed key-value caches across multiple agents. This technique significantly reduces compute requirements for local deployment scenarios.
-
Apple Unveils AFM 3 Core Advanced with 20 Billion Parameters for On-Device AI
Apple introduced the AFM 3 Core Advanced architecture at WWDC26, featuring a 20 billion parameter model optimized for on-device inference. This represents a significant milestone in local LLM deployment on consumer hardware with architectural innovations to overcome memory constraints.
-
Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing
A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.
-
Gemma 4 Support Stabilized in Llama.cpp
Major fixes for Gemma 4 models have been merged into Llama.cpp, resolving known issues and enabling stable inference. Users report successful deployments of Gemma 4 31B on Q5 quantizations without problems.
-
Gemma 4 GGUF Models Updated with Critical Quantization Fixes
Unsloth has released updated Gemma 4 GGUF quantizations addressing kv-cache issues and other inference problems. New versions are available for both 26B and 31B model sizes.
-
TurboQuant in Llama.cpp Achieves 6X Smaller KV Cache
A new implementation of TurboQuant in llama.cpp reduces KV cache size by 6x, significantly improving memory efficiency for local LLM inference. This breakthrough enables running larger models on resource-constrained devices.
-
Gemma 4 KV Cache Memory Issues Fixed in llama.cpp
llama.cpp has released critical fixes for Gemma 4's KV cache implementation, dramatically reducing VRAM consumption and making the model practical for local deployment on consumer hardware.
-
VRAM Optimization Technique Cuts Gemma 4 Memory Usage by 3x
A simple llama.cpp parameter adjustment (-np 1) significantly reduces Sliding Window Attention cache VRAM requirements for Gemma 4, enabling deployment on systems with limited GPU memory.
-
TurboQuant KV Cache Compression Achieves 22.8% Faster Decoding at 32K Context
Google's TurboQuant compression method has been successfully integrated into llama.cpp, enabling 4.6x KV cache compression and 22.8% decode speedup at 32K context length by skipping 90% of dequantization work. This breakthrough makes long-context inference practical on consumer hardware like MacBook Air M4.
-
LMCache Dramatically Accelerates LLM Inference on Oracle Data Science Platform
Oracle integrates LMCache, a cutting-edge prompt caching and KV cache optimization technique, into their cloud data science platform to accelerate LLM inference and reduce computational overhead.
-
3-Path Agent Memory: 8 KB Recurrent State vs. 156 MB KV Cache at 10K Tokens
A new memory architecture demonstrates significant efficiency gains for local LLM agents, reducing memory footprint from 156 MB to just 8 KB while maintaining performance at 10K token contexts. This breakthrough is critical for deploying agents on resource-constrained devices.