Tagged "kv-cache"
7 articles tagged kv-cache, 24 March 2026 to 31 August 2026. Newest first.
-
How Much Context Actually Fits in Your VRAM
The KV cache arithmetic for any model, read straight from its config.json — plus the four architectures that break the standard formula, one of them by 57x.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
A comprehensive guide addressing one of the most critical bottlenecks in local LLM deployment: KV cache memory consumption. Learn practical strategies to manage GPU memory constraints when running LLMs on-device.
-
CacheWise Optimizes KVCache Reuse for LLM Coding Agents
CacheWise improves inference efficiency by optimizing KVCache reuse in language models used for coding tasks. This memory optimization technique reduces computational overhead and latency for agent-based LLM applications.
-
Reducing GPU Costs for AI Inference: FP8, FP4, and vLLM Optimization Techniques
New optimization approaches using FP8, FP4 quantization, and vLLM frameworks are significantly reducing computational costs for AI inference. These techniques enable efficient deployment of larger models on limited hardware.
-
Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing
A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.
-
Mixed KV Cache Quantization: Performance Risks and Pitfalls
A technical deep-dive warning against mixed-precision KV cache quantization, revealing accuracy degradation that contradicts common optimization assumptions.
-
KV Cache Quantization Levels Benchmarked on SWE-bench: Practical Trade-offs for Local Inference
Systematic benchmarking of different KV cache quantization levels using SWE-bench-lite provides early empirical data on quality-versus-memory trade-offs, helping practitioners optimize memory usage in local deployments without sacrificing reasoning performance.