Tagged "memory-management"
34 articles tagged memory-management, 11 February 2026 to 12 October 2026. Newest first.
-
Comprehensive Guide: Local LLM Inference at Scale with vLLM
A detailed technical guide covering strategies and best practices for scaling local LLM inference using vLLM, including optimization techniques for production deployments.
-
llama.cpp Fixes K-Pool Graph Reallocation: Preventing Decode-Time Performance Regressions
llama.cpp release b11412 fixes an unexpected graph reallocation issue in k-pool models that was causing decode-time performance degradation, particularly affecting recent models like Qwen and GLM variants.
-
Llama.cpp Fork Achieves 2-4x MultiGPU Speedup for MoE Models Larger Than VRAM
A community fork of llama.cpp enables efficient distributed inference for Mixture-of-Experts models that exceed single GPU VRAM capacity, achieving 2-4x speedup improvements across multiple GPUs.
-
Ollama 0.34.1 Stabilizes MLX Backend and GGUF Model Creation
Ollama v0.34.1 releases improved MLX memory handling for Apple Silicon, stabilizes GGUF creation workflows, and enhances repeat token detection for more reliable local inference.
-
Ollama v0.34.1 releases with MLX improvements and memory optimizations
The latest Ollama release brings MLX runner enhancements including prefix cache eviction, improved system memory management, and higher token repeat limits for more stable inference.
-
vLLM v0.29.0 Advances with Model Runner V2 as Default
vLLM's latest release makes Model Runner V2 the default for all models, featuring CUDA graph memory profiling and improved performance across deployment scenarios.
-
Running 104GB Qwen3.8-Flash-Next on 48GB Mac at ~12 tok/s
A developer demonstrates running a 104GB model on a 48GB Mac using innovative slot streaming techniques, achieving practical inference speeds of ~12 tokens/second and expanding the possibilities for large model deployment on consumer hardware.
-
Optimizing Qwen 3.6 for Local Development: A Developer's Guide
A practical developer guide for optimizing the Qwen 3.6 model specifically for local development environments, covering configuration and performance tuning.
-
How To Build Your Own LLM Runtime From Scratch
A comprehensive guide on constructing custom LLM inference runtimes, providing practitioners with deep knowledge to optimize and control local model deployment without relying on black-box frameworks.
-
Prefill Once, Fan Out: KV Snapshot Sharing for Multi-Agent LLM Pipelines
Towards Data Science published research on KV snapshot sharing optimization that enables efficient multi-agent LLM pipelines by reusing computed key-value caches across multiple agents. This technique significantly reduces compute requirements for local deployment scenarios.
-
Running Infinite Context Lengths on 8GB GPU Without Out Of Memory
A new engine enables running LLMs with effectively infinite context windows on consumer GPUs with just 8GB VRAM by avoiding memory exhaustion. This breakthrough makes long-context inference practical for edge and local deployments.
-
Show HN: LLM Memory Without Context Bleed – 100% Precision vs. <10% Vector Search
A new memory system for LLM applications achieves 100% precision in context retrieval compared to vector search's <10%, enabling more reliable and efficient local deployment of agentic systems.
-
Meet Memory OS: A 6-Layer Open-Source Memory Stack Built on Hermes Agent
An open-source Memory OS project introduces a modular, six-layer memory architecture designed to enhance local AI agent capabilities. The framework enables more sophisticated context management and reasoning for locally-deployed autonomous AI systems.
-
Tweaking Local Language Model Settings with Ollama
A practical guide to optimizing Ollama configurations for various hardware setups and use cases, helping practitioners maximize inference performance on local systems.
-
Dell Launches 14 Plus Laptop with Intel Core Ultra 9 and 32GB RAM at $1,499.99, Enabling Local Model Inference
Dell's new 14 Plus laptop featuring Intel Core Ultra 9 processor and 32GB RAM offers an affordable platform for running local LLMs and edge AI workloads on consumer hardware.
-
Linux 7.1-rc4 Released: Kernel Updates Relevant to Local LLM Inference
Latest Linux kernel release candidate includes optimizations impacting edge LLM deployment on commodity hardware. Performance improvements for memory management and CPU scheduling affect local inference efficiency.
-
Lemonade Gives AMD Startups a Wider Path to Local Inference
Lemonade framework expands support for AMD hardware in local LLM inference, providing startups with more accessible and cost-effective options for on-device model deployment.
-
Show HN: A Local-First Agentic Knowledge Manager
Kept is a new open-source project providing local-first infrastructure for managing agentic AI workflows with persistent memory and knowledge organization capabilities.
-
Ask HN: Real life autonomous AI Agents
Community discussion examining practical implementations of autonomous agents powered by local LLMs, sharing deployment experiences and real-world use cases.
-
Agentic AI Community Focus: Building Local Agents in 2026
The emerging agentic AI community shares resources and frameworks for building autonomous agents with local LLM backends. Focus areas include memory systems, tool integration, and edge deployment of multi-step reasoning tasks.
-
5 Things I Wish Someone Had Told Me Before I Tried Self-Hosting a Local LLM
A practical guide sharing key lessons learned from self-hosting local LLMs, covering pitfalls and best practices that can accelerate the learning curve for practitioners new to on-device inference. The article distills common mistakes and recommendations from real-world deployment experience.
-
Self-Hosted LLMs in Production: Real-World Limits and Practical Lessons
Deep dive into the operational challenges and workarounds for deploying LLMs in production environments, drawing on practical experience with self-hosted systems.
-
Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing
A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.
-
I Built a Local AI Stack with 5 Docker Containers, and Now I'll Never Pay for ChatGPT Again
A practical guide demonstrating how to assemble a complete local AI stack using five Docker containers, eliminating dependency on cloud API services. This showcases end-to-end self-hosted LLM infrastructure design.
-
CarryAI's Serverless Vision-Language Models Enable On-Device Multimodal AI
CarryAI has introduced serverless vision-language models optimized for on-device deployment, signaling a new era where multimodal AI can run efficiently on edge hardware without cloud dependencies.
-
Running a 1.7B Parameters LLM on an Apple Watch
A developer successfully deployed a 1.7 billion parameter language model on an Apple Watch, demonstrating extreme edge inference capabilities on ultra-constrained wearable hardware.
-
MemPalace, the Highest-Scoring AI Memory System Ever Benchmarked
MemPalace is a novel AI memory system that achieves record-breaking benchmark performance, with implications for improving context retention and reasoning capabilities in locally-deployed language models.
-
Octopoda: Open Source Memory Layer for Fully Offline AI Agents
New open-source project Octopoda provides persistent memory capabilities for local AI agents, enabling stateful conversations across sessions entirely on-device with no cloud services or API keys required.
-
Free AI Video Clipper Using Scene and Speech-Based Segmentation
An open-source project provides local AI-powered video segmentation and automatic clipping based on scene changes and speech patterns. This tool demonstrates practical multimedia processing with on-device inference, eliminating cloud API dependencies.
-
SmolLM2-360M Running on Samsung Galaxy Watch 4 with 74% Memory Reduction
Developer optimizes llama.cpp to run language models on smartwatches, achieving 74% RAM reduction through memory model improvements and reducing peak usage from 524MB to practical levels.
-
Local AI Ecosystem Extends Far Beyond Ollama
A comprehensive overview of the diverse tooling and frameworks that comprise the local LLM ecosystem beyond Ollama, helping practitioners understand the full landscape of available options for on-device AI deployment.
-
Forensic Beats Mem0 with 90.1% on LOCOMO Benchmark
Forensic memory system achieves 90.1% on the LOCOMO benchmark, outperforming Mem0 and demonstrating new capabilities for local context and memory management in LLM applications.
-
Book on AI Agents for the Layman: Understanding Agent-Based Systems
A new resource explores AI agents in accessible terms, helping developers understand agent architecture and design patterns relevant to local LLM deployments.
-
Developer Switches from Ollama and LM Studio to llama.cpp for Better Performance
A detailed comparison reveals why switching to raw llama.cpp can provide better control and performance for local LLM deployment compared to popular GUI tools.