Tagged "context-window"
84 articles tagged context-window, 11 February 2026 to 29 September 2026. Newest first.
-
MiniCPM5-2B vs Qwen3.5-4B vs Gemma3 4B: Comparative Benchmark Results
A new benchmark comparison tests three ultra-compact language models (2B-4B parameters) for local deployment, revealing performance trade-offs between Alibaba's MiniCPM5-2B, Qwen3.5-4B, and Google's Gemma3 4B. Results help practitioners select the right small model for their edge inference constraints.
-
Migrating Large Prompts from Anthropic to Self-Hosted Ollama
Developer shares practical lessons learned migrating 35KB preprompts from Claude Opus to self-hosted Ollama, documenting gotchas and workarounds for local LLM deployment.
-
Ollama v0.34.1 releases with MLX improvements and memory optimizations
The latest Ollama release brings MLX runner enhancements including prefix cache eviction, improved system memory management, and higher token repeat limits for more stable inference.
-
How to get better results from local LLMs with Ollama
InfoWorld covers practical strategies for optimizing inference quality and performance when running LLMs locally through Ollama, the popular self-hosted inference framework.
-
Running 104GB Qwen3.8-Flash-Next on 48GB Mac with Slotstream at ~12 tok/s
A breakthrough demonstration of running a 104GB model on a 48GB Mac using adaptive KV streaming techniques, achieving practical inference speeds of ~12 tokens/second. This showcases innovative memory optimization for consumer hardware.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM GPUs
A specialized llama.cpp fork implements adaptive KV streaming to run Qwen 3.8 27B with large context windows on 16GB VRAM GPUs, significantly reducing hardware requirements for production-grade inference.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM
A specialized llama.cpp implementation adds adaptive KV-cache streaming to run Qwen 3.8 27B with large context windows on 16GB GPUs, demonstrating significant memory optimization advances.
-
Gemma 4 MoE for Agentic Coding: Testing Open-Weight Models on AMD APU Hardware
Alex Ewerlof runs Gemma 4 26B MoE for coding on an AMD Ryzen 7 PRO 250 APU with 64GB of RAM, and reports that tooling closes much of the gap to proprietary models — at the cost of cold starts and slower inference.
-
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
vLLM introduces decode context parallelism technique to handle long-context inference efficiently, reducing memory overhead and latency for local deployments processing large documents and extended conversations.
-
How to Run Qwen3.8-27B on a Single 16GB Card
Practical guide demonstrating techniques to fit the 27-billion parameter Qwen3.8 model within 16GB VRAM constraints using llama.cpp, quantization, and RTX 3080 optimizations.
-
Ollama v0.33.1 Adds Qwen3.8 Flash Next Support and Claude Desktop Integration
Ollama releases v0.33.1 with native support for Qwen3.8 Flash Next, enabling seamless integration with Claude Desktop as a third-party gateway provider. This update improves caching and resolves stability issues with long prefills.
-
Building Local LLM Rigs with Used Server GPUs: 32GB VRAM for €220
Practical guide to sourcing used server-grade GPUs for local LLM inference, achieving 32GB of VRAM at fraction of consumer GPU costs, making large model deployment accessible.
-
Optimizing Qwen 3.6 for Local Development: A Developer's Guide
A practical developer guide for optimizing the Qwen 3.6 model specifically for local development environments, covering configuration and performance tuning.
-
Liquid AI LFM2.5-2.6B: Open-Weights Agentic Model With 128K Context and Tool Calling
Liquid AI releases an open-weights agentic model optimized for on-device deployment with 128K context window, tool calling capabilities, and support for extremely low-resource edge hardware.
-
Liquid AI Releases LFM2.5-2.6B: Powerful Agentic Model for Raspberry Pi and Edge Devices
Liquid AI's new LFM2.5-2.6B model brings agentic AI capabilities to resource-constrained devices like Raspberry Pi, featuring 128K context window and tool calling without requiring GPUs or cloud infrastructure.
-
LFM2.5-2.6B: On-Device Agentic Model With 128K Context and Tool Calling
Detailed technical analysis of Liquid AI's LFM2.5-2.6B with open weights, demonstrating how 128K context and tool-calling capabilities are achievable in a 2.6B parameter model optimized for local inference.
-
Google Chrome Reveals Storage Requirements for Integrated Local AI Models
Google discloses how much free disk space Chrome requires to install and run local AI models, indicating the browser is moving toward on-device model deployment for inference.
-
LLM Memory Doesn't Only Get Written Wrong, It Goes Wrong Later
Research on how LLM memory degrades and becomes corrupted over time during inference. Understanding memory behavior is critical for reliable local deployment.
-
K-EXAONE 2.0 Brings 262K Context to Frontier AI
K-EXAONE 2.0 introduces a 262K token context window, significantly expanding the capabilities of frontier-class models for local deployment and extended reasoning tasks. This represents a major advancement in practical context window management.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
Deep dive into KV cache management and practical strategies to prevent GPU out-of-memory errors when running local LLMs, a critical bottleneck for on-device inference.
-
Phi-4 Mini vs Gemma 3 vs Llama 3.2: 128K vs 32K Context Window Comparison
A detailed comparison of three leading lightweight LLMs optimized for local deployment, focusing on context window capabilities and performance tradeoffs. This benchmark helps practitioners choose the right model for their hardware constraints and use cases.
-
Claude Code Cut System Prompt by 80%: Implications for Small Local Models
Anthropic's dramatic 80% system prompt reduction in Claude Code raises questions about prompt efficiency for smaller, resource-constrained models deployed locally.
-
I Thought My Local AI Would Replace My Claude Subscription — Then I Tried Automating My PC
An XDA Developers article explores the practical limitations of local LLMs when applied to complex automation tasks, revealing the gap between running models locally and achieving production-grade reliability for PC automation workflows. The piece offers candid insights into real-world local AI deployment challenges.
-
Developer Ditches Ollama for llama.cpp's WebUI: A Practical Comparison
An experienced practitioner switched from Ollama to llama.cpp's WebUI after preferring its control, performance, and flexibility for local model inference. The shift highlights ongoing competition between local inference frameworks and the importance of evaluating tools for specific use cases.
-
Exploiting Sparsity for Long Context Inference: Million Token on Commodity GPUs
A new technique enables million-token context windows on standard consumer GPUs by leveraging sparsity optimizations. This breakthrough makes long-context LLM inference practical and affordable for self-hosted deployments.
-
SigMap: 97% Token Reduction for AI Coding Sessions
SigMap achieves significant token efficiency improvements for AI coding workflows, reducing context size by 97% while maintaining functionality. This breakthrough in token optimization has direct implications for running LLMs locally with constrained memory and compute resources.
-
LongCat-2.0 Released
LongCat-2.0 represents an advancement in handling long-context sequences locally. While limited details are available, this release is relevant to local LLM practitioners seeking models optimized for extended context windows on consumer hardware.
-
A Guide on How to Run Nemotron 3 Super 120B Thinking on 2 Nvidia DGX Spark
Practical deployment guide for running NVIDIA's large reasoning model (120B parameters) on a two-node DGX Spark cluster with distributed inference techniques.
-
TokenTamer: A Proxy That Reduces LLM Token Usage Through Context Compression
TokenTamer is a new proxy tool that optimizes LLM token consumption through intelligent context compression, reducing costs and improving inference performance for local deployments.
-
AI Memory Systems Show Critical Limitations: 95% Error Rate in Key Benchmarks
Research unveiled severe memory retention failures in AI systems, with error rates reaching 95%, highlighting critical challenges for long-context local LLM deployments requiring persistent memory.
-
Running Infinite Context Lengths on 8GB GPU Without Out Of Memory
A new engine enables running LLMs with effectively infinite context windows on consumer GPUs with just 8GB VRAM by avoiding memory exhaustion. This breakthrough makes long-context inference practical for edge and local deployments.
-
Show HN: LLM Memory Without Context Bleed – 100% Precision vs. <10% Vector Search
A new memory system for LLM applications achieves 100% precision in context retrieval compared to vector search's <10%, enabling more reliable and efficient local deployment of agentic systems.
-
From Specialists to Builders: How AI Agentic Coding Is Reshaping Software Teams
An analysis of how agentic AI systems are transforming software development workflows, with implications for teams deploying local LLMs in development environments.
-
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
A breakthrough in LLM inference optimization achieves 3,000 tokens per second on standard GPUs, significantly improving real-time inference performance for local deployments.
-
Tweaking Local Language Model Settings with Ollama
A practical guide to optimizing Ollama configurations for various hardware setups and use cases, helping practitioners maximize inference performance on local systems.
-
Local LLM Setup: How to Use RAG and an Embedding Model to Stop Wasting Context
A practical guide on optimizing local LLM deployments by combining retrieval-augmented generation with embedding models to maximize context efficiency and reduce token waste.
-
Show HN: Find the best local LLM for your hardware, ranked by benchmarks
A new GitHub tool helps developers identify the optimal local LLM for their specific hardware constraints by ranking models across performance benchmarks. This addresses a key pain point in the local LLM ecosystem where choosing between dozens of models requires extensive manual testing.
-
LLM Hallucinations in the Wild
A comprehensive study documents real-world hallucination behaviors in deployed language models, providing practitioners with empirical data on failure modes when running models locally.
-
Microsoft Researchers Find AI Models and Agents Can't Handle Long-Running Tasks
New research from Microsoft reveals fundamental limitations in current AI models and agents when managing long-duration operations, impacting local deployment strategies for autonomous systems.
-
One LM Studio Setting Change Makes Local LLMs Competitive With Cloud Models
A simple configuration adjustment in LM Studio dramatically improves local LLM performance, making self-hosted inference viable for production workloads previously requiring cloud APIs. This discovery highlights how software optimization can rival hardware improvements.
-
I got prompt-injected asking Claude on iOS to recommend a cycling route app
Security research highlighting prompt injection vulnerabilities in LLM applications, demonstrating why local models with controlled inputs offer advantages.
-
SigMap – Shrink AI Coding Context 97% with Auto-Scaling Token Budget
SigMap introduces an auto-scaling token budget system that reduces AI coding context by 97%, enabling more efficient local model inference for code generation and analysis tasks. This performance optimization is critical for running models on memory-constrained devices.
-
Learn LLM Internals
A comprehensive GitHub repository documenting the internal mechanics of large language models, providing developers with deep knowledge necessary for optimizing local deployments. Essential reference material for understanding how to tune and optimize models running on limited hardware.
-
Gemma 4 31B vs Qwen 3.5 27B: Comprehensive Long Context Benchmark
Community benchmark comparing Gemma 4 31B and Qwen 3.5 27B for long context workloads on 24GB VRAM, establishing these as the top local models for mid-range GPU setups.
-
MemPalace, the Highest-Scoring AI Memory System Ever Benchmarked
MemPalace is a novel AI memory system that achieves record-breaking benchmark performance, with implications for improving context retention and reasoning capabilities in locally-deployed language models.
-
Context Window Optimization: Extending Gemma 4 Context Length Through Efficient Projection Quantization
Community members discover that quantizing vision projections to Q8 format in Gemma 4 multimodal models eliminates quality degradation while enabling 30K additional context tokens without VRAM increase.
-
Gemma 4 26B A4B Outperforms Qwen 3.5 35B on Apple Silicon
Testing on Mac Studio M5 Ultra shows Gemma 4 26B achieves comparable speed (1000 tokens/sec prompt, 60 tokens/sec generation) to larger Qwen 3.5 35B while demonstrating significantly better output quality and reasoning behavior.
-
Linux Significantly Outperforms Windows for Local LLM Inference
A detailed comparison shows inference running substantially faster on Linux versus Windows on identical hardware, with implications for local deployment optimization.
-
Mixed KV Cache Quantization: Performance Risks and Pitfalls
A technical deep-dive warning against mixed-precision KV cache quantization, revealing accuracy degradation that contradicts common optimization assumptions.
-
TurboQuant KV Cache Compression Achieves 22.8% Faster Decoding at 32K Context
Google's TurboQuant compression method has been successfully integrated into llama.cpp, enabling 4.6x KV cache compression and 22.8% decode speedup at 32K context length by skipping 90% of dequantization work. This breakthrough makes long-context inference practical on consumer hardware like MacBook Air M4.
-
Qwen3 512k Context via TurboQuant on Mac mini
Qwen3 achieves 512k token context window using TurboQuant quantisation on Mac mini hardware, demonstrating significant advances in local long-context model deployment.
-
Qwen 3.5 27B Achieves 1.1M Tokens/Second on B200 GPUs with Optimized vLLM Config
A developer optimized Qwen 3.5 27B to reach 1.1 million tokens per second on 96 B200 GPUs using vLLM, with detailed configurations and all settings published on GitHub. Key optimizations included distributed parallelism, reduced context windows, FP8 KV cache, and speculative decoding.
-
Chinese LLM Ecosystem Landscape: ByteDance Doubao, Alibaba, and Open-Source Competition
Comprehensive analysis of the Chinese LLM scene reveals ByteDance's Doubao as the market leader with strong open-source alternatives from Alibaba, Deepseek, and others, highlighting the rapid innovation and diverse model ecosystem emerging from China's AI development.
-
A Little Gap That Will Ensure the Future of AI Agents Being Autonomous
A discussion examining a critical architectural or capability gap that needs resolution to enable truly autonomous local AI agents, relevant to on-device deployment paradigms.
-
ik_llama.cpp Fork Delivers 26x Faster Prompt Processing on Qwen 3.5 27B
A fork of llama.cpp called ik_llama.cpp is delivering dramatic 26x speed improvements for prompt processing on Qwen 3.5 27B models. Real-world benchmarks on Blackwell RTX PRO GPUs show tangible performance gains for production agentic workloads.
-
AI Playground for Developers Built in Vite and Python
A new developer-focused platform combining Vite frontend tooling with Python backends, designed to simplify local LLM experimentation and deployment prototyping.
-
Community Converges on Optimal KV Cache Quantization Strategies for Qwen 3.5 Models
The local LLM community is establishing practical guidelines for KV cache quantization with Qwen 3.5, balancing memory savings against accuracy loss to optimize inference on consumer hardware.
-
Mamba 3: State Space Model Architecture Optimized for Inference
Mamba 3 introduces a state space model architecture specifically optimized for efficient inference performance, offering a potential alternative to traditional transformer-based architectures for local deployment.
-
Running Qwen3.5-27B Across Multiple GPUs Over LAN Achieves Practical Speed for Local Inference
A practitioner successfully split Qwen3.5-27B across a 4070Ti and AMD RX6800 over LAN using llama.cpp's RPC server, achieving 13 tokens/second with 32K context—demonstrating that heterogeneous multi-GPU local setups are now viable. This shows path forward for GPU-poor practitioners seeking reasonable performance.
-
Memory Should Decay: Implementing Temporal Memory Decay in Local LLM Systems
Research on memory decay mechanisms suggests that implementing forgetting patterns in local LLM systems could improve efficiency and realism in agent behavior. This approach addresses context accumulation problems in long-running local inference workloads.
-
3-Path Agent Memory: 8 KB Recurrent State vs. 156 MB KV Cache at 10K Tokens
A new memory architecture demonstrates significant efficiency gains for local LLM agents, reducing memory footprint from 156 MB to just 8 KB while maintaining performance at 10K token contexts. This breakthrough is critical for deploying agents on resource-constrained devices.
-
8 Local LLM Settings Most People Never Touch That Fixed My Worst AI Problems
A practical guide exploring often-overlooked configuration parameters in local LLM deployments that can dramatically improve performance and resolve common issues.
-
Qwen 3.5 Family Benchmark Comparison Shows Strong Performance Across Smaller Models
New benchmarks reveal that Qwen 3.5's 27B, 35B, and 122B variants retain most of the flagship model's performance, while smaller 2B and 0.8B models show steeper degradation on long-context and agent tasks.
-
ETH Zurich Research Challenges Context-Length Assumptions in LLM Agents
A peer-reviewed study from ETH Zurich demonstrates that larger context windows don't consistently improve agent performance on real coding tasks, with context inflation actually reducing success rates by 2-3% while increasing costs by 20%.
-
Analysis Reveals Claude Code Sends 62,600 Characters of Tool Definitions Per Turn
A detailed technical analysis traces how Claude Code uses context window tokens, comparing it against five different CLI implementations. The findings highlight inefficiencies in current tool-passing approaches for local LLM deployment.
-
Incrmd: Incremental AI Coding by Editing PROJECT.md
A novel approach to AI-assisted development that uses a PROJECT.md file as a specification interface, enabling incremental, reproducible code generation with local LLMs. Optimizes LLM context and reasoning through structured markdown specifications.
-
Critical: Qwen 3.5 Requires BF16 KV Cache, Not FP16 for Accurate Inference
Community member Daniel Han alerts users that Qwen 3.5 models require bfloat16 KV cache precision instead of the default float16, with perplexity measurements demonstrating the accuracy impact when using incorrect cache formats.
-
Qwen 3.5 27B on Dual RTX 3090s: 170K Context Holds, 100+ Tokens/s Claim Disputed
A widely shared r/LocalLLaMA video reported Qwen 3.5 27B running at 100+ tokens/second decode with a 170K context window on dual RTX 3090s. The context claim holds and is in fact understated — 262K fits. The decode figure is contradicted by independent benchmarks measuring 41.4 t/s on the same model and hardware, and the original video has never been independently verified.
-
Researchers Develop Persistent Memory System for Local LLMs—No RAG Required
A novel approach enables local language models to retain facts learned during conversations by storing them directly in model weights through a sleep mechanism. The system runs on consumer hardware like MacBook Air and eliminates the need for traditional retrieval-augmented generation.
-
Qwen 3.5 MoE Delivers 100K Context Window at 40+ TPS on RTX 5060 Ti
Qwen3.5's mixture-of-experts variant achieves exceptional throughput with 100,000 token context window on a single mid-range GPU, reaching 41+ tokens per second using the Vulkan backend. This demonstrates practical feasibility of ultra-long context models on consumer hardware.
-
Qwen3.5-27B Identified as Sweet Spot for Mid-Range Local Deployment
Users are reporting that Qwen3.5-27B offers the ideal balance of performance and resource efficiency for local inference, with verified setups running at 19.7 tokens/sec on consumer GPUs with reasonable memory footprints.
-
Breaking the Speed Limit: Strategies for 17k Tokens/Sec Local Inference
New techniques and optimisations enable local LLM inference to achieve 17,000 tokens per second, pushing the boundaries of what's possible on consumer hardware. This breakthrough demonstrates practical strategies for maximising throughput in edge deployments.
-
Breaking the Speed Limit: Strategies for 17k Tokens/Sec Local Inference
Practical strategies and techniques for achieving ultra-high token throughput in local LLM inference, reaching 17,000 tokens per second. Essential performance optimization guide for practitioners running models on-device.
-
Google Is Exploring Ways to Use Its Financial Might to Take on Nvidia
Google explores strategic investments and partnerships to compete with Nvidia's dominance in AI accelerator chips, potentially enabling more accessible hardware options for local LLM deployment. This shift could significantly impact the economics of on-device inference infrastructure.
-
Why AI Models Fail at Iterative Reasoning and What Could Fix It
An analysis of fundamental limitations in how local LLMs perform iterative reasoning tasks and proposes solutions applicable to on-device inference and self-hosted deployments.
-
The Path to Ubiquitous AI (17k tokens/sec)
A technical analysis of achieving 17,000 tokens per second inference throughput, demonstrating the performance milestones required for truly practical local LLM deployment at scale.
-
Qwen3 Coder Next 8FP Demonstrates Exceptional Long-Context Performance on 128GB System
Qwen3 Coder Next 8FP successfully processed 12+ hours of continuous Flutter documentation conversion with 64K max tokens, utilizing 102GB of 128GB system memory. This showcases the model's capability for demanding real-world document processing tasks on high-end local hardware.
-
GLM-5 Technical Report: DSA Innovation Reduces Training and Inference Costs
Alibaba releases GLM-5 technical report detailing key innovations including DSA adoption that significantly reduces training and inference costs while maintaining long-context fidelity.
-
SnowBall Technique Addresses Context Window Limitations in Local LLMs
New SnowBall approach enables iterative context processing when content exceeds LLM context windows, offering practical solutions for local deployment constraints.
-
NVIDIA's Dynamic Memory Sparsification Cuts LLM Inference Costs by 8x
NVIDIA introduces Dynamic Memory Sparsification technique that reduces LLM reasoning costs by 8x through intelligent KV cache management without accuracy loss.
-
Context Management Identified as Real Bottleneck in AI-Assisted Coding
Discussion highlights how context window limitations and management, rather than model capabilities, represent the primary challenge for local AI coding assistants.
-
GPT-OSS 120B Uncensored Model Released in Native MXFP4 Precision
An uncensored version of GPT-OSS 120B has been released featuring native MXFP4 precision training, offering 117B parameters with MoE architecture for efficient local deployment.
-
Use Recursive Language Models to address huge contexts for local LLM
A powerful and innovative technique for extending context windows for use in local models
-
DeepSeek Launches Model Update with 1M Context Window
DeepSeek has updated their model to support 1 million token context windows with a knowledge cutoff of May 2025, currently in grayscale testing phase with potential for local deployment.