Tagged "inference-performance"
36 articles tagged inference-performance, 11 February 2026 to 24 September 2026. Newest first.
-
Llama.cpp v0.5.0: Backend Performance, Broader Model Support, and Robust Server Operations
The v0.5.0 release of llama.cpp brings significant improvements to backend performance, adds support for additional model architectures, and enhances the HTTP server for production deployment scenarios.
-
Benchmarking Local LLM Servers: Llama.cpp, Llamafile, LM Studio, and Ollama
Mozilla AI publishes comprehensive benchmarks comparing four major local LLM inference servers, providing practical performance data for selecting the right tool for on-device deployment.
-
Which Mac for Local LLMs in 2026? A Comprehensive Buyer's Guide
A practical guide helping Mac users select the right hardware for running local LLMs in 2026, comparing M-series chips, RAM configurations, and storage options for different inference workloads.
-
llama.cpp Release b10781: Vulkan Backend and Efficiency Improvements
Latest llama.cpp release includes Vulkan fixes and optimizations for cross-platform GPU inference, continuing the project's rapid iteration on inference performance and hardware support.
-
Ollama v0.32.15 Adds Model Metadata Cache to Reduce Per-Request Overhead
Ollama releases v0.32.15 with a new model metadata cache feature designed to reduce per-request overhead and improve inference efficiency. This update includes desktop onboarding improvements and MLX framework updates.
-
AMD Optimizes Qwen 3.8 27B for Ryzen AI Max and Radeon GPUs
AMD announces native support for running Qwen 3.8 27B on Ryzen AI Max processors and Radeon GPUs, enabling high-performance local inference on consumer AMD hardware.
-
vLLM v0.27.0 Released with Major Kernel Improvements and New Model Support
vLLM's latest release brings 561 commits from 242 contributors, including full-stack support for Kimi K3 models, new kernel optimizations, and expanded hardware compatibility. The release focuses on performance improvements critical for efficient local LLM serving.
-
vLLM v0.27.0rc2 Release Candidate Available
vLLM releases v0.27.0rc2, continuing its evolution as a high-performance inference engine for local and self-hosted LLM deployment. The release candidate stage indicates maturity and readiness for production use.
-
Llama.cpp B10327 Fixes CUDA Quantized Copy Kernel Performance
The latest llama.cpp release addresses critical thread and block count issues in CUDA quantized copy kernels, improving inference performance on NVIDIA GPUs. This fix ensures more efficient parallel execution for quantized model operations.
-
Show HN: Benchmark Local LLMs Fit for Your Device Specs
A new benchmarking tool helps developers evaluate which local LLMs are suitable for their specific hardware constraints. This addresses a critical pain point in local LLM deployment: matching model capabilities to available compute resources.
-
vLLM v0.27.0rc1: Latest Release Candidate for High-Performance Inference
vLLM announces v0.27.0rc1, the latest release candidate bringing continued improvements to the popular open-source LLM serving engine optimized for local and distributed deployments.
-
llama.cpp Build b10258: Sampling Architecture Refinements
Latest llama.cpp release includes structural improvements to sampling mechanisms with vocabulary handling updates that align with existing samplers like logit bias and mirostat.
-
Phi-4 Mini vs Gemma 3 vs Llama 3.2: 128K vs 32K Context Window Comparison
A detailed comparison of three leading lightweight LLMs optimized for local deployment, focusing on context window capabilities and performance tradeoffs. This benchmark helps practitioners choose the right model for their hardware constraints and use cases.
-
AI Inference is Rewriting the GPU Buying Playbook
A comprehensive analysis of how the emergence of local AI inference is fundamentally changing GPU purchasing decisions and hardware optimization priorities.
-
llama.cpp b10075 Packs Four Local AI Runtime Upgrades
The latest llama.cpp release introduces four significant runtime improvements for local LLM inference, enhancing performance and efficiency across CPU and GPU deployments.
-
Samsung's UFS 5.0 Addresses Critical Memory Bandwidth Bottleneck in Mobile AI Inference
Samsung's new UFS 5.0 technology targets the storage I/O bottleneck that has constrained on-device LLM performance, enabling faster model loading and improved inference latency on mobile platforms.
-
Developer Switches from LM Studio to llama.cpp, Citing Performance and Simplicity
A How-To Geek article documents why developers are moving away from heavier LM Studio implementations toward the leaner llama.cpp inference engine for local LLM deployment.
-
NVIDIA RTX Spark Superchip Delivers 6,144 CUDA Cores for Consumer Local AI Inference
NVIDIA's new RTX Spark superchip combines 6,144 CUDA cores with a 20-core Grace CPU, targeting consumer and creator machines with unprecedented local AI performance. The chip architecture mirrors smartphone efficiency approaches while delivering desktop-class compute for on-device inference.
-
Nvidia Enters Windows Laptop Market, Taking on Intel and AMD
Nvidia's entry into the Windows laptop GPU market with dedicated consumer hardware expands the available options for local LLM deployment on consumer machines and edge devices.
-
Microsoft and Nvidia to Unveil First Windows PCs with Nvidia CPUs and AI Capabilities
Microsoft and Nvidia are collaborating to introduce Windows PCs powered by Nvidia CPUs with integrated AI capabilities for local inference. This partnership signals major hardware vendors' commitment to on-device AI performance.
-
Tweaking Local Language Model Settings with Ollama
A practical guide to optimizing Ollama configurations for various hardware setups and use cases, helping practitioners maximize inference performance on local systems.
-
Qualcomm's AI-Device Strategy Reflects Growing Market Momentum in On-Device Intelligence
Qualcomm's strong financial performance driven by AI expansion signals industry-wide shift toward on-device AI capabilities. The trend accelerates hardware optimization for local inference deployment across mobile and edge devices.
-
Benchmarking a Portable AI Workstation: Lenovo ThinkPad P16 Gen 3, Part 2
Detailed performance analysis of the Lenovo ThinkPad P16 Gen 3 as a portable AI workstation, providing real-world benchmarks for local LLM inference and training workflows.
-
Hipfire: A Rust-Native AMD Inference Engine That Outperforms llama.cpp
Hipfire, a new Rust-native inference engine optimized for AMD consumer GPUs, demonstrates performance improvements over the widely-used llama.cpp framework. This breakthrough offers local LLM practitioners a faster alternative for AMD-based setups.
-
Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing
A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.
-
Users Report Significant Performance Improvements After Migrating from Ollama to llama.cpp
Local LLM practitioners are experiencing notable speed and stability improvements when switching from Ollama to direct llama.cpp implementations, suggesting framework-level optimization differences in inference throughput and reliability.
-
AMD Announces Day 0 Support for Google Gemma 4 Across Processors and GPUs
AMD has delivered immediate support for Google's Gemma 4 model across its processor and GPU lineup, enabling optimized local inference on AMD hardware. This expands accessibility for running powerful open-weight models on-device.
-
Unpaved: Audit Toolkit for AI Developer Tool Bias in Global South Contexts
Unpaved provides an open-source auditing framework to identify and mitigate biases in AI development tools, with specific focus on performance and fairness in Global South contexts. This toolkit is essential for practitioners deploying local LLMs in resource-constrained and underrepresented regions.
-
Mistral Small 4 119B Released with NVFP4 Quantisation Support
Mistral AI releases Mistral Small 4 119B model with official NVFP4 quantisation, enabling efficient local deployment on consumer hardware. The model family is now integrated into HuggingFace Transformers with multiple quantisation variants available.
-
Comprehensive MoE Backend Benchmarks for Qwen3.5-397B: Real Numbers vs Hype
A detailed benchmark of every major MoE backend for Qwen3.5-397B NVFP4 on workstation GPUs reveals actual sustained performance of 50.5 tok/s, significantly lower than commonly cited claims. The analysis uncovers kernel issues in Nvidia's own CUTLASS implementation.
-
HP OMEN MAX 16 Review: Is Local AI on a Laptop Viable in 2026?
A comprehensive review examining whether modern gaming laptops can effectively run local LLMs, testing real-world inference performance and practical viability for local AI deployment.
-
HP Refreshes Lineup with AI-Focused Workstations
HP introduces new AI-optimized workstations designed for local model deployment and on-device inference. These systems target professionals running large language models locally with enhanced compute and memory configurations.
-
HP ZBook Ultra 14 G1a Workstation Reclaims Local AI Workflows for Professionals
A detailed review of the HP ZBook Ultra 14 G1a demonstrates how modern workstation-class laptops enable practical local AI model deployment for professional workflows. The review evaluates performance and suitability for on-device inference tasks.
-
DeepSeek Paper – DualPath: Breaking the Bandwidth Bottleneck in LLM Inference
DeepSeek researchers present DualPath, a novel approach to address bandwidth limitations during LLM inference. This work tackles one of the primary performance bottlenecks in local and edge LLM deployment.
-
A Tool to Tell You What LLMs Can Run on Your Machine
LLMfit is a new tool that analyzes your hardware and recommends which LLMs are compatible and can run efficiently on your specific machine. This solves a common pain point for local LLM deployment by automating hardware capability assessment.
-
Community Member Builds 144GB VRAM Local LLM Powerhouse
A LocalLLaMA community member showcases a custom-built system with 6x RTX 3090 GPUs providing 144GB of VRAM, featuring modified drivers with P2P support for high-performance local LLM inference.