Tagged "cpu-inference"
73 articles tagged cpu-inference, 11 February 2026 to 19 September 2026. Newest first.
-
GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained
A comprehensive comparison of the major quantization formats used in local LLM deployment, covering GGUF, GPTQ, AWQ, and EXL2 formats and their tradeoffs for on-device inference.
-
Benchmarking Local LLM Servers: Llama.cpp, Llamafile, LM Studio, and Ollama
Mozilla AI publishes comprehensive benchmarks comparing four major local LLM inference servers, providing practical performance data for selecting the right tool for on-device deployment.
-
MoE Expert Offload: What a 35B Model Actually Costs on a 12GB Card
92.9% of Qwen3.6-35B-A3B is routed expert weights, and only 3.1% of them are read per token — which is why a 19 GiB model runs on 12 GB at all. The roofline arithmetic for expert offload, and why the same sum that permits 50 tok/s at 8K refuses it at 128K.
-
AMD EPYC ZenDNN Accelerates llama.cpp Prompt Processing 4.5x
AMD's ZenDNN library delivers up to 4.5x performance improvement for llama.cpp on EPYC processors, significantly accelerating prompt processing speeds for server-side local LLM deployments.
-
Multiverse Computing's CompactifAI Models Now Fully Compatible with Intel Xeon 6 Processors
All CompactifAI optimised models have achieved compatibility with Intel Xeon 6 processors, enabling efficient inference on enterprise server hardware and expanding deployment options for self-hosted local LLM infrastructure. This compatibility expands the practical deployment platforms for optimised models.
-
AMD Ryzen 7 7700X3D Linux Performance Review
Phoronix publishes detailed Linux performance benchmarks for the AMD Ryzen 7 7700X3D processor, providing critical data for practitioners evaluating CPU hardware for local LLM inference and edge AI workloads. The 3D V-Cache architecture offers unique advantages for memory-heavy AI tasks.
-
AMD ZenDNN 6.0 Boosts AI Inference on EPYC CPUs With FP16 and MoE Acceleration
AMD has released ZenDNN 6.0 with optimizations for FP16 inference and Mixture-of-Experts model acceleration on EPYC processors. This update enables efficient local LLM deployment on AMD server and workstation CPUs without requiring GPUs.
-
RISC-V RVV Vector Benchmarks: SpacemiT K3 SoC Performance for Edge AI
Performance benchmarking of the SpacemiT K3 system-on-chip using RISC-V vector extensions reveals competitive inference capabilities for local AI workloads on alternative CPU architectures.
-
llama.cpp Tutorial: Run a Local LLM in 12 Steps
A comprehensive guide to getting started with llama.cpp, one of the most popular inference engines for running quantized language models locally with minimal dependencies.
-
AMD PACE: New vLLM Plugin Enables Efficient CPU-Based Inference
AMD announces PACE, a vLLM plugin designed to optimize CPU inference for local LLM deployment, expanding viable hardware options beyond traditional GPU-accelerated setups.
-
AMD claims 256-core Zen 6 'Venice' CPU beats Nvidia Vera by 3.3x
AMD's new Zen 6 Venice CPU architecture delivers significant performance improvements for data center and edge inference workloads. Hardware advancement relevant to deploying and scaling local LLM inference.
-
Running Local AI Models on Old Laptops Without GPU
An XDA Developers article demonstrates that capable local language models can run successfully on aging hardware without dedicated GPUs, opening deployment possibilities for resource-constrained environments.
-
Dell Launches 14 Plus Laptop with Intel Core Ultra 9 and 32GB RAM at $1,499.99, Enabling Local Model Inference
Dell's new 14 Plus laptop featuring Intel Core Ultra 9 processor and 32GB RAM offers an affordable platform for running local LLMs and edge AI workloads on consumer hardware.
-
Running Local AI LLMs on Mini PCs Without NVIDIA GPUs
A comprehensive review demonstrates how to effectively deploy and run local language models on compact machines using CPU-based inference and alternative hardware configurations. The guide covers practical setup with Kingston storage and DDR5 memory optimization.
-
Sarvam Edge: Indian-Built AI Models Run Offline on Phones and Laptops Without Internet
Sarvam AI released Sarvam Edge, a suite of models specifically designed for on-device deployment on smartphones and laptops without internet connectivity. This represents a significant step forward in making practical, localized AI accessible across diverse hardware.
-
Intel OpenVINO 2026.1 Integrates llama.cpp with Wildcat Lake and Arc Pro B70
Intel's latest OpenVINO release brings native llama.cpp integration with support for the new Wildcat Lake processors and Arc Pro B70 GPUs, significantly expanding local inference capabilities on Intel hardware.
-
Run Qwen3.5 on an Old Laptop: A Lightweight Local Agentic AI Setup Guide
KDnuggets publishes a practical guide demonstrating how to run Qwen3.5 with agentic AI capabilities on resource-constrained hardware, making advanced local inference accessible to resource-limited environments.
-
Kokoro TTS Achieves 20× Realtime Speed on CPU-Only On-Device Inference
A developer has successfully deployed Kokoro text-to-speech with 20× realtime performance using only CPU inference via MLX Swift on iOS, enabling high-quality, low-latency speech synthesis entirely on-device.
-
Local AI Ecosystem Extends Far Beyond Ollama
A comprehensive look at the broader tooling and framework landscape for local LLM deployment, highlighting alternatives and complementary tools beyond Ollama for various deployment scenarios.
-
TurboQuant Benchmarked in Llama.cpp: Google's Extreme Compression Research Tested in Practice
Community members benchmarked Google's TurboQuant extreme compression technique within llama.cpp, providing practical performance data on the quantisation method. Results show how the research translates to real-world inference speed and memory usage improvements.
-
MacinAI Local brings functional LLM inference to classic Macintosh hardware
A complete local AI inference platform enables TinyLlama 1.1B execution on vintage PowerBook G4 (2002) hardware running Mac OS 9 with zero internet connectivity, demonstrating extreme edge inference capabilities.
-
Repurpose Old GPUs as Dedicated AI Inference Accelerators
An exploration of how older, unused GPUs sitting in drawers can be recycled into effective AI inference hardware, offering compelling performance-per-dollar compared to cloud services or newer hardware purchases.
-
Llamafile 0.10 Released with GPU Support and Rebuilt Core
Mozilla's Llamafile, the portable single-file LLM runner, reaches version 0.10 with enhanced GPU acceleration and a completely rebuilt inference core. This update makes it easier than ever to run large language models locally without complex dependencies.
-
Browser-Based Transcription Tools
Browser-based transcription solutions leverage local inference to enable audio processing entirely within the user's device, eliminating cloud dependency for speech-to-text tasks. This trend reflects growing adoption of WebAssembly and on-device AI models for privacy-preserving audio applications.
-
Run LLMs Locally with Llama.cpp
A practical guide on leveraging llama.cpp for efficient local LLM inference, demonstrating how to optimize model performance on consumer hardware without cloud dependencies.
-
Kimi Introduces Attention Residuals: 1.25x Compute Performance at <2% Overhead
Kimi has released a novel technique called Attention Residuals that achieves a 1.25x improvement in compute performance with minimal overhead, offering significant benefits for local LLM deployment and inference optimization.
-
Show HN: Voice-tracked teleprompter using on-device ASR in the browser
A new browser-based tool that combines on-device automatic speech recognition with teleprompter functionality, enabling voice-tracked presentations without server dependencies. The system processes audio locally in the browser.
-
I made Karpathy's Autoresearch work on CPU
A developer successfully optimized Karpathy's Autoresearch project to run on CPU-only systems, removing GPU dependency. This breakthrough makes advanced research automation accessible to users without GPU hardware.
-
Hybrid AI Desktop Layer Combining DOM-Automation and API-Integrations
A new desktop AI layer that combines DOM automation with API integrations, enabling AI agents to interact with existing applications. The system uses local models for task automation and desktop control.
-
Open-Source GreenBoost Driver Augments NVIDIA GPU VRAM With System RAM and NVMe Storage
A new open-source driver called GreenBoost extends NVIDIA GPU VRAM capacity by intelligently combining it with system RAM and NVMe storage, enabling users to run larger LLMs on existing hardware without additional GPU purchases. This memory-expansion approach addresses a critical bottleneck in local LLM deployment.
-
AMD Launches Agent System Optimized for Local AI Inference With Ryzen and Radeon
AMD announces a new integrated system designed specifically for local AI workloads, combining Ryzen CPUs with Radeon GPU acceleration for efficient inference.
-
Intel OpenVINO Backend Support Now Available in llama.cpp
Intel's team has contributed OpenVINO backend support to llama.cpp, enabling optimized local LLM inference on Intel CPUs and compatible hardware platforms.
-
P-EAGLE: Faster LLM Inference with Parallel Speculative Decoding in vLLM
AWS introduces P-EAGLE, a parallel speculative decoding technique integrated into vLLM that significantly accelerates LLM inference speed. This advancement is crucial for practitioners deploying local LLMs who need to optimize throughput and reduce latency.
-
Intel Updates LLM-Scaler-vLLM With Support For More Qwen3/3.5 Models
Intel has expanded LLM-Scaler-vLLM compatibility to include additional Qwen3 and Qwen3.5 models, improving inference optimization for self-hosted deployments on Intel hardware.
-
Llama.cpp Celebrates Major Milestone: From Leak to Industry Standard
The llama.cpp project marks a significant birthday, reflecting its evolution from a hobbyist experiment running leaked models to the foundational inference engine for local LLM deployment.
-
HP OMEN MAX 16 Review: Is Local AI on a Laptop Viable in 2026?
A comprehensive review examining whether modern gaming laptops can effectively run local LLMs, testing real-world inference performance and practical viability for local AI deployment.
-
Strix Halo (Ryzen AI Max+ 395) Achieves Strong Local Inference Performance with ROCm 7.2
New benchmarks on AMD's Strix Halo platform with ROCm 7.2 backend show practical inference speeds for the Qwen 3.5 model family, with recent llama.cpp optimisations delivering measurable performance gains.
-
When Running Ollama on Your PC for Local AI, One Thing Matters More Than Most
An MSN article identifies the critical performance factor for running Ollama efficiently on personal computers. The piece highlights a key optimization principle that practitioners often overlook when deploying local LLMs.
-
Turning Your Linux Terminal into a Local AI Assistant
A practical guide demonstrating how to integrate a local AI assistant directly into your Linux terminal workflow. This article shows the utility and accessibility of running LLMs on personal machines.
-
OpenWrt 25.12.0 – Stable Release
The latest stable release of OpenWrt, the popular open-source router OS, with improvements relevant to edge AI inference on network devices. Enables deployment of lightweight LLMs directly on routers and edge gateways.
-
AMD Launches Copilot+ Desktop Chips to Compete in On-Device AI Market
AMD has entered the on-device AI competition with its first Copilot+ certified desktop processors, offering an alternative to Intel and Apple for local model inference. The chips target the growing market of Windows-based AI workstations and edge devices requiring native AI acceleration.
-
AMD Ryzen AI 400 Series Desktop Processors Launch with Integrated 60 TOPS NPU
AMD unveils Ryzen AI 400 series desktop processors featuring up to 12 cores and an integrated Radeon 890M GPU with a 60 TOPS NPU. These processors enable local LLM inference on standard desktop machines with Copilot+ support.
-
GitDelivr: A Free CDN for Git Clones Built on Cloudflare Workers and R2
A new infrastructure tool that accelerates large model repository downloads using Cloudflare's edge network, addressing a practical bottleneck for developers downloading LLM weights and codebases locally.
-
AMD Expands Ryzen AI 400 Series Portfolio for Consumer and Enterprise AI PC Options
AMD announced an expanded lineup of Ryzen AI 400 Series processors, bringing more hardware options for local AI inference across consumer laptops and business workstations. The expansion increases accessibility of dedicated NPU hardware for on-device LLM deployment.
-
Browser Use vs. Claude Computer Use: Comparing Agent Automation Frameworks
A technical comparison of two emerging frameworks for autonomous agent control, relevant to deploying agentic AI systems with local or hybrid model backends.
-
Bare-Metal LLM Inference: UEFI Application Boots Directly Into LLM Chat
A novel UEFI application enables booting directly into LLM inference without operating system overhead, eliminating kernel and driver latency for minimal-footprint deployment.
-
The ML.energy Leaderboard
ML.energy launches a comprehensive leaderboard benchmarking model efficiency metrics including inference latency, memory consumption, and energy usage across diverse hardware platforms, providing crucial data for local deployment decisions.
-
LLmFit: Terminal Tool for Right-Sizing LLM Models to Your Hardware
LLmFit is a new command-line tool that automatically detects system hardware specifications and recommends the optimal LLM from a database of 497 models across 133 providers, scoring candidates on quality, speed, fit, and cost.
-
Krasis: Hybrid CPU/GPU MoE Runtime Achieves 3,324 Tokens/Second Prefill on RTX 5080
New open-source runtime optimises mixture-of-experts models by splitting prefill to GPU and decode to CPU, enabling larger MoE models to run on single consumer GPUs with dramatic throughput improvements.
-
What Breaks When AI Agent Frameworks Are Forced Into <1MB RAM and Sub-ms Startup
A deep dive into the fundamental constraints and trade-offs when deploying AI agent frameworks on severely resource-limited devices, exploring what architectural patterns fail and what succeeds at the edge.
-
A Tool to Tell You What LLMs Can Run on Your Machine
LLMfit is a new tool that analyzes your hardware and recommends which LLMs are compatible and can run efficiently on your specific machine. This solves a common pain point for local LLM deployment by automating hardware capability assessment.
-
Open-Source llama.cpp Finds Long-Term Home at Hugging Face
The popular llama.cpp project, essential infrastructure for local LLM inference, has secured a long-term home at Hugging Face. This partnership ensures continued development and maintenance of the widely-used C++ inference engine.
-
GGML Joins Hugging Face: What This Means for Local Model Optimization
GGML, the foundational library for efficient local LLM inference, joins Hugging Face, promising deeper integration and optimization capabilities for edge deployment.
-
CPU-Trained Language Model Outperforms GPU Baseline After 40 Hours
A developer successfully trained FlashLM v5 'Thunderbolt' on CPU hardware, achieving a 1.36 perplexity with just 29.7M parameters and beating established GPU baselines. This demonstrates the viability of efficient CPU-based model training for resource-constrained environments.
-
AI Is Stress Testing Processor Architectures and RISC-V Fits the Moment
RISC-V architecture emerges as a compelling alternative for AI workloads as traditional processor designs face thermal and efficiency challenges under LLM inference loads, opening new possibilities for local deployment on custom silicon.
-
Ouro 2.6B Thinking Model GGUFs Released with Q8_0 and Q4_K_M Quantization
Ouro 2.6B, a looped inference model, is now available as quantized GGUFs (Q8_0 at 2.7GB and Q4_K_M at 1.6GB) compatible with LM Studio, Ollama, and llama.cpp. This enables accessible local deployment of an innovative thinking model architecture.
-
I Thought I Needed a GPU to Run AI Until I Learned About These Models
A practical guide demonstrating that modern optimized models and inference engines enable effective LLM deployment on CPU-only hardware, removing a major perceived barrier to local AI.
-
Google Is Exploring Ways to Use Its Financial Might to Take on Nvidia
Google explores strategic investments and partnerships to compete with Nvidia's dominance in AI accelerator chips, potentially enabling more accessible hardware options for local LLM deployment. This shift could significantly impact the economics of on-device inference infrastructure.
-
At India AI Impact Summit, Intel Showcases Its AI PCs and Cost-Efficient Frugal AI
Intel demonstrates cost-effective AI PC solutions optimized for local inference, highlighting accessible hardware options for deploying LLMs in resource-constrained environments.
-
GGML.AI Acquired by Hugging Face
Hugging Face has acquired GGML.AI, the organization behind llama.cpp, a critical infrastructure project for local LLM inference. This acquisition has major implications for the future development and support of local model deployment tools.
-
PaddleOCR-VL Now Integrated into llama.cpp for Multilingual OCR
PaddleOCR-VL, a 900M parameter multilingual OCR model, has been integrated into llama.cpp, providing open-source optical character recognition capabilities for local LLM workflows. This addition enables fully local document processing pipelines without cloud dependencies.
-
Hardware Economics Shift: DDR5 RDIMM Pricing Now Comparable to GPUs for Local Inference
Analysis shows DDR5 RDIMM memory costs have reached parity with high-end GPUs like RTX 3090s on a per-gigabyte basis, forcing local LLM builders to reconsider their hardware stacking strategies.
-
Matmul-Free Language Model Trained on CPU in 1.2 Hours
Researcher demonstrates training a 13.6M parameter language model entirely on CPU without matrix multiplications, achieving training time of just 1.2 hours with a working model available on Hugging Face.
-
Asus ExpertBook B3 G2 Laptop Features Ryzen AI 9 HX 470 CPU in 1.41kg Ultraportable Form Factor
ASUS launches the ExpertBook B3 G2, an ultralight laptop featuring AMD's Ryzen AI 9 HX 470 processor, delivering significant local AI inference capabilities in a portable 1.41kg package. This hardware development enables practical on-device LLM deployment for mobile professionals.
-
ASUS Zenbook 14 Launches in India with AI-Capable Hardware, Starting at Rs 1,15,990
ASUS introduces the Zenbook 14 in the Indian market with processors optimized for local AI inference, making capable on-device LLM deployment accessible to a broader geographic audience at competitive pricing. The launch reflects growing demand for edge AI capabilities in emerging markets.
-
GPU-Accelerated DataFrame Library for Local Inference Workloads
A new DataFrame library that runs on GPUs, accelerators, and alternative hardware, enabling efficient data processing for local AI inference pipelines.
-
Scaling llama.cpp On Neoverse N2: Solving Cross-NUMA Performance Issues
Deep dive into optimizing llama.cpp performance on ARM Neoverse N2 processors, addressing critical NUMA topology challenges for better local inference scaling.
-
GNOME's AI Assistant Newelle Adds llama.cpp Support and Command Execution
The open-source GNOME AI assistant Newelle now integrates directly with llama.cpp for local inference and includes new command execution capabilities for system automation.
-
Running Mistral-7B on Intel NPU Achieves 12.6 Tokens/Second
A developer created a tool to run LLMs on Intel NPUs, achieving 12.6 tokens/second with Mistral-7B while using zero CPU/GPU resources, though integrated GPU still performs better at 23.38 tokens/second.
-
GLM-5 Released: 744B Parameter MoE Model Targeting Complex Tasks
Zhipu AI releases GLM-5, a massive 744B parameter MoE model with 32B active parameters, designed for complex systems engineering and long-horizon agentic tasks with significant performance improvements over GLM-4.5.
-
Arm SME2 Technology Expands CPU Capabilities for On-Device AI
Samsung and Arm announce SME2 technology that significantly enhances CPU performance for local AI inference, potentially reducing reliance on dedicated AI accelerators.
-
NAS System Achieves 18 tok/s with 80B LLM Using Only Integrated Graphics
A community member successfully runs an 80B parameter language model on a NAS system's integrated GPU at 18 tokens per second, demonstrating efficient local inference without discrete graphics cards.
-
Community Member Builds 144GB VRAM Local LLM Powerhouse
A LocalLLaMA community member showcases a custom-built system with 6x RTX 3090 GPUs providing 144GB of VRAM, featuring modified drivers with P2P support for high-performance local LLM inference.