Tagged "gpu"
26 articles tagged gpu, 9 March 2026 to 22 August 2026. Newest first.
-
vLLM's Disaggregated Serving Cuts GPU Interference, Delivering 2.5x Higher Goodput
vLLM introduces disaggregated serving architecture that significantly reduces GPU memory interference, achieving 2.5x improvement in goodput on the same hardware. This breakthrough enables more efficient batch processing and higher throughput for local and self-hosted LLM deployments.
-
llama.cpp b10524 Makes MoE Expert Scatter Deterministic in OpenCL
llama.cpp releases build b10524 with deterministic MoE expert scatter operations in OpenCL backend, improving reliability for Mixture of Experts models on GPU acceleration. This optimization is crucial for consistent inference behavior.
-
Building Local LLM Rigs with Used Server GPUs: 32GB VRAM for €220
Practical guide to sourcing used server-grade GPUs for local LLM inference, achieving 32GB of VRAM at fraction of consumer GPU costs, making large model deployment accessible.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
Deep dive into KV cache management and practical strategies to prevent GPU out-of-memory errors when running local LLMs, a critical bottleneck for on-device inference.
-
The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
A comprehensive guide addressing one of the most critical bottlenecks in local LLM deployment: KV cache memory consumption. Learn practical strategies to manage GPU memory constraints when running LLMs on-device.
-
Squeezing Silicon Limits: Effective Strategies to Eliminate GPU Idle Time and Maximize GPU Utilization
Practical techniques for maximising GPU utilisation during local LLM inference, addressing idle time and throughput bottlenecks that waste expensive compute resources.
-
GPU Half-Idle: The Hundred-Billion-Dollar Race to Squeeze 10x Efficiency from Silicon
An analysis of the hardware and software optimization challenge driving the race for inference efficiency, directly impacting the feasibility of local model deployment.
-
Building a Dual V100 AI Workstation for Local LLMs
A practical guide to constructing a high-performance local LLM inference workstation using dual NVIDIA V100 GPUs, providing both cost-effective and capable hardware for serious local deployment work.
-
Can a 2.8T Model Run on a Single Node of Nvidia B300 X8?
A practical deployment analysis examining whether ultra-large trillion-parameter models can be efficiently served on a single high-end GPU node, providing real-world benchmarks for modern hardware.
-
K3 Model Achieves 20 Tokens/Second on 80x RTX 5090 Cluster
Benchmark results show K3 model inference achieving 20 tokens per second across an 80-GPU RTX 5090 setup, providing insights into scaling strategies for high-throughput local deployments.
-
CPU vs GPU vs NPU: Which Semiconductor Does What?
A technical breakdown comparing CPUs, GPUs, and NPUs (Neural Processing Units) and their respective roles in AI inference. This educational piece helps practitioners understand hardware trade-offs when selecting platforms for local LLM deployment.
-
Nvidia Isn't the Only Choice for Local LLMs Anymore, and AMD Test Proves It
A practical benchmark demonstrates that AMD GPUs are now competitive for running local LLMs, challenging Nvidia's dominance and expanding hardware options for self-hosted inference.
-
AI/ML Benchmark Tool for Local LLM Inference and XGBoost Training
A new benchmarking tool has been released for measuring local LLM inference performance and XGBoost training across GPU and CPU hardware. This resource helps practitioners evaluate their on-device deployment setups and optimize inference performance.
-
$200 NVIDIA V100 Server GPU Mod Beats RTX 3060 in Local LLM Test
A creative hardware modification using refurbished NVIDIA V100 server GPUs demonstrates strong price-to-performance for local LLM inference, outperforming newer consumer-grade GPUs at a fraction of the cost.
-
Intel's $949 GPU Has 32GB of VRAM for Local AI, but the Software Is Why Nvidia Keeps Winning
Intel's new discrete GPU offers compelling hardware specifications for local LLM inference but faces software ecosystem challenges that maintain Nvidia's competitive advantage.
-
Privilege Escalation Attacks on GPUs Using Rowhammer
Security researchers document rowhammer-based privilege escalation vulnerabilities affecting GPUs, raising important security considerations for anyone running sensitive workloads on local GPU infrastructure.
-
Running AI Natively on Windows 11 Using an eGPU
A technical guide demonstrates how to leverage external GPUs for local AI inference on Windows 11, providing affordable hardware acceleration for on-device model deployment. The approach expands options for practitioners with limited built-in GPU resources.
-
GPU Memory for LLM Inference (Part 1)
A detailed technical guide exploring GPU memory optimization strategies for running large language models efficiently during inference, critical knowledge for anyone deploying LLMs locally with limited VRAM.
-
GPUs vs. TPUs: Decoding the Powerhouses of AI
A comprehensive comparison of GPU and TPU architectures for AI workloads, examining trade-offs between general-purpose graphics processors and tensor-optimized units for local and edge LLM deployment scenarios.
-
Intel's $949 GPU Has 32GB of VRAM for Local AI, but Software is Why Nvidia Keeps Winning
Intel's new GPU offers impressive hardware specs with 32GB of VRAM at a competitive price point, yet software ecosystem maturity and optimization remain the deciding factor favoring Nvidia for local LLM deployment.
-
Intel's $949 GPU has 32GB of VRAM for local AI, but the software is why Nvidia keeps winning
Intel's new discrete GPU offers compelling hardware specs for local AI workloads at competitive pricing, but software ecosystem and driver maturity remain critical challenges compared to Nvidia's dominance.
-
Intel Launches Arc Pro B70/B65 with 32GB VRAM for Local AI Inference
Intel has released the Arc Pro B70 and B65 GPUs with 32GB GDDR6 memory at competitive pricing, offering 608 GB/s bandwidth and 290W power consumption. The hardware is positioned as an affordable option for running quantized local LLMs like Qwen 3.5 27B.
-
Powerful AI Search Engine Built on Single GeForce RTX 5090
An enthusiast successfully deployed a fully-featured AI search engine on a single GeForce RTX 5090 GPU, demonstrating the viability of complex local inference workloads on consumer hardware.
-
Repurpose Old GPUs as Dedicated AI Inference Accelerators
An exploration of how older, unused GPUs sitting in drawers can be recycled into effective AI inference hardware, offering compelling performance-per-dollar compared to cloud services or newer hardware purchases.
-
This External GPU Enclosure Tries to Break Cloud Dependence for Local AI Inference
New external GPU enclosure hardware aims to democratize local AI inference by enabling retrofit GPU acceleration for standard PCs. The solution targets users looking to reduce cloud costs and latency for LLM workloads.
-
When Running Ollama on Your PC for Local AI, One Thing Matters More Than Most
An MSN article identifies the critical performance factor for running Ollama efficiently on personal computers. The piece highlights a key optimization principle that practitioners often overlook when deploying local LLMs.