Tagged "gpu-utilization"
10 articles tagged gpu-utilization, 26 February 2026 to 7 October 2026. Newest first.
-
GridCore: A GPU Scheduler for Running Multiple Local LLMs on One GPU
GridCore is a new scheduler that enables multiple local LLMs to share a single GPU efficiently. This tool addresses a critical constraint for local deployment: allowing developers to run multiple models simultaneously without requiring multiple GPUs.
-
VRAM Optimization Breakthrough: Single Setting Change Doubles Local Model Speed
A practical discovery reveals that a single configuration change can double inference speed on local AI models by eliminating wasteful VRAM usage, offering immediate performance gains for existing deployments.
-
llama.cpp Build 10605: Mamba2 GEMM Optimization Improves State-Space Model Performance
The latest llama.cpp release optimizes Mamba2 models by flattening input/output projections to dispatch GEMM operations instead of GEMV, delivering better GPU utilization and inference speed for state-space architectures.
-
Squeezing Silicon Limits: Effective Strategies to Eliminate GPU Idle Time and Maximize GPU Utilization
Practical techniques for maximising GPU utilisation during local LLM inference, addressing idle time and throughput bottlenecks that waste expensive compute resources.
-
Prefill Is Compute-Bound, Decode Is Memory-Bound: Optimizing GPU Utilization for LLM Inference
A deep dive into why GPUs shouldn't handle both prefill and decode phases equally, and how understanding this fundamental bottleneck can dramatically improve local LLM inference performance.
-
NVIDIA and Google Optimize Gemma 4 AI Models for Local RTX Deployment
NVIDIA and Google have collaborated to optimize Gemma 4 models specifically for NVIDIA RTX GPUs, enabling high-performance local inference. The optimization work ensures efficient utilization of consumer and professional GPUs for on-device AI workloads.
-
GPU Passthrough to LXCs in Proxmox Simplifies Local LLM Deployment
GPU passthrough to Linux containers in Proxmox offers superior performance and simplicity compared to virtual machines for running local LLMs, enabling efficient on-device inference without virtualization overhead.
-
Cutile.jl Brings Nvidia CUDA Tile-Based Programming to Julia
Cutile.jl enables tile-based CUDA programming in Julia, offering improved GPU utilization and performance optimization capabilities for compute-intensive workloads including LLM inference.
-
Llama.cpp Prompt Processing Optimization: Ubatch Size Configuration Guide
A community member shares practical troubleshooting advice for improving prompt processing performance on larger models like Qwen 27B by configuring ubatch size parameters in llama.cpp.
-
DeepSeek Releases DualPath: Addressing Storage Bandwidth Bottlenecks in Agentic Inference
A new paper from DeepSeek, Peking University, and Tsinghua University presents DualPath, a technique for breaking storage bandwidth limitations in agent-based LLM inference. The research tackles a fundamental performance constraint affecting local deployment at scale.