Tagged "multi-gpu-inference"

12 articles tagged multi-gpu-inference, 17 February 2026 to 26 September 2026. Newest first.

• Llama.cpp Fork Delivers 2-4x Speedup for Multi-GPU MoE Model Inference 26 September 2026

  A specialized llama.cpp fork optimizes mixture-of-experts models for multi-GPU setups, achieving 2-4x performance improvements for models exceeding single-GPU VRAM limits. This enables practical local deployment of large MoE architectures.

• llama.cpp b10549: Tensor Parallelism Support for LFM2/LFM2MOE Models 21 August 2026

  Latest llama.cpp release enables tensor split for LFM2 and LFM2MOE models, expanding multi-GPU inference capabilities for local deployment.

• Building a Dual V100 AI Workstation for Local LLMs 30 July 2026

  A practical guide to constructing a high-performance local LLM inference workstation using dual NVIDIA V100 GPUs, providing both cost-effective and capable hardware for serious local deployment work.

• K3 Model Achieves 20 Tokens/Second on 80x RTX 5090 Cluster 28 July 2026

  Benchmark results show K3 model inference achieving 20 tokens per second across an 80-GPU RTX 5090 setup, providing insights into scaling strategies for high-throughput local deployments.

• RTX 5080 and RTX 3090 Setup Achieves 80 Tok/s on Qwen 3.6 27B Q8 13 June 2026

  A practical benchmark demonstrating impressive inference throughput using dual NVIDIA GPUs running quantized Qwen 3.6 27B model. This setup showcases real-world performance metrics for local LLM deployment on consumer-grade hardware.

• Pluggable's TBT5-AI: First Thunderbolt Dock Explicitly Targeting Local LLM Workstations 26 March 2026

  Pluggable announces the TBT5-AI, a Thunderbolt 5 dock designed specifically for local LLM inference and GPU-accelerated workloads, addressing connectivity bottlenecks for distributed local inference setups.

• This External GPU Enclosure Tries to Break Cloud Dependence for Local AI Inference 16 March 2026

  New external GPU enclosure hardware aims to democratize local AI inference by enabling retrofit GPU acceleration for standard PCs. The solution targets users looking to reduce cloud costs and latency for LLM workloads.

• Qwen3.5-397B Achieves 282 tok/s on 4x RTX PRO 6000 Blackwell Through Custom CUTLASS Kernel 15 March 2026

  A developer achieved a 5x performance improvement on the massive Qwen3.5-397B model by building a custom CUTLASS kernel to fix SM120's broken MoE GEMM tiles, reaching 282 tokens/second on Blackwell GPUs. This breakthrough demonstrates significant optimization potential for running large models locally with multi-GPU setups.

• Running Qwen3.5-27B Across Multiple GPUs Over LAN Achieves Practical Speed for Local Inference 15 March 2026

  A practitioner successfully split Qwen3.5-27B across a 4070Ti and AMD RX6800 over LAN using llama.cpp's RPC server, achieving 13 tokens/second with 32K context—demonstrating that heterogeneous multi-GPU local setups are now viable. This shows path forward for GPU-poor practitioners seeking reasonable performance.

• Llama.cpp Celebrates Major Milestone: From Leak to Industry Standard 11 March 2026

  The llama.cpp project marks a significant birthday, reflecting its evolution from a hobbyist experiment running leaked models to the foundational inference engine for local LLM deployment.

• Qwen3.5 122B Achieves 25 tok/s on 72GB VRAM Setup 26 February 2026

  Users report exceptional performance running Qwen3.5 122B across three 3090s with 72GB total VRAM, reaching 25 tokens/second with full GPU loading. The model demonstrates strong inference speed and practical viability for enthusiasts with mid-range hardware stacks.

• Qwen3-Next 80B MoE Achieves 39 Tokens/Second on RTX 5070/5060 Ti Dual-GPU Setup 17 February 2026

  A community member has optimised Qwen3-Next 80B mixture-of-experts to run at 39 tokens/second on dual RTX 50-series GPUs with 32GB total VRAM, sharing previously undiscovered configuration solutions for consumer-grade hardware.