Tagged "model-serving"
16 articles tagged model-serving, 4 March 2026 to 10 October 2026. Newest first.
-
Local LLM Inference at Scale with vLLM
An in-depth technical exploration of vLLM's architecture for achieving high-throughput local inference, covering optimization techniques that enable efficient batch processing and token generation at scale.
-
llama.cpp b10924: Server Router Child State Improvements
The latest llama.cpp build includes critical improvements to the inference server's router and child state handling, enhancing logging reliability and command processing for multi-node inference deployments.
-
Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin
vLLM now supports Tenstorrent hardware through a dedicated plugin, enabling efficient LLM inference on alternative accelerators beyond NVIDIA and AMD. This expands deployment options for self-hosted inference with optimized performance on specialized silicon.
-
vLLM v0.29.0 Advances with Model Runner V2 as Default
vLLM's latest release makes Model Runner V2 the default for all models, featuring CUDA graph memory profiling and improved performance across deployment scenarios.
-
Ask HN: How Are You Operating OSS AI Infrastructure?
Community discussion on practical approaches to running and maintaining open-source AI infrastructure. Direct insights from practitioners deploying LLMs locally.
-
Titan Transients and LLM Scalability
An ACM Queue article examining scalability challenges and solutions for large language models, relevant to understanding infrastructure requirements for local deployment scenarios.
-
Netflix Details Its In-House LLM Serving Platform with Triton and vLLM
Netflix has published details about its production LLM serving infrastructure, combining NVIDIA Triton and vLLM for efficient model deployment. This real-world case study demonstrates battle-tested patterns for scaling LLM inference at enterprise scale.
-
Ollama Just Raised $65 Million to Become AI's Quiet Infrastructure Layer
Ollama secures significant funding to expand its role as a foundational tool for running and managing local LLMs, signaling strong market demand for accessible on-device AI infrastructure.
-
Intel-Scaler-vLLM 0.21.0-b1 Brings Latest Features for vLLM on Intel GPUs
vLLM, the high-performance LLM inference engine, has released version 0.21.0-b1 with optimized support for Intel GPUs. This update enables developers to leverage Intel's discrete graphics for efficient local model serving.
-
SynapseKit: A New Production Framework for Deploying LLMs
Engineers have released SynapseKit, a production-focused LLM framework addressing real-world challenges in deploying language models at scale. The framework aims to solve gaps identified in existing deployment solutions.
-
I Built a Local AI Stack With 5 Docker Containers, and Now I'll Never Pay for ChatGPT Again
Step-by-step guide for containerizing a complete local LLM infrastructure using Docker, eliminating cloud API dependencies while maintaining production-ready deployment patterns.
-
Minisforum Launches N5 Max AI NAS with OpenClaw
Minisforum introduces the N5 Max AI NAS, a specialized hardware device designed to facilitate local LLM deployment and management, targeting organizations building on-device AI infrastructure.
-
Build a Sovereign Local AI Stack: Ollama and Open WebUI and Pgvector 2026
A comprehensive guide to building a complete local AI infrastructure using Ollama for model serving, Open WebUI for the interface, and Pgvector for vector database capabilities. This stack enables fully self-hosted AI applications without cloud dependencies.
-
Build a $1,500 AI Server with DeepSeek-R1 on RTX 4090
Practical guide for assembling and configuring a sub-$1,500 AI inference server using NVIDIA RTX 4090 and DeepSeek-R1, including setup instructions and performance expectations for local deployments.
-
llama-swap Emerges as Superior Alternative to Ollama and LM-Studio
Community members report that llama-swap provides significantly better model switching and multi-model serving compared to established tools like Ollama and LM-Studio. Early adopters highlight breakthrough improvements in model management workflows.
-
RunAnywhere Launches Production-Grade On-Device AI Platform for Enterprise Scale
RunAnywhere has released a production-ready platform designed to deploy and manage AI inference at scale across diverse edge and on-device environments. The platform addresses enterprise requirements for local LLM deployment with infrastructure-level tooling for model management and optimization.