Tagged "inference-server"
5 articles tagged inference-server, 1 May 2026 to 9 October 2026. Newest first.
-
llama.cpp B11521: Server Infrastructure Improvements and Performance Optimisations
Latest llama.cpp release includes default port configuration changes and continued optimisations across CUDA, Metal, and other acceleration backends.
-
HackerNoon Compares 7 Best Self-Hosted Inference Servers for Open-Source Models
A comprehensive 2026 comparison of leading self-hosted inference servers evaluates deployment options for running open-source models locally, covering performance, ease of use, and feature parity across major frameworks.
-
7 Best Self-Hosted Inference Servers for Open-Source Models Compared (2026)
Comprehensive comparison of leading self-hosted inference server solutions, evaluating performance, features, and deployment characteristics for local LLM inference.
-
Llama.cpp Adds LRU Scheduler for Multi-Model Serving
Llama.cpp B10313 introduces an LRU (Least Recently Used) scheduler for its router, enabling better resource management when serving multiple models simultaneously. This enhancement improves request handling and model eviction policies for local inference servers.
-
How to Make SSE Token Streams Resumable, Cancellable, and Multi-Device
A practical guide to improving server-sent event (SSE) token streaming for LLM inference, enabling better user experiences with resumable downloads and multi-device support in local deployments.