Tagged "inference-engines"
7 articles tagged inference-engines, 10 March 2026 to 26 September 2026. Newest first.
-
Oh My Pi Adds Custom Model Support via vLLM, Llama.cpp, and SGLang
A new guide demonstrates running custom quantized models on Raspberry Pi using multiple inference engines including vLLM, Llama.cpp, and SGLang. This enables practical multi-engine inference workflows on edge devices with detailed configuration examples.
-
Major Smartphone Brands Introduce Advanced On-Device AI Features
Leading smartphone manufacturers are rolling out sophisticated on-device AI capabilities, signaling broad industry momentum toward local model inference on mobile hardware.
-
Warp Decode vs. vLLM's Triton Kernel: Performance Crossover Analysis
A detailed technical comparison analyzing where Warp Decode and vLLM's Triton kernel each excel for local LLM inference, with implications for choosing the right decoding strategy for your hardware.
-
Ultra-Large 400B-Class LLM Runs on iPhone in Test
A 400B-parameter language model has been successfully demonstrated running on an iPhone, marking a significant breakthrough in on-device inference capabilities. This achievement suggests that ultra-large models can now fit and execute on consumer mobile devices through advanced optimization techniques.
-
OmniCoder v2 Released: Improved Code Generation for Local Deployment
OmniCoder-v2 has been released with notable improvements over the previous version, available as a 9B GGUF quantised model for efficient local inference and code generation tasks.
-
Community Converges on Optimal KV Cache Quantization Strategies for Qwen 3.5 Models
The local LLM community is establishing practical guidelines for KV cache quantization with Qwen 3.5, balancing memory savings against accuracy loss to optimize inference on consumer hardware.
-
Qwen 3.5 Ultra-Compact Models Enable On-Device AI from Watches to Gaming
The latest Qwen 3.5 lineup, including the 0.8B variant, demonstrates that state-of-the-art small language models can now run on severely constrained devices while maintaining impressive capabilities, from vision tasks to game-playing agents.