Tagged "real-time-inference"
22 articles tagged real-time-inference, 20 February 2026 to 24 July 2026. Newest first.
-
Nota AI Joins AMD Robotics Partner Network to Expand On-Device AI Optimisation
Nota AI's partnership with AMD's robotics network will accelerate development of optimised on-device AI solutions for physical AI applications, extending model compression and inference optimisation technology into the robotics sector. This collaboration targets real-time inference constraints critical for autonomous systems.
-
Tiny microphone on my balcony to listen for any birds passing by
A practical demonstration of edge AI inference using miniature audio hardware and local ML models for real-time bird species identification without cloud connectivity.
-
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
A breakthrough in LLM inference optimization achieves 3,000 tokens per second on standard GPUs, significantly improving real-time inference performance for local deployments.
-
Privacy-Focused Raspberry Pi Zero 2W DIY Security Camera with On-Device AI and End-to-End Encryption
A new Raspberry Pi Zero 2W-based security camera project demonstrates practical on-device AI inference with end-to-end encryption, showcasing edge deployment on ultra-low-power hardware.
-
New 8B Local LLM Design Marks Biggest Shift Since DeepSeek R1
A new 8-billion parameter local language model introduces significant architectural innovations that could reshape how efficiently local LLMs are designed and deployed. This development represents a major evolution in the efficiency-to-capability tradeoff for on-device inference.
-
Hardware LLM Taalas Reaches >14,000 TPS on Llama 3.1 8B
Taalas demonstrates breakthrough throughput of over 14,000 tokens per second on Llama 3.1 8B, showcasing specialized hardware acceleration for local and edge LLM deployment.
-
Nvidia Raises Video Encoder Limit to 12 on Consumer GPUs
Nvidia increases the concurrent video encoding capacity on consumer GPUs from previous limitations to 12 encoders, enabling new possibilities for multimodal LLM applications and real-time inference pipelines.
-
Kog AI – Building a Real-Time Inference Stack on AMD Instinct GPUs
A technical presentation on building production inference systems using AMD Instinct GPUs, expanding the hardware ecosystem for local LLM deployment beyond NVIDIA dominance. The talk covers real-time inference optimization techniques applicable to on-device deployments.
-
DFlash Speculative Decoding Delivers 8.5x Speed Improvement for LLM Inference
A new speculative decoding technique achieves dramatic speedups in local LLM inference without sacrificing output quality. This optimization is particularly impactful for latency-sensitive applications and resource-constrained deployments.
-
What Type of AI Usage? Deployment Patterns and Implementation Considerations
A framework for categorizing different AI implementation patterns, helping developers choose appropriate architectures for local versus cloud deployment.
-
Waterloo's Live AI-Goose Tracker: Real-Time Edge Vision
An innovative real-time computer vision project using local AI to track geese across Waterloo, Ontario, demonstrating practical edge inference for public safety and wildlife monitoring.
-
LlaMa.cpp Robot Wars
A creative demonstration of llama.cpp being used to power autonomous robot decision-making and strategy in a competitive robotics setting.
-
115 TOPS in 0.67L: CHUWI AuBox X Packs On-Device AI Power Into a Palm-Sized Mini PC
CHUWI releases the AuBox X, an ultra-compact mini PC delivering 115 TOPS of compute in just 0.67 liters, making it an attractive form factor for edge LLM deployment. This hardware advance pushes the boundaries of portable on-device inference.
-
DFlash Doubles Token Generation Speed of Qwen3.5 27B on Mac M5 Max
New DFlash support in oMLX 0.3.5 RC1 achieves 2x speedup for Qwen3.5 27B inference on Apple Silicon, reaching 22 T/S from 9 T/S using speculative decoding with draft models.
-
Google Gemma 4 Delivers Exceptional Speed and Accuracy for Local Inference
Early adopters report that Google's Gemma 4 model runs with remarkable speed comparable to 4-9B parameter models while maintaining accuracy levels reminiscent of early Gemini releases, making it a compelling option for resource-constrained local deployments.
-
DFlash Speculative Decoding Achieves 3.3x Speedup on Apple Silicon
A native MLX implementation of DFlash speculative decoding reaches 85 tokens/second on Qwen 3.5-9B running on Apple M5 Max, delivering a 3.3x performance boost through parallel draft token generation and single-pass verification.
-
CarryAI's Serverless Vision-Language Models Enable On-Device Multimodal AI
CarryAI has introduced serverless vision-language models optimized for on-device deployment, signaling a new era where multimodal AI can run efficiently on edge hardware without cloud dependencies.
-
Lenovo Korea Launches AI-Powered Industrial Edge Solutions
Lenovo Korea has introduced artificial intelligence-based industrial edge solutions targeting manufacturing and enterprise environments. The products enable real-time AI inference at the edge without cloud connectivity dependencies.
-
A Journey to a Reliable and Enjoyable Locally Hosted Voice Assistant
An in-depth guide documenting the development and deployment of a fully local voice assistant, covering the complete stack from speech recognition to language understanding and synthesis without cloud dependencies.
-
AI-Native Store Research
An exploration of how AI is being integrated into retail environments, including potential applications of local LLM deployment for edge-based customer interaction and inventory management systems.
-
Arduino, Qualcomm Bring On-Device AI and Robotics Learning to Indian School Systems
Arduino and Qualcomm partner to integrate on-device AI and robotics education into Indian schools, democratizing access to edge ML training and embedded systems development.
-
The Path to Ubiquitous AI (17k tokens/sec)
A technical analysis of achieving 17,000 tokens per second inference throughput, demonstrating the performance milestones required for truly practical local LLM deployment at scale.