Customed version of Google's tflite-micro
-
Updated
Sep 17, 2025 - C++
#
inference-embedded-engine
Here are 2 public repositories matching this topic...
High-performance LLM inference platform with vLLM continuous batching achieving 12.3K+ req/sec, 42ms P50/178ms P99 latency, INT8/INT4 quantization (70% memory savings), tensor parallelism across 4 GPUs, and comprehensive monitoring serving 1500+ concurrent users.
-
Updated
Sep 28, 2025 - Python
Improve this page
Add a description, image, and links to the inference-embedded-engine topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the inference-embedded-engine topic, visit your repo's landing page and select "manage topics."