Routing Inference Requests
·
#ai#distributed-systems
Building a cache-aware router, pointing it at real GPUs, and measuring the tradeoff between latency and throughput.
← All thoughts
Building a cache-aware router, pointing it at real GPUs, and measuring the tradeoff between latency and throughput.
Weight streaming, KV caches, and why inference routing is a different problem.
Tracing a request through the system that sits between you and the LLM.