Qorinix inference control plane

The board wanted "instant" AI responses for real-time agents, voice interfaces and trading alerts, and owned the decision on whether Qorinix would become a product rather than stay an internal capability. The end users are developers who call one API and expect a fast first token no matter which provider serves the request.

A hard latency budget (targeting sub-200ms time-to-first-token at p50), provider cost that had to be visible per call rather than discovered on the monthly invoice, and automatic failover when a provider degrades or rate-limits. A five-person team, so the design had to be operable without a dedicated platform crew.

A multi-tier router that classifies each request by latency sensitivity and routes it across 12 providers (OpenAI, Anthropic, Gemini, DeepSeek, Qwen, Groq, Cerebras and others), with streaming SSE end to end, a per-call cost ledger, usage-based billing and entitlements. Token tiering sends non-latency-critical work to cheaper or self-hosted models. I wrote the specification and the evaluation criteria; AI coding agents generated the code under a three-gate review and evaluation process, and I reviewed, tested and accepted every change.

• 12 LLM providers behind one API
• Groq and Cerebras LPU inference for the low-latency tier
• Streaming SSE
• Per-call cost ledger and usage-based billing
• Entitlements and API keys
• AWS and Cloudflare

Every call is logged with provider, model, latency and cost. Provider failover follows a deterministic fallback order. Model and provider changes go through the same three-gate review as feature code. I am the escalation point for production incidents and own the rollback path.

Measured by time-to-first-token p50 and p95 per provider in the LLM Arena benchmark that runs on this site, plus cost per thousand tokens per tier. The control plane targets sub-200ms TTFT (p50) and 100-200+ tokens per second; token tiering reduced API spend for non-latency-critical work. Customer and revenue figures are not published here; I walk through the benchmark numbers on a call.