Inference at the Edge — Workers AI on 330+ POPs

Section 1

The 800-Millisecond Truth

Interactive AI applications — chat-streaming, tool-using agents, copilots, anything where a human waits — need to respond in under ~800 ms total round-trip for the interaction to feel natural. Slower than that and users start to disengage. That single threshold drives most of the edge-AI story.

Here's Cloudflare's own latency budget for the most demanding case — a real-time voice pipeline (Realtime Voice AI blog, Aug 2025):

Phase	Budget
Microphone input	40 ms
Speech-to-Text	300 ms
LLM inference	400 ms
Text-to-Speech	150 ms
Total	890 ms

What that looks like elsewhere

The honest comparison — a customer in Munich, same kind of voice or interactive stack, three different providers. Source: artificialanalysis.ai GPT-4o provider benchmarks (Nov 2025) plus typical EU → US RTTs.

Stack (Munich user)	End-to-end	Where the time goes
Workers AI · 200+ cities	~890 ms	Inference happens in your nearest GPU city
Azure OpenAI · EU region	~1.3–1.8 s	+1.54 s GPT-4o TTFT (artificialanalysis.ai)
Azure OpenAI · US region	~2.2–2.8 s	+180 ms RTT Frankfurt ⇄ Virginia, per round-trip
OpenAI Realtime API	0.5 s tuned · 5–6 s extended	Self-reported, varies with session length

Apples-to-apples is hard — we measure first audio chunk, others first token text. But the two big buckets are the same everywhere: ~400–600 ms network penalty when inference is far away, plus ~1 s of provider-side inference overhead.

Three forces that shape every edge-AI conversation

Latency is the headline, but it's never alone. Every customer conversation eventually runs into one of three constraints. Keep them in your back pocket.

Latency

Light is finite. Sub-second interaction can't happen 4,000 km from the user.

Sovereignty

EU AI Act, GDPR, sectoral compliance. Region matters, not just region count. Customers ask where the model runs, not just whose it is.

Economics

Streaming raw video or 24/7 LLM traffic to a central cloud is a budget problem, not a tech problem.

Section 2 · LIVE

See It Work — Five Models, One Binding

Eight lines of code, one binding, real GPUs. This page's Worker just found a GPU in the nearest enabled POP.

export default {
  async fetch(req, env) {
    const result = await env.AI.run(
      '@cf/meta/llama-3.1-8b-instruct',
      { prompt: 'Summarize: ...' }
    );
    return Response.json(result);
  }
};

About the engines — Workers AI runs three inference engines, picked per model. Infire is Cloudflare's own Rust engine, optimized for the edge constraints in §4; today it powers Llama 3.1 8B. The rest of the catalog runs on vLLM (open source, broad model coverage) or a Python custom engine for special-case workloads. The engine choice is a Workers AI implementation detail — customers see one API.

Model Guardrails

Try:

Pre-recorded benchmark — same haiku prompt, five models

Captured live, AMS-DOG, June 2026. The three big reasoning models think out loud before answering — that's the model, not a bug.

Model	Engine	First token	Throughput	Behaviour
Llama 3.1 8B	Infire	~2.1 s	159 tok/s	Direct answer, fast throughput
Llama 4 Scout	vLLM	373 ms	40 tok/s	Direct, MoE architecture
GPT-OSS-120B	vLLM	350 ms	140 tok/s	Reasoning — thinks aloud, then answers
Kimi K2.6	vLLM	1.6 s	80 tok/s	Reasoning
GLM 4.7 Flash	vLLM	254 ms	65 tok/s	Reasoning, multilingual

Catalog also runs Flux (image gen) and Whisper (audio transcription). Different UI than text streaming, same one-binding pattern.

Section 3

Footprint & Competition

The demo just worked because Cloudflare placed GPU capacity inside a selected subset of its global cities. The exact GPU-city count isn't publicly disclosed; the official line is "200+ cities". The remaining cities still handle TLS, caching, WAF, and routing — they just hand off to a GPU-capable neighbour over the backbone.

337+

Data center cities

cloudflare.com/network, June 2026

200+

GPU-enabled cities

Workers AI inference

125+

Countries

across 8 backbone regions

How we compare — June 2026

Customers will compare. Here's the honest one-line read on each — useful when the conversation drifts to "why not just Bedrock?"

Stack	Footprint	Wins on	Loses on
Workers AI	200+ GPU cities	Latency, dev experience, AI Gateway as door	Model breadth vs. Bedrock, enterprise procurement maturity
AWS Bedrock	30 regions	Model breadth (102+ models, 17 publishers), enterprise contracts	Latency to non-US users, no edge story
Azure OpenAI / Foundry	9 EU Data Zone countries	GPT-5 family, Microsoft co-sell, now multi-model (Claude, Llama, Mistral)	Regional, no edge inference
Akamai Inference Cloud	~20 GPU cities · AI-Grid routing over 4,400 edge locations	NVIDIA Blackwell partnership, marketing reach	GPU footprint <½ of Cloudflare; no in-product gateway
Groq Cloud	4 regions · LPU hardware	Extreme throughput on supported models	Concentrated, not edge-distributed; narrow model catalog
Fastly Compute@Edge	Wasm-based, no GPU	Lightweight edge logic	No GPU inference at all — AI Gateway-pattern only

The right customer answer is often "both" via AI Gateway — we'll come back to that in §5.

Section 4

Inside One Inference Call

We saw it work in §2 and saw the footprint in §3. Now the engineering tour, at one altitude: what actually happens between env.AI.run() and the first token streaming back.

Step 1 · Request lands at a GPU city

Anycast routes to the closest POP. TLS, KV, WAF, V8-isolate boot — usual edge work, no cold boot. Frontline (FL) sees the AI binding, picks the nearest GPU-capable city, and forwards over Cloudflare's private backbone.

Step 2 · Omni schedules the model

On the GPU metal, Cloudflare's scheduler Omni makes one physical GPU look like many — by over-committing VRAM and spilling inactive weights to host RAM.

400%

over-commit, in production

80 GB physical → ~320 GB effective · 13 models co-resident on one H100

Effective capacity

~320 GB · CUDA Unified Memory + host RAM spill

Llama 3.1 8B

Llama 4 Scout

GPT-OSS-120B

Mistral

Kimi K2.6

GLM 4.7

SDXL

headroom

080160240320 GB

Physical VRAM

80 GB · one H100. The hard ceiling that isn't.

80 GB ceiling

080160240320 GB

HOT · in VRAM, ready WARM · recently used, partial cache COLD · spilled to host RAM swapping right now

Kimi K2.6 alone is bigger than the physical card — and Omni still hosts it alongside twelve others by streaming inactive weights between VRAM and host RAM. PCIe 4.0 moves 32 GB/s, so a 5 GB swap takes ~156 ms.

How the swap actually plays out

Three models (A, B, C). A+B fit together in VRAM, C alone takes the whole card. Block colour shows each model's state, not its identity — same legend as the VRAM bar above.

HOT in VRAM, ready · ~100 ms inference WARM weights fresh in host RAM · ~156 ms swap-in COLD spilled long ago · seconds to re-load

① Idle

GPU · 80 GB

A

B

Host RAM

C

A, B HOT · C COLD

→

② Request for C

GPU · 80 GB

C

Host RAM

A

B

C HOT · A, B WARM (fresh in host)

→

③ Request for B

GPU · 80 GB

B

Host RAM

A

C

B HOT · A, C WARM. Hot models stay hot.

The reason this matters for SE conversations: the swap penalty is bounded. A re-warm from host RAM is ~156 ms, not seconds. "Truly cold" (weights gone, R2 pull + JIT compile) is rare in production — catalog models get ambient traffic from other tenants that keeps them at least WARM. P99 conversations are about WARM swaps, not COLD starts.

Step 3 · The engine runs the model

Workers AI runs three inference engines, picked per model. vLLM serves most of the catalog (open source, broad model coverage). Python-custom handles special cases. Infire is Cloudflare's own Rust engine — today only Llama 3.1 8B. More models will move to Infire over time; the rest stay on vLLM. Customer sees one API.

+7 %

Infire vs. vLLM 0.10 on the same H100 NVL

25 %

CPU load (Infire) vs. 140% (vLLM) — same workload, ShareGPT v3 benchmark

< 4 s

Infire cold start for Llama 3.1 8B (16 GB BF16)

OpenAI

Both engines speak the OpenAI Chat Completions API

Why two engines? vLLM is excellent — for hyperscaler datacenters. At the edge, its CPU contention (140% — has to run inside gVisor sandbox), Python interpreter overhead, and memory-layout assumptions fight Omni's over-commit trick. Infire is the Rust answer for workloads where those constraints bite hardest. Llama 3.1 8B was the first; more will follow.

Section 5

The Other Door: AI Gateway

The official theme of this teach-to-lead names two products, not one: Workers AI and AI Gateway. AI Gateway has been GA since May 2024 — and it's what gets adopted in the easiest first customer conversation, even when the customer is locked into OpenAI direct.

What it is in one sentence

AI Gateway

A reverse proxy between your customer's app and OpenAI / Anthropic / Google / Workers AI. They change one URL; you give them logging, caching, rate-limiting, cost reports, and BYOK in week one.

What lands in week one

Full request & response logs for every model call — the thing every CFO and CISO asks for the second they hear "AI".
Caching of identical prompts. Repeat questions = zero spend. For RAG-heavy workloads, this alone justifies the migration.
Rate-limiting & spending caps per user, per key, per app. No more "intern wrote a loop, $40k bill on Monday".
Stripe-backed cost reporting & Unified Billing. OpenAI & Anthropic spend on the Cloudflare invoice. Procurement loves one vendor, one PO. Unified Billing for third-party providers is new in 2026.
BYOK — bring your own OpenAI / Anthropic key. No migration required. Gateway is the on-ramp, not the off-ramp.
Dynamic routing & fallback between providers. Primary on GPT-5, fallback to Claude on rate-limit. One config, no code change.

The SE pitch in three sentences

01

"You're going to want observability on your AI spend before the next budget cycle." Open with cost, not latency. CFOs sign these.
02

"Change one URL. Keep your model, keep your keys." Lowest possible friction. Engineering buys in because there's nothing to re-architect.
03

"Once you're on Gateway, switching models gets cheap." The migration ramp to Workers AI is built in. Gateway sells it over time.

Section 6

Quick Check — Three Questions

Three questions on what we just covered. Click an answer for the reasoning. No score, no leaderboard — just a self-check.

01

A customer in Munich wants sub-second voice or interactive AI. Which stack realistically hits the ~800 ms budget?

02

Llama 3.1 8B runs on Infire — the rest of the catalog (Llama 4, GPT-OSS-120B, Kimi K2.6, GLM) runs on vLLM. Why two engines?

03

A customer is committed to OpenAI direct and isn't switching. What's the easiest first conversation?

Right. The 800 ms budget is a network problem, not a model problem. Workers AI in 200+ GPU cities lands you ~890 ms total. Azure US adds ~180 ms RTT round-trip Frankfurt ⇄ Virginia before the model even starts. (§1)

Optimistic. EU ⇄ US round-trip alone is ~180 ms — half your budget gone on network before the GPU does anything. Azure US end-to-end lands at 2.2–2.8 s for a Munich user. (§1 compare table)

Wrong. Where the model runs decides whether you hit 800 ms. Inference 4,000 km away can't be made fast by good code. (§1)

Right. vLLM is excellent — for hyperscaler datacenters. At the edge, its CPU contention, Python overhead, and memory layout fight Omni's over-commit trick. Infire exists for the workloads where those constraints bite; the customer sees one API. (§2, §4 Step 3)

Wrong. Infire is +7% faster than vLLM 0.10 on the same hardware — and uses a fraction of the CPU because it runs bare-metal instead of inside gVisor. The engine choice is per-model. (§4 Step 3)

Wrong. Infire is written in Rust but serves the same standard catalog models (Llama 3.1 8B today). Model format and serving engine are separate. (§4 Step 3)

Right. Gateway in front of existing OpenAI calls gives observability + caching + Stripe cost reports + BYOK in week one. Lowest friction, highest value, GA since May 2024. (§5)

Risky. The customer just told you they're not switching. Latency pitch = re-architecture conversation. Lead with Gateway instead — it lets them stay on OpenAI and get Cloudflare value. (§5)

Off-topic. They asked about staying on OpenAI, not about your footprint. Footprint is a §3 argument; AI Gateway is the §5 answer to lock-in scenarios. (§5)