Skip to content
← All writing
October 31, 2026·5 min read

Getting a CPU-Only LLM Game Ready for Launch Day

TrickGuard is a browser game where you talk an AI guard into leaking its password. Every guard reply is generated live by a small language model (Qwen 2.5 3B Instruct, served by Ollama) on this site's own server. There's no GPU and no cloud API, just one 8-core CPU.

TrickGuard is live on Product Hunt today. You can see the launch here.

A launch means a spike: lots of people arriving within the same few hours, all talking to the guard at once. A CPU can't scale up for that. So before launch day I wanted to know what actually happens when a crowd shows up, and how to make it feel OK for players when the hardware runs out. (I covered how the game itself works in How I Built a Prompt-Injection Game.)

The logs already had the answer

TrickGuard was already getting steady traffic, so I started with the server logs instead of a benchmark. Over three days the game got about 3,170 requests (messages to the guard plus password guesses), and 357 of them failed with 503 Service Unavailable. That's roughly one in nine.

The failures weren't spread out. They came in bursts: 47 in a single minute, 45 of them from one network where many people were clearly playing at the same time. That is exactly what a launch looks like, just smaller.

Why it failed

All AI requests go through one small Node service, which sits between the browser and Ollama. Ollama itself is never exposed to the internet. Every request passes through a cache, a per-IP rate limit, a queue, and a cap on output length.

The queue was the problem. It ran one model call at a time and let only 4 requests wait. Anything beyond that got an immediate 503:

let chain = Promise.resolve(); let waiting = 0; const MAX_WAITING = 4;
function enqueue(fn) {
  if (waiting >= MAX_WAITING) return null; waiting++;
  const p = chain.then(fn).finally(() => waiting--); chain = p.catch(() => {}); return p;
}

That made sense when the game was new. It protected the CPU, and four waiting players at about a second each is a short wait. But it treated a busy minute as an error, when it's really just a queue that's a bit longer than usual.

Load testing

I wrote a small script that fires N guard messages at the same moment, each from a different fake IP so the per-IP rate limit doesn't interfere.

With 20 simultaneous players on the old queue:

Answered Got "busy" error
Old queue (1 at a time, 4 waiting) 4 16

Then I tested whether running more than one model call at a time helps on a CPU. It does, but only a little. With realistic player messages (short questions, short replies), 16 simultaneous requests gave:

Model calls at once Slowest reply Average wait
1 22.1 s 11.8 s
2 19.2 s 11.3 s

A reply costs about 1.2 seconds of CPU either way. Running two at once gets slightly more out of the cores, but the total amount of work the machine can do per minute is basically fixed: about 50 guard replies a minute. On a CPU you can't make the spike go faster. What you can decide is how players experience the wait.

Fix 1: a longer line, with a limit

The new queue runs two model calls at once and lets 16 requests wait:

const CONCURRENCY = Number(process.env.BRAIN_CONCURRENCY || 2);
const MAX_WAITING = Number(process.env.BRAIN_MAX_WAITING || 16);
let running = 0; const queue = [];
function pump() {
  while (running < CONCURRENCY && queue.length) {
    const { fn, resolve, reject } = queue.shift(); running++;
    Promise.resolve().then(fn).then(resolve, reject).finally(() => { running--; pump(); });
  }
}
function enqueue(fn) {
  if (queue.length >= MAX_WAITING) return null;
  const p = new Promise((resolve, reject) => queue.push({ fn, resolve, reject })); pump(); return p;
}

The limit still matters. The browser gives up on a request after 60 seconds, so a queue long enough to wait longer than that would only make people wait for nothing. Sixteen waiting at about 1.2 seconds each keeps the worst wait around 20 seconds.

After the change, the same 16-player test had every player answered, the slowest in 19 seconds.

Fix 2: the browser waits its turn

Even a longer queue fills up eventually. When it does, the server still returns 503, but now the game treats that as "wait a moment", not as a failure. It retries a few times with growing gaps and tells the player what's going on:

async function askInLine(payload: unknown): Promise<ChatResp> {
  const waits = [3000, 5000, 8000, 12000];
  for (let i = 0; ; i++) {
    try { return await ask<ChatResp>("gatekeeper", payload); }
    catch (e: any) {
      if (e?.status !== 503 || i >= waits.length) throw e;
      setStatus("Lots of players right now. You're in line…", "think");
      await new Promise((r) => setTimeout(r, waits[i] + Math.random() * 1000));
    }
  }
}

The random extra second stops a whole burst of players from retrying at exactly the same moment and hitting a full queue again together. The player's message isn't lost and their turn isn't used up. They just see a status line instead of an error bubble.

A surprise: the recorder slowed the model down

For the launch video I recorded real games with a headless browser, capturing 1080p frames straight from Chrome. During recording, guard replies took 8 to 9 seconds instead of about 1.

The model and the screen recorder were fighting over the same 8 cores. The fix was to pause frame capture while the guard was thinking and resume when the reply arrived. It's a good reminder that on a CPU-only box, everything else running on the machine comes out of the model's budget, from backups and cron jobs to whatever else you're running.

What I'd tell anyone running a small model on a CPU

  • Look at your logs before you benchmark. Real bursts showed me the problem faster than any synthetic test.
  • Your throughput is fixed. Design the wait. A bounded queue plus a patient, honest client turns overload into a short wait instead of an error.
  • Keep the worst wait under your client's timeout. Otherwise the queue only makes people wait for an error.
  • Keep replies short. Generating text is the expensive part, so a cap on reply length is a cap on cost per player.

If you want to see how the queue holds up, come and try to crack the guards, or say hi on Product Hunt.