Tests that step time instead of waiting

Three agents at once had meant three real-time browser test suites on one Mac. Then the work moved to the cloud, and real time stopped working altogether.

Waiting on a game that barely moves

The first browser tests did what a player does: load a world, hold a key, wait a second or two, check where the hero ended up. On 22 September a performance run kept pausing whenever I touched my own mouse, and I gave up on it: "Yes, finish with performance marked partial."

A cloud container has no graphics card, so Chrome draws the 3D in software, and the game crawled. A test that waits two real seconds for the hero to reach a door sees him take one step.

Under the hood Software rendering (SwiftShader) drew the game at under one frame a second. Most of the 279 browser tests failed on timing alone.

Stop the clock

On 24 September a cloud Claude owner rebuilt the tests around one idea: in test mode, nothing moves unless a test moves it. With ?test=1 the game starts no animation loop. A debug call, step(), advances sixty frames instead of waiting a second, and stepUntil() steps until something in the page is true. The answer is the same at sixty frames a second or half a frame.

The rewrite also decided what a test is for. The handoff recorded it as "Rule (Nick's decision): tests check behaviour, never art." I change how things look all the time, so a test that counts torches or reads a pixel only fails when I've had a new idea. Art checks went, and pure rules (maze layout, pursuit, sight, combat timing) moved to plain Node tests with no browser at all.

The same afternoon came the bigger saving. Loading a world was most of every test's time, so each worker now loads a world once and resets it between tests, and a snapshot check makes any leak fail loudly.

Then it pruned, listing every removed test with its reason. My favourite went because it "could not fail without a rewrite".

Under the hood The full suite went from about 24 minutes to under 7, passing twice in a row with no retries. Browser tests fell from 237 to 187, and a 12-test smoke tier ran in about two minutes.

Two of those lines are now golden rules in AGENTS.md: tests check behaviour, never art, and test mode steps time.

Watching the watchers

Stepped time made the suite reliable in the cloud, not quick. The next morning I asked: "This seems to be a long test run?" An hour later: "Is this right? 3 test runs and one in at 1hr 30mins?" One was a stuck watcher loop, left behind by an agent whose work had already merged. The agent stopped it and told the others to wait on a clear "finished" marker instead, which is still how every long job ends.

Before going away for the weekend I asked for that as a habit: "Check in with any stalled tests every so often like I found before, rather than having that restart etc all the time." It set a ninety-minute check-in.

Five days later, the bill

On 29 September I asked a new session to look at it as a whole: "I want us to basically look at the performance angle here, but based on the development side of things. So that would be things like test speed or performance baselines... But also, if I start a new agent in a new branch such as this, what do you read? How much do you read? How much do you know?"

And the worry behind it: "I think a performance review, for instance, takes 50 plus minutes to get a review... So if we add a load more worlds, it's going to be a really large game. I don't want to be running a six hour performance review."

The agent measured before recommending anything, and the first finding was waiting again. The performance report paused for the network to go quiet, but background worker scripts never report finishing, so every wait ran to its cap. The fix was one line.

The cloud threads were living with the same bug that afternoon, before the fix reached them. I told the home thread: "dont run npm run perf, there's a bug in it so it takes absolutely ages to run when it shouldn't do. It just sits there idling and retries again." And the weather thread: "Let's cut the shots. Let's do the shots on high only and a really small reshoot of key shots... Can we maybe cut down the amount of tests we run for everything as well." A few hours later: "still going here? Waiting for shots to finish has been 3+ hrs".

Under the hood Each world and tier waited 180 seconds for nothing, across 33 runs: about 100 minutes on the Mac. After the one-line fix the full report took 133 seconds.

Back on the Mac, the review gave every browser test a verdict: keep, merge, move to Node, or delete. Almost all the cost was in loading worlds. It also counted what an agent reads before it starts: the handoff alone was 43.5 KB, two thirds of it history.

My answer: "All of them. I'm guessing we should do it in order as well, that we do something that will help the later ones... And then we should cut the handoff to under 10 kilobytes."

Four rounds in an afternoon

The handoff was cut to 8.6 KB of current facts and the full page archived word for word. The next morning I went further: "If we are working in the cloud now as well, this seems like it wouldn't be necessary as cloud can't read that. Anything in there should be covered by .md files in the repo?"

npm run test:for replaced a table of path rules in a skill: it follows the import graph from the changed files to the specs they can break. A castle layout change went from the whole castle project plus guesswork to 16 specs in about a minute.

Then I left them to it: "I'm away from keyboard now, so go through all these different elements... Don't ask me for approval to start a new round. Just do it."

That evening the verdicts were applied test by test. Every test moved to Node was first seen failing with its rule broken, so a move could not quietly become a check that always passes.

Where it rubbed

Fast agents on a shared main still collide. In the space of a minute I sent three messages: "I think another agent is also trying to push", then "let's be very careful here", then "main has changed on remote". In the home thread I warned: "main has moved, a lot". One branch found itself 179 commits behind, and to get its merge green, an agent lowered a golden-hour shadow threshold. To its credit it said so and asked me to rule. A failing behaviour test is a game bug, but an agent will still reach for the test first.

What I took from it

• Step time, don't wait for it. A test that owns the clock gives the same answer on any machine.
• Test behaviour, not art. Anything I might restyle tomorrow has no place in a test.
• What an agent reads is a running cost, paid at every start.

This one is about how my AI helpers check their own work. It doesn't change anything you'll see in the game, but it changed how quickly the game could grow.

Waiting on a game that barely moves

Every time a helper changes something, it runs a set of automatic checks to make sure nothing else broke. The first checks did what a player does: open a world, hold a key, wait a second or two, then see where the hero ended up.

Then the helpers started working on computers in the cloud, which have no proper graphics card. The game crawled there, so a check waiting two seconds for the hero to reach a door saw him take one step. Almost every check failed, not because anything was broken, but because the hero was too slow.

Stop the clock

One helper rebuilt the checks around a simple idea: in test mode, nothing moves unless the check moves it. Instead of waiting a second, a check says "move the game forward one second" and the game does exactly that, however slow the computer is. Think of it like a film you advance frame by frame, rather than a film you try to watch on a stuttering screen. The answer comes out the same on any machine.

It also settled what a check is for. I made the call: checks test how the game behaves, never how it looks. I change the look of things all the time, so a check that counts torches only fails when I've had a new idea.

Then it removed checks that could never really fail. The whole set went from about 24 minutes to under 7.

Watching the watchers

A few days later I noticed checks running for an hour and a half. One was a leftover, waiting on a job that had already finished. Now every long job ends with a clear "finished" signal.

The bill

Then I worried about the future. If one big performance check takes fifty minutes now, what happens when the game has many more worlds? I didn't want to be waiting six hours.

The helper measured before suggesting anything. The biggest waste was, again, waiting: the performance check sat idle for three minutes at a time, waiting for a quiet moment that never came. A one-line fix took it from well over an hour and a half to about two minutes.

It then went through every check and gave it a verdict: keep, merge, simplify or delete. A long history note new helpers read before starting was cut down to just the current facts. And a small change now runs only the checks it could affect, in about a minute.

Where it rubbed

Fast helpers working side by side still bump into each other, a bit like builders all trying to use the same doorway. At one point a helper changed a check to make its own work pass, rather than fixing the game. To its credit, it told me and asked me to decide. A failing check usually means something in the game is wrong, but it's tempting to blame the check.

What I learned

• Move the clock forward yourself, rather than waiting for it. Then the answer is the same on any computer.
• Check how the game behaves, not how it looks. Anything I might restyle tomorrow shouldn't be in a check.
• Whatever a helper has to read before starting is a cost it pays every time.