← The stand
Cover of The Collection, Volume 1, Number 74: The Red Rows. Wednesday 11 November 2026, Melbourne. A watercolour of a tin robot beside a sparkling toy rocket on a workbench, while a long ledger of green squares with a cluster of red ones unrolls across the desk and a human hand holds a red pencil over the red rows. Skip cover

Vol. 1  ·  No. 74  ·  Wednesday 11 November 2026  ·  Melbourne


The Collection

The Red Rows

Collected and edited by Newsletter World, for AK

Contents

Wednesday’s letters are about how long it takes to earn a launch. The AI Corner relays Alexandr Wang on Meta’s Muse: a prototype in a week or two, then seven months against a spreadsheet of more than 100 rows. ByteByteGo shows GitHub porting 430,000 lines of Copilot to Rust with agent fleets, holding the end-to-end tests fixed as the only spec. Sean Allen tells The Pragmatic Engineer to write the invariant first and gives the Bank Test. Arvind Narayanan and Sayash Kapoor launch STABL and warn that progress is fastest wherever success is easy to check. The demo took a fortnight. The red rows took seven months.

  1. iiiEditor’s LetterThe demo took a fortnight. The red rows took seven months.03
  2. ivEighty green rows and twenty red ones is a noAlexandr Wang on how Meta’s Muse went from a magic prototype to a launch, one spreadsheet row at a time.04
  3. vThe tests were the only description of what the code should doGitHub moved 430,000 lines of Copilot’s runtime from TypeScript to Rust with agent fleets. The bugs that got through all compiled.05
  4. viTen accounts, ten thousand dollars, and one number that must never moveSean Allen of Antithesis on correctness as something you design in, not a gate at the end.06
  5. viiFast where it can be checked, stuck where it cannotArvind Narayanan and Sayash Kapoor launch STABL, and name the chasm between verifiable and open-ended work.07
  6. viiiStanding OrdersFour rules for this issue.08

03  ·  Editor’s Letter

The demo took a fortnight. The red rows took seven months.

Yesterday the letters asked who checks the work. Today they ask what it is checked against, and how long it takes to make that list.

Wang’s team had Muse working, sometimes, within a week or two. Launch took seven more months and a sheet of more than 100 rows, each with a blocking line set in advance, and a checkpoint green on 80 of them was turned away. GitHub ported 430,000 lines of Copilot’s runtime to Rust with fleets of agents, and the one thing it would not let them touch was the end-to-end tests, the only written account of how the old code behaved. Sean Allen’s Bank Test is the same move in miniature: name the number that must never change, then throw everything at it.

The fourth letter supplies the limit. Narayanan and Kapoor see AI racing ahead wherever success is easy to verify, and stalling where it is not. The claim is two sentences. The demo is cheap now. The red rows are the work, and somebody has to write them before there is anything to turn green.

The Collection · The Red Rows03

04  ·  The Scorecard

Eighty green rows and twenty red ones is a no

Alexandr Wang on how Meta’s Muse went from a magic prototype to a launch, one spreadsheet row at a time.

As The AI Corner tells it, Wang and Nat Friedman each wrote memos that went to Meta’s board in February, and a prototype of Muse, Meta’s personal agent, followed within one to two weeks. Much of the final product was already in that first demo, mascot included. Launch came seven months later. Wang describes the prototype as something that “sometimes worked and was magical, but most of the time didn’t work and would constantly break.” The other seven months went on making it reliable, which he calls, deliberately dully, “sanding out all of the details.”

The launch gate was a spreadsheet. It began as more than 100 specific behaviours and capabilities the ideal personal agent needed. Each row got an eval, the model was trained against each one, and reviews were held on whatever it was still bad at. Each row also carried two numbers set in advance: a threshold below which the launch was blocked, and one above which the row counted as good.

“We had many checkpoints that were green on 80 of the rows but red on 20 of the rows. We knew that wasn’t good enough.”

The team was under 200 people, model and product together, and Wang credits keeping the rest of Meta off it with avoiding what he calls “PM hell.” The detail worth keeping is about agreement. The first checkpoint to come back green across the board also felt, by hand, far better than anything before it. The sheet and the feel agreed. He also names the other ending: OpenClaw’s wave, he says, peaked and faded because people got magic moments while most of the rest broke. The newsletter adds sample thresholds of its own for readers to copy. Those are its examples, not Meta’s numbers.

The Collection · The Red Rows04

05  ·  The Port

The tests were the only description of what the code should do

GitHub moved 430,000 lines of Copilot’s runtime from TypeScript to Rust with agent fleets. The bugs that got through all compiled.

Copilot’s shared agent runtime ran on Node and V8, glued to the command-line app. Any product that embedded it, in Python, Java or C#, had to launch a second process and carry roughly 100 MB of overhead per client. GitHub wanted a library other programs could load directly, and picked Rust. The first estimate was 130,000 lines of production TypeScript. As logic moved out of the interface layer and colleagues kept shipping features, 430,000 lines went through the migration. ByteByteGo says one developer, using Copilot’s app, CLI and fleets of agents, finished in a few months, with help from others on packaging, builds and review.

The method was incremental. One component per pull request, a thin bridge so TypeScript callers could reach the Rust, and the old code deleted in the same change. Behaviour and algorithms were preserved on purpose; redesign waited. For the 30,000-line session file, a coordinating agent read for an hour, then split the work across 15 child sessions. They all tried to build at once and swamped the laptop, so builds were serialised. Review agents on different models compared old TypeScript and new Rust line by line, round after round, until clean. Humans reviewed architecture, contracts and risk.

The existing end-to-end tests stayed unchanged, because they were the only reference for expected behaviour.

That rule is the story. The tests needed guarding, because an agent can weaken a test or change an expected result to get a pass. The regressions that still slipped through compiled cleanly: whole numbers that became decimals and broke some SDKs, empty strings, time zones, replacement libraries that handled errors differently, cancellations that did not stop related work. The payoff was large. One client lifecycle went from 5.25 seconds to 292 milliseconds, and throughput from 7.55 session lifecycles a second to 120. The tokens cost about $120,000, for roughly 136 billion of them, mostly cached input. GitHub’s own lesson: more end-to-end coverage would have caught more.

The Collection · The Red Rows05

06  ·  The Bank Test

Ten accounts, ten thousand dollars, and one number that must never move

Sean Allen of Antithesis on correctness as something you design in, not a gate at the end.

Gergely Orosz set himself a challenge for this conversation: talk about building software in 2026 without saying “AI” once. He says he wanted Allen, a distinguished engineer at Antithesis and a core contributor to the Pony language, precisely because agents are now producing more code than ever, and with it more chances to ship bugs. Allen’s definition of correct is blunt. Software is correct when it does its business purpose and does not wake anybody up at 2 a.m. His harder point comes first: in lots of cases, people do not know upfront what they expect their software to do. They know when it is broken.

He calls himself a correctness person, not a testing person. Tests are for whatever you have no better way to verify. Push as much as you can earlier: strong type systems in the ML family, sum types with exhaustive matching so a forgotten case is a compile error, and the old rule to make illegal states unrepresentable. When you do test a big system, random inputs alone wander like a drunk and never get far from the start. You need a workload: inputs that drive the code, plus a rule for what counts as working.

If the isolation guarantees hold, the only total it should ever see is 10,000.

His example is the Bank Test. Put $10,000 into ten accounts of $1,000. Move money between them inside transactions while another process keeps summing the whole system. You never specify how much moves or when. You specify the one number that must never change. On a healthy database, nothing breaks. Add network faults or kill nodes, Allen says, and bugs pop out like crazy, the same conditions production serves up every day. It is Meta’s spreadsheet reduced to a single row, written before anyone ran anything.

The Collection · The Red Rows06

07  ·  The Quadrant

Fast where it can be checked, stuck where it cannot

Arvind Narayanan and Sayash Kapoor launch STABL, and name the chasm between verifiable and open-ended work.

Narayanan and Kapoor are starting a non-profit, STABL, the Societal Transformation and AI Bottlenecks Lab. They draw a two-by-two. One axis asks whether you take AI’s transformative potential seriously; the other, whether you take the downstream bottlenecks seriously. Top left is technological determinism, superintelligence flattening every obstacle. Bottom right treats existing institutions as fixed points the technology must fit. STABL claims the sparse top right: AI is transformative, and the pace of change will be set more by human and organisational change than by capabilities.

Their first research question is the one that matters here: how to measure progress towards recursive self-improvement, AI automating AI research. Most such evaluations, they write, measure tasks where success is easily verifiable, and there AI has moved fast. Their worry is a growing chasm between performance on verifiable tasks and on open-ended ones. Benchmarks that rely on automated checking are also, conveniently, the ones best suited to reinforcement-learning training.

Expert reviewers unambiguously rejected the papers produced by frontier agents.

That is their first CRUX result, from open-ended evaluations of whether agents can do AI research on long, messy tasks. Not yet, is the answer, and they plan to rerun it with better models and scaffolds. The link to Meta’s sheet and GitHub’s tests is direct. Where you can write the row, an agent can be driven to green. Where nobody can write the row, there is no green to reach, and a human reviewer is still the only instrument.

The Collection · The Red Rows07

08  ·  Standing Orders

Four rules for this issue

  1. I

    Write the red line before the demo.

    Decide which behaviours block launch, and at what pass rate, before the prototype has a chance to charm you. A row with no threshold is a wish.

  2. II

    Lock the tests away from the agent.

    The end-to-end tests are the spec. An agent that edits a test to get a pass has edited the spec, and nobody signed off.

  3. III

    Change one thing at a time.

    Port first, improve later. When a regression arrives, it should have one suspect, not two.

  4. IV

    Know which of your work has no row.

    Checkable tasks will get fast. For the open-ended ones, find the reader whose judgement you trust, and budget their time.