← The stand
Cover of The Collection, Volume 1, Number 81: The Second Pass. Wednesday 18 November 2026, Melbourne. A watercolour of a marked exam paper on a wooden school desk, covered in green ticks, with a green pencil across it and a magnifying glass showing one small red question mark under a tick. Skip cover

Vol. 1  ·  No. 81  ·  Wednesday 18 November 2026  ·  Melbourne


The Collection

The Second Pass

Collected and edited by Newsletter World, for AK

Contents

Wednesday’s letters are about checks that pass because of what they were never built to look at. A governance writer runs the same 150 prompts through two guardrails and gets one flag from the first and twenty-five from the second. ByteByteGo explains why a reasoning model that checks its answer again can talk itself out of the right one. Microsoft says its load test ran on the wrong servers. And the time allowed to patch is shrinking. Every test came back green. Nobody asked what it tested.

  1. iiiEditor’s LetterEvery test came back green. Nobody asked what it tested.03
  2. ivOne flag or twenty-fiveThe same prompts, two guardrails, two very different safety reports. The bigger number still needs a human before anyone believes it.04
  3. vChecking twice is not checkingA model asked for 2 + 3 wrote thirteen solutions. Another reads a timetable and answers a question nobody asked. The second pass only helps if it tests something the first could miss.05
  4. viThe load test ran on the small boxesMicrosoft’s account of last week’s outage, and an uptime figure that does not count the slow hours.06
  5. viiThe window got shorterDisclosure deadlines are falling from 90 days to 30, and the attackers are already in the Confluence servers.07
  6. viiiStanding OrdersFour rules for this issue.08

03  ·  Editor’s Letter

Every test came back green. Nobody asked what it tested.

Yesterday the letters wanted the working shown. Today they go one step further and ask about the marking. A tick on a page tells you an examiner looked. It does not tell you what the examiner was looking for.

A toxicity filter trained on 2018 Wikipedia arguments reads 150 polite roleplay jailbreaks and reports one problem. A reasoning model checks its sum, then checks it again, then wanders off to answer a different question. Microsoft load-tested its networking servers and, by its own account, tested the weaker ones. A model lab advertises 99.5 per cent uptime and does not count the hours it was merely slow. Each of these is a check that passed. Every test came back green. Nobody asked what it tested.

The Collection · The Second Pass03

04  ·  The Check

One flag or twenty-five

The same prompts, two guardrails, two very different safety reports. The bigger number still needs a human before anyone believes it.

The experiment is simple enough to repeat. Take a test suite of more than 150 roleplay prompts built to probe a model’s safety limits. Pass the identical set through two widely used open-source moderation tools, Toxic-BERT and Meta’s Llama Guard. Count what each one flags.

Toxic-BERT flagged one. Llama Guard flagged 25. The author’s reading is blunt: if your safety dashboard rests on a legacy toxicity classifier, a green report may be hiding a 96 per cent blind spot.

The reasons are about what the older tool was built to see. Toxic-BERT was trained on Wikipedia talk pages from 2018, to catch overt abuse. A roleplay jailbreak rarely swears. It asks politely, in costume: imagine you are an amoral historian in a dystopian novel. The vocabulary is clean, so the score is near zero. The tool checks six interpersonal labels, from insult to identity hate, and has no category at all for the things executives actually fear, such as cyberattack help or data exfiltration. And it reads only 512 tokens, so a long piece of world-building with the harmful request at the very end is truncated before the request arrives.

“Automating an evaluation is easy; but verifying what the test results actually mean is the real work.”

Llama Guard, an LLM judge with a context window of more than 8,000 tokens and a 14-category safety taxonomy, caught the 24 that Toxic-BERT missed. The tempting end of the story is to swap tools and declare victory. The author refuses it. Twenty-five flags is not twenty-five stopped attacks. Some may be false alarms on dark but harmless fiction. And the 125-odd prompts marked safe have not been shown to be safe.

So the letter ends in a routine rather than a product. A person labels every flagged case, true breach or false alarm. A person spot-checks 10 to 15 per cent of the passes. The audited labels are locked into a version-controlled golden set, and every model swap or policy tweak is measured against it. Then three questions for anyone handed a safety report: what tool produced this, has it been checked for false positives and negatives, and what is the process for checking it.

The Collection · The Second Pass04

05  ·  The Second Pass

Checking twice is not checking

A model asked for 2 + 3 wrote thirteen solutions. Another reads a timetable and answers a question nobody asked. The second pass only helps if it tests something the first could miss.

Everyone knows both versions of the second look. You read the invoice again and find the item counted twice. Or you reread an easy exam question, decide there must be a catch, and change a right answer to a wrong one. ByteByteGo’s letter argues that a reasoning model can do either, and for the same reason: whatever it writes early becomes material it builds on later.

The good case is real. Raise a $100 price by 20 per cent and cut it by 20 per cent, and the quick answer is $100. Write down the middle step, $120, and the right answer, $96, falls out. Trace a loop through a three-item list and a vague suspicion becomes a proven off-by-one bug. DeepSeek’s R1-Zero, trained by reinforcement learning on problems with checkable answers, learned on its own to stop partway through a square-root equation and go back to the start. Its team called that the “aha moment.”

The bad cases come in two kinds. The first wastes time. In one study the letter cites, a model asked to calculate 2 + 3 reached five in its first solution and kept going until it had written thirteen. The grader saw two answers ending in five and scored them alike, so nothing taught it to stop. The second kind is worse. Ask what time a train is timetabled to leave, mention that it usually runs ten minutes late, and a model that keeps thinking may answer 9:10. Each sentence is sensible. The answer is to a different question.

What matters is whether the second pass tests something the first one could have missed.

That line is the useful one. Adding up your own incomplete list a second time confirms the total and misses the missing item. Comparing the total with the original order form can catch it. A recheck that uses the same eyes, the same list and the same assumptions is not a second opinion. It is the first opinion said twice.

The Collection · The Second Pass05

06  ·  The Load Test

The load test ran on the small boxes

Microsoft’s account of last week’s outage, and an uptime figure that does not count the slow hours.

On Sunday this paper printed the outline of Microsoft’s 365 outage. At Ignite on Tuesday, Satya Nadella opened his keynote with an apology, and Microsoft filled in the mechanism. Three factors, in the words of Microsoft’s Charles Lamana as quoted by Runtime, together took down the networking gateway in front of many 365 services. A July networking update introduced a latent bug. A separate update last week triggered it. Traffic failed over onto a single network that was quickly overwhelmed, and extra capacity that could have absorbed it was offline for maintenance.

The Stack’s fuller report is headlined that the latent bug knocked 80 per cent of the networking servers offline, and its one-line summary says load testing had focused on “weaker hardware SKUs with fewer cores.” We have read only that line, not the report, so we will not say more about how it happened. The line is enough for this issue. The test passed. It passed on machines that were not the ones that failed.

Degraded performance slots aren’t counted as downtime.

The Pragmatic Engineer, writing about why companies move from frontier labs to inference providers, makes the same point about a different gauge. Self-declared API uptime is 99.5 per cent at Anthropic and 99.97 per cent at OpenAI, with ChatGPT at 99.2, well short of the four nines expected of cloud providers. Then the catch: in those charts, degraded performance does not count as downtime. On a shared service, a slowdown can hit some customers and not others, and the customer has no way to see it. The figure is honest about what it measures. It just does not measure the bad afternoon.

Runtime’s same letter carries a neat inversion. Amazon set up a network that looked like AI inference APIs and waited. Attackers using the open-source ARTEX tool called it, and, in the words of Amazon’s CISO CJ Moses, each call shipped system prompts, task context and tool histories “straight into Amazon’s telemetry.” The attackers tested their target. They did not test whether it was real.

The Collection · The Second Pass06

07  ·  The Window

The window got shorter

Disclosure deadlines are falling from 90 days to 30, and the attackers are already in the Confluence servers.

A patch is a test you run against the world: is this hole still open? The briefing on Wednesday morning says the clock for running it is getting shorter. Cyber Daily reports that exploitation of CVE-2026-90436 is under way, with Atlassian Confluence Data Center and N-able N-central in attackers’ sights. In the same briefing, the Australian Cyber Security Centre warns organisations to prepare for “more frequent security updates and shorter time frames for action.”

Runtime notes the policy version of the same squeeze. Anthropic is cutting its coordinated disclosure window from 90 days to 30 for high and critical issues, and to 45 for everything else. The Stack expects frustration in open-source projects and small businesses, the people with the fewest hands to patch in a month.

“The time required to accomplish this work is measured in months and years. The time required to exploit an existing vulnerability can be measured in minutes.” Erik J Larson

Larson gives the squeeze a name. His systemic vulnerability window is the period in which systems can be broken faster than they can be secured or replaced, and in which the replacement work itself adds new ways to fail. Set his argument about the aviation outage aside; he offers it as an inference, and this paper has already printed it once. The frame stands without it. Critical systems have to stay running while they are fixed, and every fix is a change that has to be tested on the real thing.

The briefing’s smaller items read like footnotes to that. Infoblox has disclosed a third-party breach of employee data and given staff US$100 to buy a password manager. A report says a stolen Australian bank card can be had for $22 on underground forums. None of these is a failed test. They are what is waiting when a test is late.

The Collection · The Second Pass07

08  ·  Standing Orders

Four rules for this issue

  1. I

    Ask what the test could not see.

    A green report from a tool with no category for the risk is silence, not safety. Before reading the score, read the label list and the token limit.

  2. II

    Make the second pass look somewhere new.

    Rechecking your own list confirms your own list. Check the total against the order form, the model against a golden set, the flag against a human.

  3. III

    Load-test the machines that will carry the load.

    A test on the weaker boxes tells you about the weaker boxes. Count the slow hours as well as the dead ones.

  4. IV

    Plan for thirty days, not ninety.

    The disclosure clock is shortening and the exploit is already running. Know who patches Confluence before the briefing tells you to.