AOS for Security Review

Eight issues of machinery. Now the machinery meets a real job.

Issue 8 closed the Patterns arc by proving one profile runs the same on four vendor CLIs. This issue opens the Domains phase, where that machinery finally has a live production system and a threat model behind it. The Council's best first production use is security review, because security has something most decisions lack: observable failure. A missed vulnerability is a fact you can point at, and that makes it the safest place to start trusting the output.

Why security goes first

Run a strategic-council on whether to ship autonomous deploys and you get a memo with the dissent still attached. Good. But you cannot check it. The recommendation is right or wrong over quarters, tangled up with the market, the team, and luck. There is no moment where a red light comes on.

Security is different. Point a Council at a system, ask what breaks, and the answers are testable. Either an Org A user can read Org B's row or they cannot. Either the reset token expires or it does not. When the Council is wrong, you find out, and you find out with a log line. That is the whole case for starting here. You get to audit the auditor.

The profile, recalled

Issue 4 put security-review next to strategic-council and showed the swap: same Arbiter, Catalyst out, Steward in, the memo becomes a report. The `security-review` profile is seven agents, not twelve. Architect, Sentinel, Provocateur, and Steward are required. Operator and Navigator ride along by default.

The assembly is the evidence standard. Catalyst asks "will customers pay for this in 90 days," which is the wrong question to put to a threat model, so it has no seat. The Steward is required instead, because someone in the room has to speak for regulatory exposure. And the Sentinel carries the security bar specifically: what breaks the trust relationship and cannot be un-broken. In a strategy call the Sentinel argues change-failure rate against velocity. Here it argues which failures are recoverable and which end a company.

The threat model we fed it

The system under review is the one this newsletter has been carrying since Issue 6: the multi-tenant B2B auth system the Crew planned and built across Issues 6 and 7, and now the Council reviews. tRPC and Prisma, org-scoped, two public endpoints (password reset and an OAuth callback). I wrote it up as a sanitized threat-model brief in the shape the profile demands: ## System Description, ## Known Threats, ## Compliance Requirements, ## Key Question.

The Key Question did the real work. It skipped "is this secure," which no review can answer, and asked for a split: of everything you find, which risks are real and reachable, and which are theoretical? Rank them so we know what blocks the ship.

Then aos run security-review, session 2026-07-01-security-review-mr25i7ur. Three rounds, eighteen agent responses, a real debate.

The debate

Five of six specialists led with cross-tenant object access: broken object-level authorization at the tenant boundary, OWASP A01. When five voices say the same thing in nearly the same words, that is not five signals. It is one, restated. A single competent reviewer would have found it too.

The signal was in where they refused to agree.

The Architect reframed it from a bug into a structural property:

"Every request is org-scoped" is a sentence. The only thing that matters is where that scoping lives.

Then it drew the two worlds. Fail-open, where each resolver hand-writes where: { orgId } and every new endpoint is a fresh chance to forget:

When they forget, the query returns every tenant's rows, the test suite passes because your seed data has one org, and it ships green.

Versus fail-closed, where the data layer physically cannot emit a tenant-scoped query without an org id. That is the insight a single reviewer usually skips past. The finding is not "there is an IDOR somewhere," it is "enforcement is per-resolver by convention instead of at one chokepoint," and those two have opposite failure modes. The Steward sharpened it into an evidence standard:

"every request is org-scoped" is an assertion, not a control, and assertions are exactly what I don't accept as evidence.

Then the room split productively. One agent pushed the failure site past the obvious GET /thing/:id: nested include, aggregates, exports, background jobs with no org in context at all. The per-resolver fix misses those by design. The Steward carried a second blocker nobody else was holding, the compliance one:

when that bug is found, can you prove the blast radius? Under Art. 33 you have 72 hours to notify with scope.

If your audit trail is itself not tenant-scoped, you cannot answer that, and under Art. 33 the incomplete answer is its own violation. Pure appsec finds the hole. The compliance seat asks whether you can measure it after.

And the noise. Four separate agents flagged a full CSRF token layer as over-weighted, sameSite=lax plus JSON-only POST already covers the surface. They were right, but I did not need four of them to tell me. That is the tax on a Council: agreement is cheap and gets repeated, and you pay for the seats whether they add a perspective or echo one.

The report that never got written

The deliberation ran, but security-review sets agent_timeout_seconds: 120, and the Arbiter's final turn, synthesizing everything into the formatted report, blew past it. The run ended with no report.md. The profile's on_orchestrator_failure: save_transcript_and_exit did its job, so the agents' analysis is intact, but I assembled the ranked list below by hand, since the Arbiter never produced it. I am not going to show you a clean report the machine did not produce. (The fix is a longer timeout for the synthesis turn than for the specialists.)

What the agents converged on, ranked:

1. BLOCKING, real. Cross-tenant object access, broken object-level authorization at the tenant boundary (A01), because org-scoping is per-resolver by convention. The remediation the room agreed on: move isolation to one fail-closed layer, Postgres RLS or a mandatory Prisma extension that injects orgId, gated by an "Org A cannot read Org B" test in CI.
2. BLOCKING, real. Pre-auth account takeover through the two public endpoints. Reset tokens stored in plaintext or non-single-use, host-header-poisoned reset links, and an OAuth callback missing state that auto-links a Google login to an existing email account.
3. Documented dissent on severity. No one contested either blocker. The second tier was where the room split. One agent explicitly demoted login rate-limiting to a fast-follow "with an SLA," arguing a throttle gap does not end a B2B contract. The job of a risk matrix is to record that split, not average it away.

Real exposure versus theoretical

That was the Key Question, and the Navigator answered it with a rule the security seats did not have:

rank by irreversibility, not CVSS. A bug you can patch-and-force-reset next sprint is a fast-follow. A bug that, once triggered, is already notifiable and already gone from the market's memory is a blocker.

That rule reordered the board. A weak password policy scores high on a severity scale and is completely reversible: tighten it, force resets, nobody remembers. A cross-tenant read scores about the same and is unrecoverable. The data is out, the breach is notifiable, and in B2B the reference-customer trust that took years to build dies in one disclosure email. CVSS cannot see that difference. Irreversibility can, and it is exactly the judgment a risk matrix is supposed to encode.

Which is why the disagreement mattered more than the consensus. A security council that agrees on everything reviewed nothing. The disagreement on severity is where the real risk lives.

Next

Security review was the easy case: the machine's mistakes surface as log lines you can point at. Architecture is the hard one, where the right answer is often documented disagreement rather than a single recommendation, and being wrong shows up quarters later in a migration nobody can afford. Issue 10, AOS for Architecture Review. The profile is at harness.aos.engineer.