249 agents to book one condo

The goal was almost embarrassingly simple: two people, twelve nights on Maui, celebrating a birthday. Under $3,500 for housing. An ocean view. And a beach you could actually swim at — soft sand, no rocks.

That last constraint is the whole story. It's not a field in any database. No listing will ever tell you its beach is rocky. And it turned a weekend of browsing into a genuinely hard search problem — one that ended up consuming 17 multi-agent workflows, 249 agent runs, and 41 shipped versions of a self-updating trip dossier.

Three separate times, a "winner" was crowned, written into the plan, and published. Three times, a skeptic agent tore it down: one was already booked. One was "oceanfront" onto a shelf of lava rock. One had an owner-direct link that quietly 301'd to a different condo nine miles up the coast, so the "verified price" we'd been quoting was for a property that didn't exist at that URL.

The unit we finally booked was never the front-runner. Total cost of the exercise: less than one night of the condo. Here's the whole machine.

The stack

Everything ran inside Claude Code (Opus 4.8) as the orchestrator. The interesting part isn't the model — it's what it was allowed to touch.

The pattern: fan out, then attack

The naive approach is to ask an LLM to search harder. That produces more plausible-sounding candidates, not more true ones. What actually worked was a two-phase shape, repeated over and over:

1. Fan out. Spawn N agents in parallel, each with a different search angle — one hunts owner-direct sites, one hunts Airbnb, one hunts timeshares, one hunts beach guides. Each returns a schema-validated object, not prose.

2. Attack. For every candidate that survives, spawn skeptics whose only job is to refute it. Not "evaluate" — refute. Default to rejection when uncertain.

flowchart TD
  A["Constraints — the hard filters"] --> B{"Fan out"}
  B --> F1["Finder: marketplace"]
  B --> F2["Finder: owner-direct"]
  B --> F3["Finder: adjacent categories"]
  B --> F4["Finder: independent sources — build the missing column"]
  F1 --> C["Candidates — schema-validated"]
  F2 --> C
  F3 --> C
  F4 --> C
  C --> D{"Adversarial verify — refute by default"}
  D -->|PROBLEM| X["Killed"]
  D -->|CONFIRMED| S["Survivors"]
  S --> N["Prove the negative"]
  N --> R["Synthesize into the dossier"]
  R --> AU{"Self-audit for contradictions"}
  AU -->|contradiction| R
  AU -->|clean| RV["Re-verify at commit time"]
  RV --> H(["Human decides"])

Here is the verify phase, verbatim from the file that ran (trimmed for length — the prompt was longer). It's JavaScript because the workflow runtime executes plain JS in a sandbox; this is not pseudocode:

// verify-kamaole.js — executed by the Workflow runtime
export const meta = {
  name: 'verify-kamaole-winners',
  description: 'Adversarially confirm the 3 finalists before crowning one',
  phases: [{ title: 'Verify', detail: 'refute each candidate' }],
}

const CANDS = [
  { key: 'Royal Mauian',              room: '50815427', claim: 'BEACHFRONT on Kamaole I, full ocean view, ~$3,423 all-in…' },
  { key: 'Kamaole Beach Royale #204', room: '21244792', claim: 'ocean-view lanai, ~3-min walk to Kam I soft sand, ~$2,556…' },
  { key: 'Maui Kamaole I-118',        room: '22531397', claim: 'ocean-view lanai (filtered), Keawakapu ~5-10 min, ~$2,970…' },
]

phase('Verify')
const verdicts = await parallel(CANDS.map(c => () =>
  agent(
    `Adversarially verify this condo before we crown it.
     Dates: 2026-07-25 -> 2026-08-06, 2 adults.
     CLAIM TO TEST: ${c.claim}

     Try to REFUTE each of these. Report what you actually find:
     1) AVAILABLE for the exact dates? (active Reserve button?)
     2) REAL ocean view from the unit? (photo/amenity evidence, not marketing fluff)
     3) Exact ALL-IN price incl. Hawaii tax ~17-19%. Is it truly <= $3,500?
     4) Nearest soft-SWIMMABLE sand + walk time. Rocky? Murky?
     5) Red flags: construction, for-sale showings, no kitchen.

     Final verdict: CONFIRMED (claims hold) or PROBLEM (what breaks).`,
    {
      label: `verify:${c.room}`,
      phase: 'Verify',
      schema: {
        type: 'object', additionalProperties: false,
        required: ['verdict','availableDates','oceanView','allInPrice','underCap','beach','redFlags'],
        properties: {
          verdict:        { type:'string', enum:['CONFIRMED','PROBLEM'] },
          oceanView:      { type:'string', enum:['yes','partial','no','unknown'] },
          underCap:       { type:'string', enum:['yes','no','unknown'] },
          availableDates: { type:'string' },
          allInPrice:     { type:'string' },
          beach:          { type:'string' },
          redFlags:       { type:'string' },
        },
      },
    }
  )
))
// → all three came back CONFIRMED. Two came back oceanView: 'partial'.

That last line is the point. The enum on oceanView is doing quiet, load-bearing work: it denies an agent the option of writing "stunning ocean vistas!" and forces it to pick one of four words. Two finalists whose listings both said "ocean view" resolved to partial — a strip of blue over a neighbor's roof, and a lanai filtered through landscaping. Free-text output would have laundered both into a yes.

(The diagram above is live Mermaid — the source is in the repo.)

What actually broke

This is the part worth reading. Every one of these cost us a crowned winner.

Proving the negative

Late in the process came the best question of the whole project: "Is a condo really the best value? Is there nothing else?"

This is exactly the question a search engine can't answer, because it requires exhausting a space rather than ranking within one. So: six agents, one per lodging category, all pricing live for the exact dates.

Verdict: the condo wins, and now we know why — it's the only category that satisfies ocean view and soft swimmable sand and a whole private place under the cap. Every cheaper option drops one leg. That's a far more useful answer than "here are 40 listings."

What we booked

That's the division of labor I'd defend: agents narrow, humans decide. The machine's job was never to pick. It was to guarantee that every option still standing was real, bookable, honestly priced, and actually met the constraints. Nine survivors out of a few hundred candidates. The final call took thirty seconds.

The human in the loop: a fair post-mortem

It would be dishonest to write this up as though the human just supervised. I went back through the logs — 59 turns — and the honest picture is that the four hard filters split into two very different kinds:

flowchart TD
  C["Every constraint on the trip"] --> Q{"Could it be written down?"}
  Q -->|"budget and dates"| SPEC["SPECIFIABLE — stated in the brief on day one — the agent's job is to REMEMBER it"]
  Q -->|"'a good beach'"| DISC["UN-SPECIFIABLE — has no definition — must be DISCOVERED by counterexample"]
  SPEC --> P["A harness you can trust needs BOTH: durable memory and counterexample elicitation"]
  DISC --> P

The budget and the dates were in the brief from the first message. The two filters that ended up deciding everything — walk to soft, swimmable sand; no rocks — were not, and could not have been. "A good beach" isn't a spec; it's something you recognize. Those two got discovered the only way they could be: by rejecting a finished answer and reading what the rejection revealed.

Two honest complications, and neither is the user failing to specify. First: the budget tightened mid-project — from the brief's flexible "~$6–8k" to a hard $3,500 cap — and across a long, multi-session run the agent lost the thread on price. When the user caught it, the words were "you lost the price goals," which was exactly right: an agent had dropped a constraint it was handed. That's context rot, a real failure mode of long agent runs, and it's the machine's bug to fix. Second: the dates moved to August for a birthday, and in travel every price and availability check is keyed on dates — so each change silently rotted every number in the document. The trip legitimately changed; the system just didn't keep up.

What that cost

• The agent dropped a constraint it was given. The budget was in the brief; when it tightened, a long multi-session run let it slip, and "you lost the price goals" was an accurate bug report, not a mood. Long agent sessions rot context — a serious harness has to re-pin the hard filters at every commit, not quietly forget them.
• Decorating before validating. Turns were spent on a "bouginess slider," image thumbnails, sortable tables, and a wide-screen layout — while the crowned recommendation was simultaneously over budget, without an ocean view, and in one case already booked. Polish applied to a wrong answer is worse than no polish: it makes the wrong answer look trustworthy.
• Global rewrites for local problems. "This is too much info, boil it down" triggered a teardown from 26 sections to 6. The very next message: "oh that is like maybe not enough now." The actual desire — "put a summary on top, keep the detail below" — would have been one edit.
• Burying the simple answer. The map thread ran four turns and three implementations (embedded Google, custom Leaflet pins, then links). The user proposed the winning solution themselves, mid-thread: "or just make it simple and link to the google map in a link column." That was available on turn one.

What the human got right — and it's the good stuff

• The single best idea in the project was theirs, not the machine's. "The listing doesn't have to say it has sandy beaches — you could cross-reference with other known good beaches." That reframed the task from ranking listings to constructing the column nobody had. Everything downstream depends on it.
• "Don't waste my time with listings that are already booked." This forced the availability re-check that discovered the crowned pick was, in fact, booked. That one sentence saved the trip.
• "Is it really only condos? Is there nothing else?" Demanding proof of the negative when you already have an answer you like is rare and expensive and correct.
• "I'm logged into CostcoTravel.com now if you want to use the browser." Recognizing that authentication is the one thing the agent cannot do, and supplying it. That's sophisticated collaboration, not prompting.
• "Burn my tokens, I don't like waiting." Correct resource intuition: agent time is cheap and parallel, wall-clock is neither.
• Never letting the machine buy anything. Links only, human at the checkout. Every time.

You don't know what you want until you see what you don't

Turn 44 of this project is one sentence long:

"The rocks are not good."

Before that sentence, "soft sand" was not a requirement. After it, it was the requirement — the one that ultimately decided everything, invalidated three prior workflows, and forced us to build a dataset that doesn't exist anywhere. The constraint did not exist, and then it did. A counterexample conjured it.

This is not a story about a user who failed to write good requirements. It's a fact about people, and once you take it seriously it changes how you build the tool.

Preferences are constructed, not retrieved

Justice Potter Stewart's "I know it when I see it" (Jacobellis v. Ohio, 1964) gets quoted as a punchline about judicial laziness. It's actually an accurate report of how categorization works for concepts that resist definition by necessary-and-sufficient conditions — what Wittgenstein called family resemblance. "A good beach" has no definition. It has exemplars.

Decision researchers have the harder-nosed version. Paul Slovic's line of work on the construction of preference holds that people don't read values off a stable internal utility function; they build one at the moment of choice, and the construction depends on what they're shown and how it's framed. Ask someone to specify what they want and you get a lossy, partial reconstruction. Show them a violation and you get an instant, confident, correct rejection.

Which yields the asymmetry the whole harness should be designed around:

Recognition is cheap and reliable. Specification is expensive and lossy. Rejections are high-bandwidth.

This is RLHF, done by hand

Here's the part that should feel uncomfortable to anyone who's trained a model. The user could not write down their objective function. Neither could the agent. But the user could reliably label a candidate as bad, and say roughly why.

So what we ran, for 59 turns, was online preference learning from negatives. Each rejection was a comparison label. The "locked constraint table" was a reward model, fitted incrementally, badly, and mostly from things the user hated. RLHF exists for exactly the same reason "gather the requirements" fails in the first place: for the things that matter most, humans can evaluate but not specify. We built a whole industry on that asymmetry and then act surprised when it shows up in a condo search.

Why this only became viable recently

Stigler's 1961 The Economics of Information gives the boring, correct frame: optimal search depth is a function of the marginal cost of one more query. When each iteration costs you a weekend of browsing tabs, the rational play is to commit early to a spec you haven't tested — or to satisfice in Simon's sense and stop at "good enough, I'm tired."

Collapse the cost of an iteration to ten minutes and a few hundred thousand tokens, and the calculus inverts. "I'll know it when I see it" stops being an unaffordable luxury and becomes the efficient strategy.

Which also gives the fix. If people can't state constraints but can recognize violations, then stop asking and start showing. Before any expensive search, spend ~1% of the budget generating five deliberately diverse candidates — the cheapest thing that qualifies, the extravagant one, one that's perfect-but-for-a-flaw, one that's just weird — and ask a single question: "which of these are wrong, and what makes them wrong?"

That probe, run on turn 3, would have surfaced the rock allergy in minutes instead of on turn 44 — and it would have caught the budget tightening the moment it happened, instead of letting a long session quietly lose it. It's now Step 0 of the skill.

The hazard nobody warns you about

Cheap generation makes mis-specification more dangerous, not less. An agent will pursue a stated objective with superhuman efficiency and hand you a confidently wrong answer, beautifully typeset. We lived this: for about twenty turns the dossier was a gorgeous, sortable, thumbnail-rich, dark-mode-aware monument to a recommendation that was over budget and had no ocean view. Goodhart's law with better typography.

And the failure is self-reinforcing, because a polished artifact teaches you to want what it found. Anchoring plus the persuasive authority of good design. Left unchecked, the tool stops helping you discover your preferences and starts manufacturing them.

Verification is the only thing standing between those two outcomes. Which is why "verify before you decorate" is a moral instruction, not a style guide.

Where the human is not replaceable

Once every hard constraint is satisfied, look at what's actually left. Nine survivors. Beachfront-on-Kamaʻole-I at $3,423, versus 1,000 square feet across the road from a better beach at $2,937. Which is correct?

Neither. There is no exchange rate between "steps onto sand" and "an extra room" and "$486." These are incommensurable goods, and any weighted score that pretends to rank them is just someone's opinion wearing a lab coat. (Arrow has a theorem about this mood.)

So the division of labor isn't a hedge about AI safety. It falls out of the structure of the problem:

Agents enforce facts. Humans adjudicate values.

Constraints are factual, verifiable, and therefore delegable — is it available, is it under $3,500, is the sand actually soft. Tradeoffs among satisfied constraints are not factual at all. 249 agent runs took a few hundred candidates down to nine. A human took nine down to one in about thirty seconds, and picked the roomy one next to the best water. Both numbers are correct. Neither party could have done the other's job.

Anticipated objections

In the spirit of saving everyone some time:

"Couldn't you have done this yourself in an hour?"

I could have gotten a worse answer in an hour. An hour of browsing surfaces neither the front-runner that was silently booked, nor the owner-direct link redirecting to a different building, nor a beach-quality ranking that exists in no database. I did spend the hour — on a version of the trip that later changed, chasing a "good beach" I couldn't define until an agent showed me a bad one.

"249 agents to book a condo is absurd."

Correct, and that's the interesting part. It's absurd at today's prices, for this stake. Both of those numbers move, and they move in opposite directions. The economics only work when the cost of being wrong is large and the commit is hard to undo. Which brings us to:

"So when should I actually do this?"

When P(wrong) × cost(wrong) > cost(verification). For a $3,000 booking with a birthday attached, verification is nearly free by comparison — verify aggressively. For a $30 reversible purchase, don't run any of this. Buy the socks.

"Why not just use the site's filters?"

Because the deciding attribute wasn't a filter. There is no beach_is_soft checkbox, and the only parties positioned to add one are the parties with an incentive to always check it.

"Did the agents hallucinate?"

Constantly, and always in the direction of optimism. Free text let them launder "a strip of ocean visible over the neighbor's roof" into "ocean view." The fix wasn't a better model — it was forcing them to pick one of four enum values and then paying a second model to attack the answer. Hallucination is a formatting problem more often than people admit.

"What did it actually cost?"

Over 1.5M subagent tokens across the measured workflows, roughly 35 minutes of agent wall-clock for the five biggest, and less money than one night of the condo it booked.

The reusable recipe

Strip out Maui and this generalizes to any high-stakes, low-frequency, multi-constraint purchase — apartments, contractors, weddings, used cars, schools:

1. Write the constraints down as hard filters. Ours: available · ocean view · walk to soft-swim sand · ≤ $3,500. Anything that fails one is dead, regardless of how good it looks.
2. Identify the missing column. The attribute that decides the purchase but exists in no database. Build it from independent sources. (For us: beach swimmability.)
3. Fan out along orthogonal search angles. Different agents, different sources, blind to each other. Structured output, always — schemas beat prose.
4. Adversarially verify every survivor. Spawn skeptics prompted to refute. Majority-refute kills the candidate. Uncertainty defaults to rejection.
5. Prove the negative. Exhaust the adjacent categories so "this is best" is a finding, not an assumption.
6. Re-verify before committing. Availability and price are perishable. The check that mattered most was the last one.
7. Audit your own artifact. Generated documents accrete contradictions. Chunk it, hand agents the ground truth, ask what conflicts.

We turned it into a skill

The recipe above isn't a blog-post abstraction — it's a file. Claude Code lets you write skills: markdown documents that get loaded into context when a matching task shows up. So the hard-won lessons don't evaporate when the session ends; the next search inherits them.

Two came out of this project:

• /trip-planner — the domain workflow (fan-out research, live MCP pricing, all-in cost model, HTML dossier). It existed before Maui. It has now been amended with everything that went wrong: lock the dates before deep pricing, don't start until you have the ceiling and the hard filters, and a list of the traps that each cost a crowned pick.
• /adversarial-verify — the general one, and the more interesting one. It has nothing to do with travel. It's the harness for any decision where the person writing the data benefits from your believing it: rentals, contractors, vendors, marketplaces, job posts.

It isn't a manifesto; it's machinery. Nine files: the skill itself, four references, three runnable workflow templates, and a checklist you read out loud before the user clicks buy.

adversarial-verify/
├── SKILL.md                          # the harness
├── references/
│   ├── preference-construction.md    # why constraints arrive late; design for it
│   ├── traps.md                      # the 200-OK lie, marketing laundering
│   ├── schemas.md                    # enum library; why free text is unsafe
│   └── stopping-rules.md             # Δp·L > c  — when to stop verifying
├── templates/
│   ├── verify.workflow.js            # skeptics w/ diverse lenses; majority-refute
│   ├── prove-negative.workflow.js    # adjacent-category sweep
│   └── artifact-audit.workflow.js    # chunked contradiction sweep
└── checklists/pre-commit.md

Step 0 is the one I'd defend hardest, because it's the fix for the failure this whole post is about:

# Step 0 — Probe before you search

> The user cannot tell you their constraints. They can only recognize
> violations. So don't ask. Show.

Spend ~1% of budget on 5–7 deliberately diverse candidates — including
ones you expect to be wrong — and ask exactly this:

    "Which of these are wrong, and what specifically makes them wrong?"

Then: filter in code, never in a prompt. Uncertainty fails.
Majority-refute kills. Ties go to the skeptics.

That's the artifact I'd actually hand a friend. Not the trip plan — the harness that produced it.

Why this might be a company

Aggregators solved finding. Nobody solved trusting. Every booking site on earth will show you a price for a room that's unavailable, a beach that's rocky, and a "verified" owner link that redirects to another property. The listings are an adversarial data source, and the incumbents' incentives are not aligned with telling you so.

The defensible layer isn't the index — it's the verification harness: the browser fleet that confirms a Reserve button is actually live, the skeptic agents that demand photo evidence for an "ocean view," the constructed columns for qualities no schema captures, and a receipt trail showing exactly how each claim was checked.

It's expensive per query — 249 agent runs and well over a million tokens for one booking. Which is precisely why it works for decisions where being wrong costs $3,000 and a ruined birthday, and doesn't work for buying socks. That's a real market, and it's the one where "AI search" currently fails the hardest.

The trip is booked. The beach is soft. The rest is a product idea.