I’ll Know It When I See It

The brief sounded easy. Twelve nights on Maui for two people, celebrating my partner’s birthday. Under $3,500 for lodging. An ocean view. And the requirement that actually mattered: a beach you could genuinely swim at. Soft sand under your feet. No rocks.

That last one is the whole story. “Is this beach soft and swimmable?” is not a field in any booking database. There’s no filter for it. And every single listing — every one — claims its beach is wonderful. One unquantified requirement turned what should have been a weekend of browsing into a real search problem.

So I did what you do in 2026 when you have a real search problem: I threw agents at it. A lot of them. By the end: 17 multi-agent workflows, 249 individual agent runs, more than 1.5 million tokens of agent work, across 41 versions of a living trip-planning document. Total cost: less than one night of the condo it eventually booked.

The stack, briefly

The brain was Claude Code running Opus 4.8 as the orchestrator, wired to a handful of data sources over MCP (the open protocol for connecting models to tools). Each one earned a specific level of trust:

• An Airbnb data connector — great for structured listing details; not trustworthy about whether anything is actually available.
• A real automated web browser — the only reliable source of true availability and final checkout prices.
• Hotel-rate connectors (DirectBooker, TripAdvisor) for comparison pricing.
• A geocoder (Nominatim) to turn addresses into map coordinates.
• Open web search for the thing no booking site would ever tell me: independent beach reports.

That trust hierarchy — structured APIs for facts about listings, a real browser for facts about right now — turned out to matter more than any prompt I wrote.

The core pattern: fan out, then attack

Every workflow had two phases.

Fan out. Spawn many agents in parallel, each hunting a different way: one combs Airbnb, one hunts owner-direct sites, one chases timeshare rentals, one reads independent beach guides. Each returns structured data — fields, not essays.

Attack. For every surviving candidate, spawn skeptic agents whose only job is to refute it. Not evaluate it — refute it. Is it really available? Is that really the price? Is that beach really sand? When unsure, the skeptic’s instructions were to default to rejection.

Searching produces plausible options. It doesn’t produce true ones. Disproof was the scarce resource.

This sounds paranoid until you watch it work. Three separate times, a “winner” was crowned, written into the plan, published — and then executed by a skeptic agent.

1. The crowned pick was already booked. The listing looked perfect; it was also gone. Availability isn’t a property of a listing — it’s a property of a moment. You have to re-check right before committing, every time.
2. The “verified price” was for a condo that didn’t exist at that address. This one is wild. Load a sold-out Airbnb listing with your dates in the URL and the site quietly shows you a different property — same page layout, a real price, wrong unit. A naive scraper cheerfully reports that price as belonging to the original. Separately, an “owner-direct” booking link silently redirected to a different condo nine miles up the coast. So a number we’d been quoting for days described a place that wasn’t the place. The lesson generalizes to every scraper and every integration you will ever build: a page loading successfully — 200 OK — means nothing. You must confirm the page you landed on is the thing you asked for.
3. “Oceanfront” meant lava rock. One building genuinely fronted the ocean — across a shelf of jagged lava. Another’s “ocean view” was a sliver of blue over a neighbor’s roof. So we stopped trusting listings for that dimension entirely and built our own beach-quality ranking from independent guides, matched to each property by GPS coordinates. Never let the seller grade its own homework.

The small trick that did huge work

One change improved data quality more than anything else: we forbade the agents from answering in prose. For every judgment call, the agent had to pick from a fixed menu:

ocean_view:      yes | partial | no | unknown
swimmable_beach: yes | partial | no | unknown
walkable_to_sand: yes | no | unknown

Two listings that both advertised “ocean view” were forced, under this scheme, to admit partial. Free text would have laundered both into a yes — “stunning ocean vistas!” is a sentence a language model is very happy to write.

The bug nobody wrote

Then the system failed in a way I did to myself. After enough edits across 41 versions, the planning document started contradicting itself. The top recommendation was a property a later section had marked disqualified. The booking checklist calmly instructed the reader to book a $6,204 condo — against a $3,500 budget. Nobody wrote that bug. It accreted, the way contradictions accrete in any long-lived document that many hands (or many agents) touch.

The fix was more agents: an 8-agent consistency audit that chunks the document and asks each agent one question — what here contradicts the current known truth? It now runs before anything ships.

Proving the negative

Late in the process came the best question anyone asked: is a condo even the right category? Maybe the whole framing was wrong. So six agents each priced out a different category, live: timeshare rentals, whole houses, hotel-plus-car bundles, the book-a-whole-month-for-the-monthly-discount play, glamping and eco-stays.

Verdict: the condo genuinely wins — and now we know why. It’s the only category that delivers ocean view and soft swimmable sand and a whole private place under budget. Every cheaper option drops one of those legs. That’s a fundamentally better answer than “here are 40 listings,” because it comes with its own refutation attempts attached.

The constraint I couldn’t write down

Here’s the part I keep thinking about. I did write down a budget and dates on day one — the brief said so, in black and white, and I kept pushing on price the whole way through. What I couldn’t write down was the requirement that turned out to decide everything: the beach had to be soft and swimmable. That rule didn’t exist until an agent showed me a place that was “oceanfront” onto lava rock and I recoiled. The counterexample created the constraint.

(The budget did move — it tightened from “flexible” to a hard $3,500 cap partway through, and across a long, multi-session process the system briefly lost the thread on price. My exact words when I caught it were “you lost the price goals.” That’s a real failure mode of long agent runs, and it’s the machine’s bug to fix — not a mystery about what I wanted. The thing nobody could have written down in advance was the beach.)

This isn’t a story about being bad at writing requirements. This is how people are. You often don’t know what you want until you’re shown something you don’t want. “I know it when I see it,” Justice Potter Stewart famously said in 1964 — about a much spicier topic — and it’s a genuinely accurate description of how we recognize things we can’t define. People are far better at recognizing a bad option than at specifying a good one up front.

Rejections carry more information than requirements.

Which reframes what this tool actually is. It’s not a search engine. Every rejection was a training signal: I couldn’t write down the goal, but I could reliably say “not that, because rocks.” The system was quietly learning my preferences from my nos. If that sounds like how modern AI models are themselves trained from human feedback — it is, and for the same underlying reason. For the things that matter most, people can judge but not specify.

The economics are what recently made this sane. When each re-search costs a weekend, you commit early to a spec you haven’t tested — you have to. When a re-search costs ten minutes and pocket change, “I’ll know it when I see it” stops being lazy and becomes the smart strategy: you can afford to discover your real constraints by bumping into them.

Where the human stays essential

After everything that violated a hard rule had been eliminated, nine genuinely good options remained. Choosing among them meant trading things that don’t convert into each other: beachfront versus an extra thousand square feet versus $500. There is no correct formula for that. Any “weighted score” is just an opinion wearing a lab coat.

So the division of labor ends up clean and, I think, kind of beautiful: the agents enforce the facts; the human judges the values. 249 agent runs narrowed a few hundred candidates to nine. A person picked the final one in about thirty seconds — the roomy place next to the best water.

What we booked

A one-bedroom, two-bath condo. A thousand square feet. 4.92 stars across 185 reviews. Directly across the road from Keawakapu — the highest-rated soft-swim beach in the area by our own independent ranking, not the listing’s. $2,937 all-in for twelve nights, comfortably under the hard budget cap.

And it was never the front-runner. My agents had ranked it third on their own shortlist — behind a beachfront splurge they’d crowned first and a cheaper unit they’d slotted second. I overrode them, because the thing I wanted most — an extra room and the better beach — was exactly the thing they couldn’t score. It was simply the last option standing after every attempt to kill it, plus a human’s taste.

We didn’t find the perfect condo. We disproved everything else.