Skip to content
All posts

AI

My AI voted against my own project. I'm publishing the verdict.

· 8 min read

Isometric blueprint line-art of a deliberation chamber: council chairs arranged in a circle around a small project structure at the center, thin glowing lines of scrutiny converging on it over a blueprint grid

Every builder I know has the same complaint about AI: it loves everything. Pitch it a marketplace for left-handed scissors and it finds the TAM inspiring. Describe a note-taking app in a market with forty note-taking apps and it calls your angle "compelling."

I hit this wall while validating one of my own side projects. I asked my coding agent whether the idea was worth the next six months of my nights and weekends, and I got back an enthusiastic, well-structured, completely useless yes. So I asked again, more skeptically. Another yes, now wearing a blazer. After a few rounds I understood what I was actually looking at: every "evaluation" I asked for was just my own optimism, laundered through a model and returned to me with confidence.

That's a dangerous loop for a solo builder. Nights-and-weekends time is the scarcest resource I have, and the thing I most needed — someone to tell me no, with reasons — was structurally unavailable from the tool I was asking.

Why "be brutally honest" doesn't work

The obvious fix is the one everyone tries first: "Be brutally honest with me. Don't sugarcoat it."

It doesn't work, and it's worth being precise about why. The honesty lasts about two turns. Then you push back on one point — as any founder will, because we're all advocates for our own ideas — and the model concedes. You push on another; it concedes again. Within a few exchanges you're back where you started, except now the flattery carries the authority of something that sounded adversarial for a moment. The model isn't lying to you; agreeing is simply what it's optimized to do, and one instruction at the top of a conversation is no match for that gravity.

You can't prompt your way out of sycophancy. The tone is not the problem. The structure is — a single voice, in a single conversation, with you as both the petitioner and the judge of whether the criticism was fair.

So I stopped trying to change the model's tone and changed the structure it argues inside instead.

The Shiproom: seven seats, five rules

The result is a protocol I'm open-sourcing today. I call it the Shiproom — the go/no-go room your idea has to walk out of.

Seven seats deliberate about your project, in sequence, in one session: a CTO, a CFO, a VC, a CMO, a CEO, your target customer, and a Chair who synthesizes. The seats aren't decoration — each one carries a mandate written to kill bad ideas from a different angle. The CFO exists to do the unit math you've been avoiding. The customer seat exists to say "I would never install this" while everyone else is admiring the architecture.

The rules do the work the adjectives can't:

  1. Kill mandates. Every seat must make the case for failure first, before anything else. Praise is only allowed after the seat has genuinely tried to bury you.
  2. A sourced fact base. Before deliberation, the facts get gathered and cited — competitors, pricing, comparable outcomes. Claims without sources are struck. Numbers that can't be verified are labeled ESTIMATE, visibly.
  3. Forced engagement. Every seat must explicitly agree or disagree with a prior seat, by name. This is the rule that stops seven monologues from pretending to be a debate.
  4. Pre-committed verdicts. Every seat ends with a vote — INVEST, SHIP_AND_SEE (run the cheap experiment), or SHELVE — plus a flip condition: the one concrete metric that would change its mind. The thresholds are written down before you know the outcome, and the protocol forbids softening them afterwards.
  5. The unanimity alarm. A unanimously cheerful run is treated as a failed run. If everyone agrees your idea is great, the council didn't work.

The whole thing is markdown plus one HTML file. No accounts, no hosted service, no telemetry. It runs inside whatever agent you already use — Claude Code, Codex, Cursor, Copilot, Gemini CLI — or you paste a single file into any chat and run it there.

Then I made it judge itself

Here's the part where I had to eat my own cooking. Before launching, I ran the council on this very repo. If a tool that promises honest verdicts couldn't produce one about itself, it wouldn't deserve to exist.

The verdict: Ship & See, 5–1–1. Not unanimous — and that's the point.

The VC seat voted to shelve my own project. Its words, which I'm publishing unedited:

"No revenue mechanism, no moat, and I will not launder the vocabulary to avoid saying so. […] My dissent is the system working: if this council could not produce a SHELVE against its own creator, it would be worthless."

The CTO seat opened by pointing out that my moat "is exactly zero" — the entire product can be copied in under an hour — and then named a risk I genuinely hadn't considered: prompt rot. The protocol's bite depends on models actually obeying adversarial ground rules, and that compliance drifts with every model release. Without a regression check, the council would silently degrade back into flattery and die of the exact disease it claims to cure.

The customer seat refused, flatly, to use the product as I'd built it:

"I will not git clone a repo to ask a question, ever; I WILL paste one file."

And the CFO reframed the entire project in one line I've been chewing on since: this is not a business, it's a reputation asset, and its P&L is denominated in credibility, not dollars — so the only real liability is the calendar time it steals from my primary project.

The verdict drew blood, and the product is better for it

Two of those attacks turned directly into features. They weren't on my roadmap; the council put them there.

Because the customer seat refused to clone a repo, there is now a zero-install edition: one self-contained markdown file you paste into any capable AI along with a description of your project. That file is now the primary way to try the tool, not an afterthought.

Because the CTO predicted prompt rot, the repo now ships a drift canary: a fixture project with deliberate, known flaws and a known honest verdict. Run the council on it with any new model — if the verdict comes back cheerful, the protocol failed, and the canary caught it before your real decisions did.

That's the strongest argument I can make for the whole approach: the tool's first verdict materially changed the tool. A flattering evaluation would have changed nothing.

Grill mode: the appeal

A paper verdict can only judge what's written down, and founders are excellent at writing documents that answer the easy questions. So there's an appeal process, and it judges you.

In grill mode, the seats question you directly, one question at a time. Answers must be numbers, facts, names, or decisions — "we believe strongly in our vision" doesn't parse. Dodge a question twice and it gets recorded verbatim in your verdict as an open wound. Answer well, with the numbers you actually know, and seats revise their votes; the Chair re-tallies. It's an appeals court where the evidence is whether you know your own project as well as you think you do.

It's frank, not cruel — the protocol bans insults and theatrics. Specificity is the aggression.

What this can't do

I want to be as honest about the limits as the council was about mine.

All seven seats run on the same model wearing different mandates. The structure — kill-first arguments, forced disagreement, sourced facts, pre-committed thresholds — provably changes what comes out, but it cannot manufacture information the model doesn't have. The council can't replace talking to real customers. Nothing can.

What it does is narrower and, I think, still valuable: it structures the decision, surfaces the arguments you were avoiding, and forces you to write down — in advance, in public if you dare — exactly what evidence would change your mind. Then the market test it prescribes does the rest. My own verdict came with hard numbers attached: if this project doesn't hit its pre-committed thresholds three weeks after launch, it goes to maintenance mode and the hours return to my primary project. I don't get to renegotiate that, and that's precisely the value.

Try it in 30 seconds

The one-file edition: grab SHIPROOM.md, paste it into any capable AI chat along with a description of your project and your real constraints (hours per week, funding, team, goal). That's the entire setup.

The full version adds web-verified fact bases, structured JSON verdicts, a guided interview, grill mode, and an interactive verdict page — it's on GitHub, MIT-licensed: github.com/nicobts/shiproom.

And if you run it: the repo has a public Docket of published verdicts, starting with the one it passed on itself, unedited, dissent and all. PR yours — the SHELVE verdicts especially. Getting roasted well should count for something.

The full verdict the council passed on this project is here: Docket #001. The seat that voted against this article existing is on page one.

AIAGENTSOPEN SOURCESIDE PROJECTS