AILOS← All modelers

AI Leadership Operations / 02

Pro

Where do humans need to check the AI?

Cost-optimal review rate vs the floor your situation actually requires — and which one's binding.

About this tool

About this modeler

Every AI workflow needs a review policy — and most are set by gut. This modeler shows total cost (review effort + cost of escaped errors) across 0%, sample, and 100% review, and surfaces the optimal sample rate the math actually points to. Use it when you're designing a new AI process, auditing an existing one, or defending an oversight policy to compliance or legal.

When to use it

  • Designing an AI-in-the-loop workflow
  • Justifying a sampling review rate to compliance
  • Auditing an existing review policy that feels off

Related tools

Blast radius

Reversibility

Sensitivity

Novelty

Who owns it

Owner seniority

Your numbers

What AI produces

Volume, error rate, and what one mistake costs.

The review

What review costs and how well it catches mistakes.

Pro modeler

Read the framing free. Run the numbers on Pro.

Inputs are locked on the Free plan. Upgrade to run every modeler with your own numbers.

Upgrade to Pro →

Sample numbers shown. Upgrade to Pro to edit and save your own.

Annual savings from review gate

$728,000

Review every AI output. That gate is worth about $728K a year in mistakes you don't ship.

Gate everything · $728K/yr saved

Annual savings: full review / recommended / cost-optimal

Pessimistic
$728,000
Realistic
$728,000
Optimistic
$728,000

Key numbers

No review · weekly cost$20,000
Your sample (25%) · weekly$16,500
Full review · weekly$6,000
Cost-optimal review %100%
Recommended review % (with context)100%
Reviewer hours / week (recommended)33.3 hr

What to do next

  1. 01

    Tag every AI output by risk (customer-facing, high-$, regulated) before it hits the queue.

    Owner · AI / pipeline
  2. 02

    Set the review rate at 100% — pick the risk tags, not a random sample.

    Owner · the reviewer
  3. 03

    Track every miss with a one-line reason; revisit in 4 weeks.

    Owner · the reviewer

Save this scenario

Create a free account to name, save, and revisit this run from your dashboard.

Save it →

Plain-English read

Baseline read

500 AI outputs/week at 10% error rate. Skipping review entirely costs $20K/wk in mistakes downstream.

25% sample-review costs $750/wk; full review costs $3,000/wk and catches ~85%.

Optimal review rate: 100%. That's 33.3 hours/week saving $728K/yr vs no gate.

Gate every output. Re-evaluate when error rate drops below 5% or stakes drop.

Recommended process

  1. 01

    Route every AI output through a triage queue

    Owner AI / pipeline

    Triage tags each output with risk markers (e.g. customer-facing, high-$ value) so the gate isn't random.

  2. 02

    Review 100% of outputs — risk-weighted, not random

    Owner the reviewer

    Cost-optimal rate balances $3,000/wk reviewer time against $3,000/wk leakage.

  3. 03

    Catch rate target: 85%; log misses with reasons

    Owner the reviewer

    Tracking misses turns the gate into a feedback loop instead of a checkbox.

  4. 04

    Re-rate the gate every 4 weeks

    Owner the reviewer

    If AI error rate drops or stakes shift, the floor and the cost-optimal rate both move.

Share this decision

  • 500 outputs/wk × 10% × $400 = $20K/wk leakage at zero review.
  • Reviewer 90/hr × 4min × 85% catch rate.
  • Cost-optimal: 100%. Recommended (with situation): 100%.
  • Annual savings vs no-gate: $728K.
  • Situation: blast: med · rev: costly · sens: routine · nov: trodden · owner: mid · reviewers: 1.