AI Learning Lab #3 — Designing a Team-Based AI Experiment

Learning Lab #3

Designing a Team-Based AI Experiment

How to pick the problem, size the bet, and know whether it actually worked.

Why The ROI Never Shows Up

It isn't missing. It's unobservable.

Individual

Real value — trapped in one person's chat history. Can't be aggregated, can't be attributed, doesn't survive them leaving.

Enterprise

Too big to experiment on. No control group, and the measurement cycle is slower than the technology cycle.

Team ←

Shared work. Shared context. A bounded edge. The only level where value becomes visible.

The team is the smallest unit where AI value becomes visible — and the largest that can sustain an experiment.

Check-In

What's one new thing AI has been doing for you?

A task, a decision, a small assist, a surprise. Drop it in the chat.

The Mechanism

You can't measure what you replaced. Only what you ran alongside.

Deployment destroys the counterfactual. An experiment keeps it on purpose — and that's where every number in this session comes from.

Two bots we built for ourselves

Penny

Expenses & month-end close

Replaced: monthly expense reports and the chase.

Lives in: Slack. She DMs you.

Live since: July 2026.

Clean data, clear steps

🧩

PrioPals

Team resourcing & capacity

Replaced: a spreadsheet, Trello, and a weekly meeting.

Lives in: Slack. You talk to it in channel.

Live since: May 2026.

Messier data, tangled process

Both built by one person, in evenings, with Claude. One we can give you numbers for. One we can't.

The Spine

Give the experiment SPACE

S

Spot

Painstorm the friction that's already public.

P

Parallel

Run the new way beside the old. Switch nothing off.

A

Assess

Score it against what the human actually decided.

C

Cut over

Only in the zone the evidence earned.

E

Embed

Persist the corrections. Scribe the role.

P · A manufacture the ROI evidence

C · E convert evidence into absorption

SSpot

The pain was already public

#finance

P

PurviJun 4, 9:00 AM

SUBMIT 👏 YOUR 👏 EXPENSE 👏 REPORT 👏 TODAY

Scheduled to post while she was on vacation.

#priopals

M

Mike ArauzMay 12, 2:14 PM

is there a record of past allocation saved somewhere?

There wasn't. That one question started the build.

What makes a good first bet?

Turning pain into value

Data ready →

Clean data, clear steps

You could write out every step in an afternoon.

Penny · start here

Clean data, tangled process

The data's there; the workflow lives in someone's head. Map it first.

Messy data, simple task

Low stakes, but you'll fight the inputs. Clean-up may be the real project.

Messy data, tangled process

Seductive and slow. Not your first experiment.

🕺💃 PrioPals · the harder one

Task complexity →

Your Turn

Put your pain on the 2×2.

Where is your data most ready — and where is the task most complex?

PParallel

We froze a draft before the human finished

What we did

Penny drafted May's month-end close — 314 expenses, $36,992 — and we locked it before our bookkeeper finished hers.

Why it mattered

Two independent answers to the same question. Neither contaminated the other. That's a ground truth you don't control.

The parallel run isn't caution. It's instrumentation.

Penny

AAssess

Half-right — and that was the useful part

~50%

First-pass accuracy across all charges

97.7%

Precision inside the high-confidence band

~36%

On merchants it had never seen before

The score didn't tell us it worked. It told us exactly where to draw the line between what ships unreviewed and what needs a human.

CCut over

A workaround, or a redesign?

#finance

E

EricaJul 1, 11:28 AM

you are (temporarily) off the hook from submitting Expensify reports this month

🎉 9🙌 6

Note the word temporarily. The walk-back stayed open.

And PrioPals?

The weekly resourcing meeting just… stopped appearing. Nobody cancelled it. The spec had said replacing it was a non-goal.

A redesign cuts a step out. A workaround automates the annoyance around it.

EEmbed

We gave the bot a role

The Bot

Maintains a current picture of who's available.

Bot Product Owner

Trains it, fixes it, evaluates its outputs, changes it.

Human in the Loop

Approves the decisions — and brings a belonging lens to them.

#finance

M

MelissaJul 28, 4:12 PM

ok woops- i had been doing that wrong for 3.5 years 🙈

Encoding our policy in an agent revealed nobody had been following it.

The Honest Part

It was not a smooth ride

#finance

A

AlexisAug 4, 1:47 PM

lol, I want it to work so badly! It feels like it could be so much easier. Thanks for working through it Erica!!!

E

EricaAug 4, 2:02 PM

i think I got ahead of my skis on reimbursements 😬 but we will get there!

#finance

L

LeviJul 20, 10:33 AM

If you're not getting terse with Claude are you even pushing into new frontiers of human<>AI collaboration??

😂 7

Adoption is a curve, not a launch. Expect the friction and stay in it.

What Counts As Return

Three kinds of value

Hard

Did the work get faster, cheaper, or more accurate?

Soft

Did the experience improve—and would people choose the old way again?

Strategic

Did the experiment make something newly possible?

Decide which one you're betting on before you start — or you'll declare a working experiment a failure.

Your Turn

What could you run in parallel next month?

Not build. Not launch. Just run beside the thing you already do — and keep the receipts.

What's Next

Pick your next step

Scope one with us

Bring the pain you'd most like to kill. We'll spend a couple of hours helping you design the experiment around it.

Stay in it

Leave your email for the follow-up, the SPACE one-pager, and an invite to the next lab.