Why AI pilots stall, and the four questions that unstick them

Most AI pilots we're asked to rescue work fine technically. The demo impressed people. Then the pilot ran for three months, usage drifted down and nobody could say whether it had paid off. The problem was rarely the model. It was that nobody agreed up front what 'working' meant.

Before we build anything, we ask four questions. They take an afternoon to answer and they save months.

1. Whose week gets better?

Name the role, not the department. 'Customer service' is too broad. 'The two people who triage the shared inbox each morning' is specific enough to design for, and specific enough to measure.

2. What does it replace?

If a new tool doesn't replace a step, it adds one. Write down the current steps, then cross out the ones the tool removes. If you can't cross anything out, keep looking.

3. How will we know it's right?

Collect 30–50 real examples with the answer you'd expect. That's your evaluation set. It turns 'the AI seems good' into a number you can track every time the prompt, model or data changes.

# A tiny evaluation loop: run every example, score it, keep the misses
results = []
for case in eval_set:
    answer = assistant.run(case.question)
    results.append({
        "id": case.id,
        "passed": grader(answer, case.expected),
    })

accuracy = sum(r["passed"] for r in results) / len(results)
print(f"Accuracy: {accuracy:.0%} on {len(results)} cases")

4. Who owns it after launch?

Someone needs to read the feedback, update the examples and decide when it's good enough to expand. If that person doesn't exist, the pilot will stall when the project team moves on.

You don't need more AI tools. You need the ones you have working, measured and owned.

None of this is complicated. It's just easy to skip when a demo looks good. If you'd like a hand working through these for your own use cases, that's what our two-week Readiness Sprint is for.