How to run a 60-day pilot that actually decides something
Health systems run a lot of pilots. Few of them decide anything.
The usual story goes like this. A vendor or an internal team proposes a new tool. Leadership agrees to try it for a couple of months. At the end, the people who liked it say it worked and the people who didn’t say it didn’t. The pilot quietly turns into a renewal, or quietly fades. Nobody learned much.
The problem is rarely the tool. It’s how the pilot was set up.
This matters more in 2026, because so many pilots are now AI. An AJMC study found that 62.6% of US hospitals on Epic had adopted ambient AI documentation by June 2025. Agents for prior authorization, coding and denials are close behind. Each one arrives with a good demo and an offer to try it free for a few months.
A pilot is a test. A test needs something to compare against and a rule for what you’ll do with the answer. Here’s how I’d set one up.
Write the decision down before you start
Before anything goes live, write one sentence: “If we see X by day 60, we will do Y. If we don’t, we will do Z.”
For example: “If the new worklist cuts the time to work a denial by at least a set amount compared with the control group, we roll it out to every team. If not, we stop.”
Get the people who will make the call to sign off on that sentence. The CFO, the department leader, whoever owns the budget. This is the most important step, and the one most often skipped.
When you agree on the bar up front, you take the politics out of the ending. Nobody can move the goalposts after seeing the results, because the goalposts are already in writing.
Pick one main measure. You can track others, but one number decides. If you pick five, someone will find the one that looks good.
Keep a control group
This is the part that makes a pilot honest.
Split the work into two groups that look alike. One group gets the new thing. The other keeps working the old way. Then you compare them over the same 60 days.
Without a control group, you end up comparing this month to last month, or this fall to last fall. A lot changes in between. Volume shifts. Payers change their rules. Either one can make a pilot look great or terrible for reasons that have nothing to do with the pilot.
A control group lives through all the same outside changes. So the difference between the two groups is mostly the thing you’re testing.
How you split depends on the work:
- By work item: half the accounts, claims, or orders go through the new process, chosen at random.
- By team: two similar teams, and one gets the tool.
- By site: two similar facilities.
Random assignment of work items is the cleanest when you can do it. Splitting by team is easier but weaker, because teams differ. If you split by team, pick teams that look alike on your measure before the pilot starts.
Hold the control group steady. The urge to help it along by sharing the new tool or a few tips is strong, especially when the pilot is going well. Resist it for 60 days.
Running the 60 days
Sixty days is short, and that helps. It forces you to pick a measure that can actually move in that time.
A few practical points:
- Measure the starting point. Know where both groups stand on your main number before day one.
- Check that you’ll have enough volume. If the pilot group only touches a handful of cases, the result will be noise. Ask your analyst to estimate this before you launch, not after.
- Don’t change the setup halfway. If you tweak the tool in week three, you’re now testing something else.
- Look at the data at the midpoint for problems, not for answers. Fix broken data feeds. Don’t call the result early.
If you’re testing an AI agent, track one more number: how often a person changes what it produced. That edit rate tells you whether the agent is saving work or just moving it to a reviewer. Also watch for harm, not only gains. Medicare’s own AI-assisted prior authorization pilot, WISeR, is a public example. In Washington state, UW Medicine said approvals that used to take one to three days were taking 15 to 20. And House Democrats wrote to CMS in June asking for performance data they said had not been released. A pilot nobody can see the results of can’t settle anything.
At day 60, compare the two groups on the one measure you agreed to. Then do what the sentence said.
Many pilots will not clear the bar. That’s a good outcome. You spent 60 days and a modest budget to avoid a multi-year contract or a system-wide rollout that wouldn’t have paid off.
The worst result is the one most pilots produce today: no clear answer, so the thing lingers. It costs money every month and nobody can say whether it’s worth it.
Treat a clean “no” as a win, and say so out loud. Teams will run more honest tests when they know a negative result won’t be held against them.
Your next step is to look at the pilots running in your organization right now. For each one, ask the owner two questions. What decision will this make? What are we comparing it against?
If they can’t answer both, there’s still time to fix it. Write the sentence, find a comparison group, and reset the clock.
Discussion