Is Your AI Pilot Stuck in Training Mode?

Probably, yes, and I say that with love (I ran interRel for 25 years, which works out to a lot of software demos), but let me back up and explain why my website is a fighting game this week.

The new Street Fighter movie opened in theaters on Friday (it’s set in 1993, which means every arcade in it is full of people feeding quarters into a machine one at a time, a business model I’d like explained to me by a CFO someday). So this week the whole of roske.ai turned into a fighting game (the AI agent that rebuilds the site every Saturday at 5AM did the actual building, which is the whole point of having one), and the part I want to talk about is a feature every one of those games has: Training Mode.

If you’ve never played one, Training Mode is a practice room. You pick your fighter, the game drops a dummy in front of you, and the dummy just stands there. It doesn’t block or move, you can turn its health to infinite and the round timer off, and you can practice your special moves on it for 6 straight hours and win every single exchange. It’s genuinely useful (serious players live in Training Mode, because it’s where you learn the inputs… wax on, wax off), but you also can’t lose there, which is sort of the problem.

Now think about the last AI pilot your company ran. Somebody picked a use case, somebody else cleaned up 3 months of data so the demo would work, a hand-picked group of friendly users tried it for 6 weeks, nobody was on a close deadline, and at the end there was a slide with a big green checkmark on it. That pilot was fought in Training Mode, and the dummy stood still the whole time. (Edward, you’ve got a Master’s in Data Science from SMU, with a 4.0 that you’d think would come with better metaphors, and this is how you explain enterprise software, with a 1993 arcade game that half your readers were too young to play and the other half were too busy losing at… okay, fine, moving on.)

The numbers back this up, by the way. Last year MIT’s NANDA initiative reported that about 95% of corporate generative AI pilots showed no measurable impact on profit and loss. People argued about the methodology for months (as people should), but if you work in finance, I’d bet you didn’t argue with the general shape of it, because you’ve probably lived it: the pilot passed, and then nothing much changed.

Once you leave Training Mode and put a real quarter in the machine, the opponent stops being a tidy sample and becomes your actual general ledger, the one with 14 years of mapping exceptions and an account somebody named MISC-OTHER-2 in 2011 that nobody is brave enough to close. Health stops being infinite, because you get one close a month and the board sees how it went. Your users start blocking (they keep the old spreadsheet open in the next window and, whenever the AI and the spreadsheet disagree, they believe the spreadsheet, which honestly is a reasonable call about half the time). And the round timer comes on, which in finance means it’s 6:40 PM on Day 3 of close and intercompany still doesn’t tie out.

I got a version of this lesson last weekend, and it had nothing to do with AI. We’d been planning the first Caribbean AI Summit since at least February, when I first wrote about it here, and on paper we had thought of everything. Then at 7:30 on the morning we opened, we found out every booth needed its own fire extinguisher, with the fire marshals already on their way. In all that planning, not one person had mentioned fire extinguishers. (Dawn drove to Walmart, bought 6 of them, and got back before the inspection! The whole story is in the recap if you missed it.)

None of which means you should skip Training Mode (you’d just lose faster). What I’d do instead is write down, before the pilot starts, what Arcade Mode looks like for this particular fight: a full month of your actual ledger, mess and MISC-OTHER-2 included, the skeptical users and especially whoever built the spreadsheet the AI is replacing, a run during a real close instead of the 2 quiet weeks after one, and a number somebody is willing to sign, plus the result that ends the pilot in either direction, decided in advance and written down somewhere the steering committee can’t misplace it (and if the vendor won’t fight there, ask them why, then write that answer down too).

The other half of it is making sure your AI is reading the real number instead of remembering one that sounds about right, which is the whole reason I’ve been building MCP servers for Essbase and Oracle EPM. An AI that quotes your Q3 revenue to 2 decimal places from memory is guessing, and it’ll guess with total confidence, in a nicely formatted table, right up until somebody in FP&A checks it against Essbase. (Full disclosure, since it’s my own product category: it’s called AgentCube, the awesome people at Caprus sell and support it, and I’m an investor there, so weigh my enthusiasm accordingly, then go ask somebody who doesn’t build them and see if they disagree. Or ask Lillian Buziak, who runs Caprus, but she might be a little biased too.)

My bold prediction, which you can hold me to: within 2 years, the finance teams actually getting value from AI will mostly be the ones that ran their pilots during a real close, and I’d guess a lot of the pilots being announced as successes right now won’t come up in conversation much by then. I could be wrong about the timing (if so, write me in October 2028 and gloat, I’ll have earned it).

And if you’re putting together a conference or an offsite for finance people and you’d like somebody to stand up and say all of this out loud (with slides, and a lot more parentheticals), that’s what the speaking page is for. Or just write me at Edward@Roske.AI and tell me about the pilot you’re running right now and which mode you think it’s in. I really do want to know.