A mate rang me a few weeks back. Sharp guy, runs a good business, not remotely a technophobe. He'd had a health thing come up — nothing dramatic, but the sort of question you want a straight answer to — and before he saw anyone about it he did what most of us do now. He asked the AI.
Then, because he's the sort of bloke who reads the terms and conditions, he asked a second one.
Two different answers. Not slightly different. Materially different numbers, pointing in different directions. And here's the part that got him: both of them were beautifully written. Calm, structured, confident. Neither one said "look, I'm not certain about this." Neither one flagged that a reasonable person might land somewhere else entirely.
"How am I supposed to know which one's right?" he said.
I didn't have a great answer. Because the honest one is: you're not. Not from reading them.
The bit that should worry us
Now, that's a health question, and most of us would go and see a doctor anyway. Fine.
But sit with the mechanism for a second, because the mechanism is the thing.
He asked a question with a checkable answer. Something a specialist would settle in about ninety seconds. And the AI still gave him two confident, well-written, contradictory versions of it — with no signal at all that it was on shaky ground.
Now think about the questions you're asking it.
Should we take this client on? Is this price right for this market? What's the risk in this contract? How do we structure this deal?
None of those have a checkable answer sitting on the internet. There's no specialist who settles it in ninety seconds. The right answer depends entirely on your business, your market, your customers, and about fifteen years of things you know that nobody's ever written down.
And the AI will answer with exactly the same confidence it used on my mate's health question.
That's the trap. It's not that the thing lies to you. It's that it sounds precisely the same when it's right as when it's got no idea.
Well-informed. Completely wrong.
I've been watching a version of this play out across businesses I'm involved with for a year now. Same shape every time.
Picture a Hamilton engineering firm. Around $6M, good reputation, twenty-odd staff. The owner asks for a pricing recommendation on a new service line. Back comes something genuinely impressive — market context, three tiers, sensible-sounding logic, a nice tidy structure. Reads like something you'd pay good money for. I've charged for worse.
Except it's pitched at a market that's three times the size of theirs. It assumes a sales cycle they don't have. It doesn't know that their two biggest clients came through the same referral source and would both walk if they moved to that model. It has no idea that they tried something similar in 2019 and it went badly for reasons that had nothing to do with the price.
The answer isn't stupid. That's what makes it dangerous. It's a well-informed answer about a business that isn't theirs.
And if you're busy, or it's Thursday afternoon with four other things on — it reads like the answer.
Then there's the smaller, sneakier version of the same thing. You catch the error. You correct it. "No, that's not right, our clients don't buy that way, here's why." Good correction. Real insight — the sort of thing that took you a decade to learn.
Next session, it's gone. You explain it again. And again the session after that.
I've done this. More times than I'd like to put in writing.
It's not broken. It's just never met you.
And here's where I'd dig my heels in. Because the obvious conclusion is the wrong one.
The easy conclusion is "AI's unreliable, don't trust it." I don't buy that, and neither should you. I use these tools every day and they've genuinely changed how I work. The capability is real and it's improving faster than anything I've seen in my working life.
The problem isn't capability. The problem is that it's answering a question about your business having never been anywhere near your business.
Think about what you'd have to explain to a very good new hire before you'd let them near a pricing decision. Not the industry stuff — they'd know that. The other things. Which clients are actually profitable and which ones just look busy. Why you walked away from that contract last year. What your team can genuinely deliver versus what the website says. The customer who's worth keeping at a bad margin because of who they talk to.
You'd spend weeks on that. And you'd expect them to be a bit useless until you had.
We don't do that with AI. We open a window, ask it something that would stump a smart outsider, and then feel let down when the answer is what a smart outsider would say.
It didn't fail. It answered the question we actually asked, which is a general question, and we heard it as a specific one — because it was written well enough to sound specific.
The question I'd sit with
Using AI to do yesterday's work faster isn't the game. Execution is getting repriced, while judgement stays yours. I still think that's right.
But there's a follow-on, and it's been nagging at me.
If judgement is the valuable half... how would the AI ever get access to it?
Everything that makes an answer right in your business is currently sitting in one place. Your head. Not in your files, not in your CRM, not in the folder of proposals — those hold what you did, not why you did it. The reasoning never got written down. It never needed to be, because you were always in the room.
So here's the one I'd put on the whiteboard:
Of the last five real decisions I made — what would someone have needed to know to get them right? And where is any of that written down?
If the answer is "nowhere, it's just in my head" — you're in good company. That's most owners I talk to. But it's worth knowing that's the actual gap, because it means no amount of better prompting is going to close it, and no new model release is going to close it either.
It's not a tool problem. And it's not going to fix itself while you're not looking.
Good luck out there. And don't forget to have fun!
