In the three and a half years since it hit the public consciousness, I think it's fair to say there's no question AI's demonstrated it can have real impact. Can it deliver real value? That question is a bit less clear; and the answer seems to be the ever-popular "it depends".
It is in my view also the wrong question. A better question is can it deliver enough value to justify the cost and earn a seat at the table? The answer here is also "it depends". Not all of the "depends" are in our control, but much is, and those things are largely non-technical.
One thing that's become clear is that the model providers are heavily subsidizing the cost of AI (source). At some point, that's going to stop. It has to: a loss-leader strategy only has limited legs. We're already seeing signs that the providers are beginning to take action: after hooking users on all-you-can eat, Anthropic is moving to metered pricing (source). The pure consumption-based costs are largely out of our hands.
Right now, AI is also based on a heavily debt and circular financing model (source). That too only has the legs for a limited run. Who survives is yet to be determined, and is similarly out of the hands of AI users.
In my opinion, OpenAI is most likely the first to fall, despite the too-big-to-fail financial ecosystem it's trying to create. Google is probably the last man standing because they're really the only provider that has successful and truly core capabilities in every element of the holy trinity (Chips, Frontier Models, Scaled Infrastructure), and they have the added advantage of having products that are in themselves, destinations that leverage AI.
Everyone else provides a commoditised substrate, where the cost of switching is relatively low. Sure AI may become a ubiquitous metered utility, as Altman has predicted, but like other utilities - electricity, ISP's etc, there's no real stickiness; loyalty is predicated on whichever model is best right now, and if that's non-differentiating, then on price.
So those two things: per-unit cost, and provider viability are uncontrollable inputs. They're also maybe the least meaningful. My view is that cost is largely the factor that primarily increases the importance of focusing on sustainable value.
There's ample evidence that AI increases the speed of coding, but also growing evidence both that the increase in speed comes at its own cost, and that the bottleneck is not coding speed, but everything outside of that, with the largest bottlenecks downstream. If the processes are not re-engineered, AI won't deliver the value it should. We're already seeing the costs are rising exponentially relative to the value realised (source).
Ok great - but not everyone's a coder or in tech, right? Set aside that virtually every business now is a tech business, let's focus on the non-technical considerations.
Evidence is already emerging that using AI to drive headcount reduction doesn't generate the value and ROI expected, as Gartner, amongst others, reports (source).
We're also seeing the impacts of AI's limited actual insight (i.e. stochastic probabilism). I'm not going to call it AI slop, but Workday found that 37% of AI productivity gain is offset by rework (source). Will 63% efficiency in AI-led gains be enough to justify AI spend when costs are increasing in a non-linear way? The jury's very much out on that.
We also know that agentic has - to be generous - less than perfect effectiveness at task completion.
One study found around a 30% success rate in autonomous real world tasks, with significant deficits in key areas, such as tasks involving communication and collaboration (source). Another found that LLM's tend to degrade work product over multiple iterations, as would be the case in an orchestrated feedback-driven agentic loop (source). Hallucinations (large and small) remain a problem, with eliminating them a mathematical impossibility.
"Thanks for dragging down the vibe, Debbie Downer. So what do we do about that?" I hear you say. Short answer: We focus on ROI and maximising value relative to cost.
- Pick the right area to leverage AI. It's not everywhere. Code generation (with significant caveats), and rough edge (pseudo-deterministic) tasks with limited need for human insight and/or small blast zones are good candidates for AI. Sharp edge tasks (with very clear, deterministic outcomes) are better handled by conventional programming, not high-cost AI agents.
- Focus on value. AI may deliver efficiencies, but you should use those savings to increase value-generating outcomes, not just cut heads. McKinsey found that high performing companies are about 60% more likely than other companies to set AI-driven growth and/or innovation goals (source).
- Transform your processes around AI - don't just apply it to automate your current ways of doing things. McKinsey similarly found that high performers are nearly three times more likely than others to be transforming their work processes around AI.
- Manage your costs. Think very carefully about where and how to use AI, including optimising your prompts. Agents use a median of 96,000 tokens, and half exceed 128,000 tokens (source). In a world where the costs of those tokens needs to increase by an order of magnitude for suppliers to finance debt and break even, those agents are going to become expensive fast. You better be generating enough value to more than offset those costs, or be in a position where those agents are expendable.
- Value comes with non-financial costs - i.e. Risk. Manage your debt and risk carefully, particularly:
- Cognitive debt - building up outputs where you don't fully understand the output - progressively increases the risk of future catastrophic failure with elongated response and recovery times.
- Generative debt - where the body of knowledge that your agents use to generate content is progressively poorer quality (because every uncaught hallucination degrades the corpus) increases the risk that what generates positive ROI now, may not do so in future. This is particularly true in code, where existing technical debt can be interpreted as "good" product by the model, thus cyclically propagating poor quality into a growing debt estate.
AI has a place at the business table, but it has to be earned. Not through the capabilities of the models themselves, but by where we use it, how we use it, whether we make it intrinsic to process, or try to bolt it on to what we do today, and whether we really manage cost and seek value equally fervently.
And finally - AI pennies are going to drop. But not all the pennies. It's probably less important to place bets on which pennies will stay up, as it is to make sure that you're ready and able to change AI providers when the pennies hit the ground. Right now, Gemini's not the top model in many cases, so there's no need to rush to it, but as I mentioned earlier, it might be the last one standing. Expect bubbles to burst, and be prepared to move with the same agility you expect of your development teams. Just expect more rework and adjustment along the way. I don't see any foreseeable future that involves AI becoming set and forget, even if you don't change providers. It's very much in the realms of possibility that rip and replace could become the new norm.