Developers predict they're going to be 24% faster when using AI, report that they were 20% faster when using AI, but were in fact 19% slower to complete tasks.
The detail is more nuanced of course, and more research is needed to extrapolate these results. Some of the possible causes (yes, correlation is not causation, but I'm going to use it as shorthand) are likely to get addressed with the progression of the tech. For example, decreased performance over large repos will likely improve with larger contexts, and wait times will decrease. Others are likely to be more sustaining: developers experienced and familiar with the codebase are more adversely impacted (since presumably they have insight the AI cannot).
I'm neither an AI idealogue nor an AI skeptic. I think as a technology, it has undefined upside, and unknown risk. Anyone who has worked with me will tell you I like data, and research has given us a lot of warning signs along with indicators of positive impact.
Humans inevitably bring vibes to their evaluation of tools, leading to the Gartner hype cycle, and the race to "value".
My sense is that the tech industry is treating AI as a blunt tool. We're using it to try and goose productivity, may not have a sense of good, objective measures to determine if we're achieving it, and very nascent understanding of how, and in what circumstances upside is maximized and downside is minimized.
I think over time we'll build insight into how to use them effectively - with confidence backed by more sophisticated portfolios of metrics that will be useful to assess the real impacts of AI on a sustaining body of work. And that we'll see an expert class of folks who help others succeed based on practical experimentation and observation.
I've been watching Kent Beck's experimentation with AI Genies - and his offering the outputs for practical evaluation. It seems likely to me that folks like Kent, in approaching it this way, will become the next well, Kent Beck, or Martin Fowler, bringing some of the same approaches to AI as we've seen in antecedents like design patterns and methodological evolutions (XP, TDD anyone?).
Full study is here: https://lnkd.in/gQXTsMZ6
Originally published on LinkedIn.