AI Can’t Be Agile for You: The Hidden Risks of Over-Reliance on AI

I wrote a piece back in May about the human capabilities that machines cannot replicate. This is the uncomfortable follow-up, because it is one thing to say those capabilities matter and quite another to notice how quietly you can put them down.

AI is doing incredible things in the business world right now. I use it every day and I would struggle to go back. So, this is not the piece where I tell you to stop, but over-reliance has risks attached, and most of them are not the ones we talk about.

Let me start with 19 days in June.

On 12th June, the US government issued an export control directive covering two of Anthropic’s Claude models, Fable 5 and Mythos 5. The order stated that no foreign national anywhere in the world could have access. This sounds fairly narrow until you realize that nobody can verify nationality in real time across Amazon Web Services (AWS), Google Cloud, Microsoft Foundry and half a dozen other platforms at once; at which point the only way to comply is to switch the whole thing off.

That is what Anthropic did.

They did it that same evening, for everybody, with no warning or date for coming back. Access was restored on 1st July, once the Department of Commerce lifted the controls.

That is 19 days. In the scheme of things that is nothing, and businesses survive worse all the time.

The reason I keep coming back to it, however, is that for 19 days a great many organizations got a look at something they would probably rather not have seen, which is how much of their own work they could still do. Some were fine, because they had people who could pick the work back up and get on with it. Others found that they had built an entire process around something they no longer owned, and there was nobody left in the building who remembered how to do it ‘the long way round’.

Which raises the question: if your provider revokes access tomorrow, or raises the price beyond anything you can justify, what then, if it is deeply embedded in your business?

Fable was the loud, obvious version of the risk, the kind that makes the news and forces a reaction. The version I am more worried about is the quiet one, the one that arrives without a government directive or a press release, and that is happening in ordinary teams on ordinary Tuesdays, one unchallenged output at a time.

What does over-reliance on AI mean?

Over-reliance on AI is the point at which people accept what the AI gives them without meaningful scrutiny, and gradually lose the ability, or simply the appetite, to do that work themselves.

It has very little to do with volume of use as is often assumed. Plenty of people use AI all day long and stay completely sharp, largely because they argue with it. What matters is how hard you push back on what comes out.

The awkward part is that from the outside, a team using AI brilliantly and a team over-relying on it can look identical. Same output, same pace, same standard, near enough. You only find out which one you have on the day something arrives that the model has never seen before.

Speed without learning was never agility.

Why do people become dependent on AI?

Two reasons, and neither of them is that people are lazy.

The first is cognitive offloading, which is a slightly fancy name for something you have been doing your entire life. You offload to a calculator, to your calendar, to the satnav, to the colleague who knows all the pricing rules off by heart. It is sensible and it is efficient and there is nothing much wrong with it, right up until the point where your brain, which cannot tell the difference between “I have handed this task over” and “I have given this skill up”, quietly stops volunteering for it.

The second is automation bias, which is the tendency to trust what a machine tells you over your own judgement, even when your judgement is better and even when there is evidence in front of you saying the machine has got it wrong. It was first documented in cockpits and hospitals, where the consequences made it impossible to ignore. Generative AI has now brought it to the rest of us at our desks, where it is considerably harder to spot, because the answer arrives beautifully written and most of us tend to read a well-written sentence as a correct one.

There is a piece of research worth knowing about here. In April, the American Psychological Association published a study of 1,923 adults across the US and Canada, who were given ten simulated work tasks and told to use commercially available AI tools to get through them. This involved planning with half the information missing, interpreting ambiguous data and explaining the reasoning behind a strategic decision.

The sort of things most of us do before lunch!

Afterwards, 58% agreed that the AI had done most of the thinking, and those same people reported feeling less confident in their own reasoning and less like the ideas they had just handed in were really theirs.

That figure needs a couple of caveats that a fair amount of the coverage glossed over, because they make it less alarming than it first sounds. The study was correlational, which means it can show the pattern but cannot prove that AI caused it, and the finding most people wanted, that AI use was making us less intelligent, was not there at all.

The part that stopped me was what separated the two groups. The people who argued with the AI, or rejected what it gave them outright, came away more confident and with a far stronger sense that the work was theirs. The researchers’ summaries it as ‘the problem was not AI use; it was passive acceptance’.

Which I find reassuring because it means nobody has to give the tool up in order to fix this.

The hidden risks of over-reliance on AI

Skill atrophy

Skill atrophy is real, and it is boring. Nothing dramatic happens. People stop flexing certain skills, and the result is that they lose the ability to do something they used to do very well. A copywriter who stops drafting loses their ear for it. An analyst who stops building the model loses their instinct for when the numbers are not telling the truth. The reason it is so difficult to catch is that the work carries on going out of the door, on time and to standard, the entire time it is happening.

The erosion of critical thinking

Over-reliance on AI erodes critical thinking, though not by breaking anything in anyone’s head. All it does is remove the occasions where you would have used it. Critical thinking is a habit long before it is a skill, and habits need repetition; so, if the first draft and the first analysis both arrive already formed, where are the reps coming from?

There is a version of this for people starting out that genuinely worries me.

AI is taking on a great deal of the lower-level work, and the lower-level work is where people have always earned their judgement. Nobody develops an instinct for when something looks wrong without having built a few hundred of them by hand first. If we automate away the tasks that would normally be carried out by someone early in their career, we could be looking at big skills gaps in a few years’ time on things we currently assume everybody can do (which is a whole other article, and one I suspect I will end up writing!).

The oversight gap

AI continues to develop at pace, you implement it at scale because everybody else is doing the same, and suddenly you are lacking in control and cannot quite say when that happened.

So, do you have enough people to be checking the outputs? Checking that communications are correct and in the right tone and voice? Checking that workflows are running as they should and that data is being collected and stored in the right way?

These are not exciting questions, and they never make it into the business case, but somebody has to be able to answer them. While it is tempting to assume that automation has removed the risk, if you scale the work faster than you scale the checking, all you have done is move that risk somewhere less visible.

Dependency

And then there is the Fable problem. Deep integration feels like commitment and maturity right up until the moment the terms change, and the more thoroughly you have embedded something you do not control, the fewer options you have when it moves.

Which brings us to the part I actually wanted to write about.

How over-reliance on AI undermines agility

AI learns from history, and disruption tends to show up in the signals that do not match history, which makes any model a very clever and deeply persuasive argument for the recent past; superb at the things we already know about and more or less blind to the thing that is about to matter. On its own that would be a manageable problem, because you would keep a human in the room to watch for what the model cannot see and carry on much as before. What makes over-reliance more serious than a simple blind spot is that it goes after the very machinery you would otherwise use to correct one.

Think for a moment about what agility really is beneath all the frameworks and ceremonies. Every agile practice is, at its core, a mechanism for making it inexpensive to challenge the current answer. That is what retrospectives are for. Reviews. Timeboxes. Backlogs that can be reordered. The entire agile toolkit exists to prevent organizations from becoming committed to a plan simply because it already exists, or because someone senior once declared it in a meeting.

Agility was never about going faster, despite what the word has come to apply. It is about how cheaply and how quickly you can discover that you are wrong and then pivot.

Now drop AI into the middle of that.

The answer arrives before anyone in the room has formed a view of their own, and it arrives fluent, which matters more than it should, because fluency is about the most reliable way there is to stop somebody asking a question. It comes with the subtle authority of having read more than any one person could, and it comes fast, so that accepting it costs you nothing while challenging it starts to feel like friction you are introducing on purpose. The result is that fewer questions get asked, alternatives never quite make it onto the table, and assumptions set like concrete without anyone ever deciding that they should. The practices carry on, of course, because the meetings still go in the diary and the board still gets its slide, but the challenge has gone out of them, and the challenge was the entire point.

You can run every ceremony in the book and still stop being agile, because agility isn’t the ceremony; it is the willingness to change your mind, and if something else is doing the deciding then there is nothing left for you to change.

A retrospective where nobody is curious is really just a box-ticking exercise, and a review where the recommendation has already been written is a formality dressed up as a decision. An organization that has unwittingly automated its own capacity for doubt has not become more adaptive, whatever the metrics happen to say; it has simply got faster at going wherever it was already heading.

There is a governance problem sitting inside this as well. Agile governance works by keeping decisions visible and reversible. An AI recommendation that nobody can explain, adopted because it read well, fails on both counts at once.

What this means for you

The practical version of all this is smaller than you might expect.

Don’t jump to the tool as your first reaction

Form a view before you ask for one, and it honestly does not need to be a good one, because even a rough and half-wrong view gives you something to hold the answer up against, and it is that act of comparing that keeps your judgement in working order. If you go to the tool first every single time, before you have thought about it at all, then you are not really using AI to extend your own capability so much as renting a capability you have stopped maintaining.

Pay attention to the moment you stop checking

Learn to notice the moment. There is a very particular one I have caught myself in more times than I would like, where you skim something that AI has produced, it looks about right, and you send it on. That’s not because you actually checked it but because nothing in it jumped out at you. That is the real risk in a nutshell. The danger is not the day you knowingly wave through something poor; it is the far more ordinary day you approve something without properly reading it because it read smoothly enough to pass. It never feels like a decision, which is exactly what makes it one worth watching for.

Keep one skill entirely your own

Keep one thing manual. Pick something that genuinely matters to you and that you would be embarrassed to have lost or something that you have done the long way round often enough that you would notice the day you started to slip.

For me it is writing; I will happily let AI research and check and tidy, but the first draft of anything that carries my name I still write myself because the day I cannot is the day I have lost something I actually care about keeping.

How organizations can use AI responsibly

For organizations the challenge is harder, because the incentives all point the wrong way.

Reward the pushback, not just the output. If your team is measured purely on how much comes out of the door then they will take the first draft every time, and it is hard to blame them for it, so the fix is to make “what did we reject, and why?” as normal a question in a review as “what did we ship?” already is.

Build AI literacy rather than AI enthusiasm. People need to understand where the model’s confidence comes from and where it is most likely to be subtly wrong, and that is a learning investment rather than a licence you buy, which is the part most organizations skip.

Keep the capability warm. Think back to those 19 days, because the whole point of agility is having options, and you only have options if somebody in the building can still do the work another way when the tool you relied on is suddenly not there.

Name the human in the loop. Not as a line in a policy that everybody nods at and nobody reads, but an actual named person who has both the time and the authority to look at what AI has produced and say no to it.

And be honest with yourself about the maths, because if you have taken headcount out on the promise that AI now covers it, you have not just removed a cost from the spreadsheet, you have also removed the very people who would have caught the thing when it went wrong.

A closing thought

AI can take a great deal off your plate, and it should. However, it cannot be curious on your behalf, and it has never once changed its mind because something “felt off”. Those are the human bits. They are also, inconveniently, the agile bits.

AI can’t be agile for you; that part was always going to be down to us.

That’s why critical thinking matters more than ever. If you would like to strengthen the skills that keeps AI honest, explore our short course on Exploring Critical Thinking today.