When everything works… and we’re still going wrong

#

There are few moments more dangerous in a tech organization than the one where everything seems fine.

No major incidents.
Projects are delivered.
Metrics go up.
Dashboards are green.

The black box with a smile answers quickly, teams move forward, and someone says, with genuine relief:

“Looks like we’ve figured it out.”

And that’s where the real problem begins.

When something fails in an obvious way, at least there’s a signal.
A production error, an outage, a complaint, an alarm that forces you to stop and think. Failure introduces friction, and friction forces understanding.

But when everything works, no one wants to ask uncomfortable questions.

If the code compiles, tests pass, and the deployment completes, why review further?
If the customer isn’t complaining, why look inside?
If the AI answers well, why challenge it?

The box’s smile is calming.
And calm, in complex systems, is often deceptive.

Let’s imagine a situation that, as always, doesn’t happen in real life.

A team has been using AI for months. The pace is good. Things ship. Times have gone down. Juniors are productive. Seniors review quickly. Nobody seems especially stressed.

From the outside, the system works.

From the inside, small signals start to appear: there are parts of the code nobody wants to touch,
certain decisions aren’t fully clear,
some explanations sound… vague,
and the phrase “AI generated this” shows up more and more.

None of that is critical. Yet.

The problem with systems that “work” is that they hide deterioration.

Technical debt doesn’t usually break anything all at once.
It accumulates quietly.

Loss of judgment doesn’t show up in the short term either.
It just becomes harder and harder to reason about the system without calling on the box.

Dependency settles in without drama.
Not because AI is indispensable, but because it’s uncomfortable to do without it.

And since there’s no immediate pain, no one prioritizes it.

Here’s something especially perverse: success validates the model.

If everything works, the narrative reinforces itself: “we’re seeing results,”
“adoption is going well,”
“the cultural change is advancing.”

Any doubt starts to sound like resistance.
Any objection sounds exaggerated.

The black box with a smile doesn’t argue.
It just keeps answering.

This is the moment when the organization starts to confuse itself.

It confuses stability with understanding,
the absence of visible errors with quality,
speed with maturity,
and functioning with health.

But a system can function for a long time without being healthy.

Especially if those maintaining it have stopped fully understanding it.

There’s a clear sign that something is wrong, even if it’s hard to admit.

When an organization starts to rely more on post-hoc explanations than on prior reasoning, something has broken.

Decisions are no longer made because they’re understood,
but because “they make sense when explained.”

The box helps a lot at that point.
It can always generate a plausible justification.

And that’s dangerous.

Because in complex systems, serious problems rarely come from obviously bad decisions. They come from reasonable decisions made without fully understanding their consequences, accumulated over months or years.

Each one works.
All of them together, don’t.

And when the failure finally shows up, it usually appears far from the point where the original decision was made.

This is where the box metaphor stops being friendly.

The box doesn’t remember past mistakes.
It doesn’t learn from organizational incidents.
It doesn’t carry consequences.

It just keeps smiling.

And when something truly fails, nobody really knows where to start looking, because understanding has been eroding little by little, right while everything seemed to work.

This is the hardest scenario to fix.

There’s no crisis to justify it.
There’s no clear urgency.
There’s no obvious “before and after.”

Just a vague sense that every change costs more,
that every decision needs more validation,
that each system is a little more opaque than the last.

And yet, the metrics are still fine.

Maybe AI is genuinely helping us.
That’s very possible.

But if we only celebrate that everything works, without asking why it works, who understands it, and what we’re losing along the way, we’ll be optimizing for the short term.

The black box with a smile is very good at making things move forward.
It’s not good at ensuring we move in the right direction.

The real risk isn’t that AI fails.
The real risk is getting used to everything working without understanding it.

Because when that happens, the day it stops working—and it will—
we won’t know whether the problem is the box, the system…
or the organization that stopped asking questions while it still could.

Note: This article is part of a series on the real use of AI in professional environments. Sometimes the biggest symptom of a problem isn’t the failure, but the absence of clear signals while deterioration advances.