A while ago I built a decision engine for an academic system: a set of rules that approved or rejected requests automatically. At the time I was proud of having encoded not just the procedure’s logic but also its context. With some distance, what interests me most about that project is something else: what it taught me about the limits of systems that decide on behalf of people.
A mirror of our rules
My algorithm wasn’t “intelligent” in the sense the word carries today. It didn’t learn or infer patterns from large datasets. It was a fast, efficient mirror of a set of rules I had defined. If a rule was fair, the system applied it with total consistency. If a rule was flawed, it repeated that flaw at a scale and speed no person could match.
That describes more technology than it might seem: a good part of the systems making real-world decisions today aren’t emergent intelligence, but the automation of our own processes, biases included. We teach the machine to decide like us, and it inherits our blind spots too.
The most important rule was not deciding
Today I use language models that write code, summarize texts, and hold conversations with surprising fluency. They process and connect information at a scale no human can reach.
And still, my rules engine had something those models don’t ship with: explicit limits. One line of my code said, in practice: “if the reason for the request is a health issue, don’t try to decide; pass the case to a person.” That wasn’t a data-driven decision, it was a design decision: there are cases an automated system shouldn’t resolve.
A model can analyze millions of medical records and estimate the probability of a disease, but it doesn’t know what receiving that diagnosis means. It can evaluate financial data and approve or reject a loan, but it doesn’t register what that loan fixes or breaks in a family. Its understanding is statistical: it processes the words, not the situation behind them.
The criterion I kept
From that project I kept a design rule I still apply: in any automated system that affects people, the escalation path to a human is one more feature of the system, and it deserves the same care as everything else. It’s not a patch or an admission of defeat; it’s defining from the start what the system can resolve and what isn’t its call.
AI still strikes me as one of the most powerful tools we have, and I use it daily. But after building a tiny version of it, I treat it as what it is: a system with limits, one that works best when those limits are written into the code instead of discovered by accident.