Start With the Rule, Not the Model

When I was building insider trade intelligence for ValuFact, I had a question that sounded like a perfect job for a language model. When an executive trades their own company’s stock, why did they do it? Did they buy because they believe in the business, or was it a stock grant, an option exercise, or shares withheld to pay taxes? Only the first kind really tells you something.

My first instinct was the same as everyone’s in 2026: send the filing to a model and ask. It would have worked, more or less. It would also have cost money on every filing, every night, forever. And when it was wrong, I wouldn’t have been able to explain why.

Then I did the boring thing and read the form. The SEC already asks insiders to label every transaction with a code. An open market purchase has its own letter. So does a grant, a tax withholding, an option exercise. A few lines of logic over those codes labelled 97% of filings at zero cost. The model got one small, well-defined job: the leftovers that are genuinely ambiguous. And when even the model isn’t sure, the system says “unclassified” instead of guessing.

That was the most useful AI decision I made on the whole platform, and it was a decision not to use AI.

The Question Is in the Wrong Order

Most AI roadmaps I see start with “where can we use AI?” It’s not a bad question, it’s just in the wrong order. The better question is: what is the simplest thing that works, and where does a model beat it by enough to pay for itself?

Starting with the model feels like progress. The demo is impressive, the slide has the word AI on it, and nobody has to read the form. But it’s a bit like hiring a surgeon to take your temperature. Expensive, slower than it needs to be, and every now and then the surgeon gets creative.

Why Rules Come First

A rule is free on the second call. A model charges you every single time. At ten filings a day nobody notices. At thousands of filings every night, a small per-call cost becomes a line item, then a pricing problem, then a strategy problem. Unit economics are a product decision, not only an engineering one.

A rule can explain itself. When a rule is wrong, you can point to the line that was wrong and fix it. When a model is wrong, you get a confident paragraph. In anything that involves money, orders or someone’s livelihood, “why did the system say that?” is not a nice-to-have.

A rule gives you a baseline. This is the one people skip. A model that is “92% accurate” sounds great until you learn a three-line rule gets 97%. Without a baseline you don’t know if your model is good. You only know it’s busy.

How I Decide Where a Model Belongs

Here is the sequence I follow now. Nothing in it is complicated, the value is mostly in the order:

1. Write the dumbest rule that could work, and measure it. Be a little embarrassed by how simple it is. That embarrassment is useful.
2. Look hard at what the rule misses. Is the remainder large, valuable and actually hard? Sometimes the honest answer is that the leftover isn’t worth a model at all.
3. Give the model only the remainder, with a strict format. In ValuFact every filing type has its own prompt and a rigid JSON schema for the answer. A model with a tight job description wanders less (people too, by the way).
4. Let it say “I don’t know.” A blank is better than a confident guess. Abstaining is a feature, and the interface should show it honestly.
5. Keep a deterministic fallback. If the model is down, slow or over budget, the product should still work, just less cleverly. Users forgive “less clever”. They don’t forgive “broken”.
6. Choose models by measuring them, not by their brand. Replay the same inputs through the alternatives and compare quality and cost side by side. Funny enough, the models running ValuFact in production are not the ones I enjoy coding with. Google Gemini and open-weight models do the production work. The frontier models I use as coding agents would be lovely in production too, and would make the invoice a lot less lovely.
7. Meter everything. Every AI process gets registered, gets a budget and gets a cap. If you can’t say what a feature costs per user per month, you don’t know yet whether it’s a feature or a subsidy.

Where Models Clearly Earn It

I don’t want to sound like a rule fundamentalist. There are jobs no rule will ever do. Reading a 200-page annual report and writing a plain-English summary of what changed is one of them. Noticing that a new kind of risk is starting to show up across a whole industry’s filings is another. For that one I trained a small language model with embeddings, because no if-statement will ever notice a pattern nobody told it to look for.

Rules also rot. A pile of if-statements grows its own maintenance problems, and at some point the rule list grows faster than the value it delivers. That is usually the moment the model has earned its place. Rules first means rules first, not rules only.

The Four Questions I Ask

When I review an AI feature now, mine or anyone else’s, I ask four things before anything else:

• What is the baseline, and how did it score?
• What happens in the product when the model is wrong?
• What does this cost at ten times today’s volume?
• What does the user see when the model says it doesn’t know?

If those have good answers, I’m usually excited. If they don’t, the feature isn’t ready, no matter how good the demo looked.

In Summary

Being AI-native isn’t about putting a model in every box of the architecture diagram. It’s knowing exactly which box needs one, and being honest about the boxes that don’t.

There’s an old saying about not using a sledgehammer to crack a nut. With AI I would add one small thing: first, check whether it’s even a nut.