Write the Pass Mark Down Before You Look

Imagine a student who grades their own exam, and only decides what the pass mark is after seeing the score. They would never fail. They also wouldn’t learn very much.

That is, more or less, how a lot of teams evaluate launches, experiments and AI models. The target gets written after the dashboard is open. A disappointing number becomes “an encouraging early signal”. A lucky one becomes a strategy. Nobody is lying, exactly. It’s just that every small choice gets made while looking at the answer.

I have done it too. So over the last year I made a rule for myself: write the pass mark down, with a date on it, before you look.

The Most Useful Result I Got This Year Was a Negative One

Alongside building ValuFact, I ran a research program on a simple-sounding question: could the data we collect time stocks better than simply buying and holding them?

I tried a lot. Neural networks, evolution strategies, gradient-boosted trees, models that read the text of filings. Before each round I wrote down what would count as a pass, against which baseline, after which costs. I ran random “null” signals next to the real ones. I used several random seeds so one lucky run couldn’t fool me. And I kept a portion of the data sealed, to be opened once and on the record, only for a candidate that had already earned it.

The answer: the timing approaches I tried did not beat simply holding the same stock, after costs, on stocks the models had never seen. Rankings built on company fundamentals did show some ordering in the years I tested, which is interesting, but an ordering is not the same thing as a return you can actually earn. The sealed data stayed sealed, because nothing earned the right to open it.

It was a little painful. It was also the most valuable thing I learned this year, because I’m quite sure that without the rules written down first, I would have found a version that “worked”. With enough knobs, something always works on the past.

(To be clear, this is a story about method, not about markets. Nothing here is investment advice.)

Why It Happens to Smart People

Statisticians call this the garden of forking paths. Every analysis involves dozens of small, reasonable choices: which date range, which companies, which threshold, which version of the metric. None of them feels like cheating. But if you make them after seeing the data, you end up walking down whichever path happens to lead to the nicest view.

Product teams have the same problem with better branding. We pick the comparison period after the launch. We choose between “users” and “sessions” depending on which one went up. And once a number becomes the target, people start optimizing the number instead of the thing it was supposed to measure. That’s Goodhart’s law in its natural habitat, the quarterly business review.

What I Do Now

1. Write the pass mark before the test, and date it. Not “it should be good”. A number, a comparison, and what happens if it’s missed.
2. Pick a baseline that would be embarrassing to lose to. Buy and hold. A plain rule. Last year’s design. If you can’t beat the simple thing, you haven’t built anything yet.
3. Run a fake next to the real thing. If a random signal performs as well as your clever one, you just measured noise, and it’s much cheaper to find that out now than after launch.
4. Hide some data from yourself. Keep a holdout you don’t touch. Open it once, and write down that you opened it.
5. Keep the misses. In ValuFact every forecast is written once, with its starting point and its deadline, and graded by code. When I changed a grading rule, the old grades were voided but stayed visible. A track record you can edit is not a track record.
6. Decide what failure means before you fail. Kill it, iterate, or keep it internal. Deciding this in advance saves you from negotiating with your own sunk costs later.

It Applies Far Outside AI

Before a launch I worked on this year, the team set its satisfaction target before the event, not after. The post-event surveys from both customers and teammates came back above it. That sentence is only worth something because the target came first.

The same discipline applies to credit. A few years ago I wrote the strategy and action plan for a large company’s website renewal. Their team implemented it, and pageviews roughly doubled in the months after. I’m proud of that work, but I didn’t build their website, so it’s shared credit and I say so. At a media startup where I led product, users and engagement grew fast and the company still wound down. A product can grow and a company can close. Both are true, and pretending otherwise makes everything else you say a little less believable.

Credibility Compounds

There is also a selfish reason to work this way. A team that reports its misses gets believed when it reports its hits. Leadership stops asking “is this real?” and starts asking “what’s next?” That trust is worth more than any single good-looking chart.

It changes what you ship, too. ValuFact has a feature I’m genuinely excited about: a map of emerging and fading patterns across thousands of filings, built on a small language model I trained. It only goes to users once it passes the test I wrote down before building it. Shipping it early would feel great for about a week.

In Summary

A pass mark you write after the test isn’t a pass mark. It’s a caption. Write it first, date it, keep your misses, and let the result be whatever it is. You will ship a bit less. What you ship will work, and people will believe you when you say so.