Distrust the Number That Flatters You
Losses get investigated. Wins get banked. That asymmetry is where self-deception lives.
We spent months building a system that produced an extraordinary result. In testing, it turned a starting figure into more than twenty-five times its value over four years. Every headline metric was strong. The obvious next move was to celebrate the number and scale the system behind it. We did the opposite. We spent the next stretch of work trying to prove the number was wrong.
The reason is a principle that took us a long time to internalise: the results you most want to be true deserve the harshest scrutiny, and they almost never get it. When a system loses, everyone investigates. The post-mortem is thorough, the blame is assigned, the fix is urgent. When a system wins, the result gets banked and the team moves on. That asymmetry feels natural. It is also exactly where self-deception lives. A flattering number buys itself immunity from the questioning that a disappointing number would never escape.
So we interrogated our best result as if it were our worst. And it did not survive. The spectacular figure was inflated by a single artifact in the simulation. The model was recording a category of win that could not occur in reality: a favourable fill that looked like profit on paper but was impossible under live conditions. It was not fraud and it was not carelessness. It was one wrong assumption, buried deep, that happened to push the number in the direction we were hoping to see. Strip that one artifact out and the picture changed completely.
The honest number was thin. On the measure that actually mattered, the strategy succeeded a little over forty per cent of the time, hovering somewhere between marginally positive and marginally negative once real costs were included. Not a twenty-five-times story. Closer to a coin flip with expenses attached. Everything we had believed about the system was built on a figure that dissolved the moment we stopped protecting it.
Then came the harder question, and it is the one most worth carrying into any business. Was there any genuine edge here at all, or were we simply riding the environment? A rising tide lifts every boat in the harbour, and for a while every captain feels like a sailor. We tested it directly, and the answer was uncomfortable. The returns tracked the broader market. When the environment was favourable, the system looked clever. When it was not, the system had nothing of its own to offer. It was not skill. It was exposure wearing the costume of skill.
This distinction, between an edge and an environment, is one that almost every business confuses at some point. A sales team looks brilliant during a boom. A campaign looks like genius when the whole category is surging. A product looks indispensable while the market is expanding underneath it. The single question that separates a real edge from a lucky environment is this: would it still work when conditions turn against you? If the honest answer is no, you do not have an edge. You have a tailwind. And the defining feature of a tailwind is that it eventually becomes a headwind.
We did not give up there. We had a promising new thesis, a different and more structural way to make the system work. The temptation was to spend weeks building it out properly, because it was an idea we liked and wanted to be right. Instead we asked a colder question: what is the cheapest test that could prove this wrong? We designed exactly that test, ran it, and watched the thesis fail in an afternoon. That afternoon saved us months.
Killing your own idea cheaply is a discipline, not a defeat, and it runs against every instinct. The natural move is to protect the idea you are attached to. Run one more test that might rescue it. Interpret the ambiguous data generously. Give it the benefit of the doubt you would never extend to someone else's proposal. The professional move is the reverse. Try hardest to kill the ideas you most want to keep, and do it with the smallest experiment that would settle the question. An idea that survives a genuine attempt to destroy it is worth building. An idea that can only survive a gentle one is not.
None of this is confined to trading systems or software. The same asymmetry shows up in every business that measures anything. The marketing channel that is "working" but was never once tested against a control. The retainer that looks profitable until you account for the hours nobody logged. The feature the whole team loves that no customer would actually pay for. The referral source everyone credits that a closer look reveals was going to convert anyway. These numbers survive not because they are true but because questioning them feels like sabotaging your own good news.
The practical discipline is straightforward to state and hard to practise: treat your best results with the same suspicion you reserve for your worst. When a number pleases you, do not bank it. Ask what would have to be true for it to be wrong, then go looking for precisely that. Verify the mechanism, not just the outcome. A result you cannot explain is a result you cannot trust, however good it looks. If you do not understand exactly why something worked, you do not actually know that it did.
The honest ending to our story is not a triumphant one. We did not find a way to rescue the system. We concluded that the edge was not there, and we redirected the engineering effort into work where it would actually compound. That is the point. The value was never in the system. It was in discovering, cheaply and honestly, that the system did not work, rather than discovering it slowly and expensively after scaling something hollow. A month spent proving yourself wrong is far cheaper than a year spent being wrong at scale.
The discipline that protects a business is not the one that celebrates its wins. It is the one that interrogates them. The costliest mistakes are rarely the ideas that fail loudly and early; those get caught. The costly ones are the ideas that appear to succeed, go unquestioned because success is pleasant, and get scaled on the strength of a number nobody checked. So when a result flatters you, slow down. Distrust it a little. It is the number most likely to be lying, precisely because it is the one you least want to doubt.
Build on numbers you have actually tested.
The discipline of interrogating your own results is the same one behind the 12 Principals programme, where we turn what you genuinely know into infrastructure that stands up to scrutiny and scales beyond you.
