How to count your trials: the trial log

Why you're asked for a number you don't have

The Overfitting Calculator asks for your trials tested: every version you tested, not just the one that survived. It is the one part of the process no backtest report can show, because the report describes the winning version and says nothing about the others. Only you know how many there were. That is why an audit prints it as a figure declared by the client: there is no way to measure it from the outside.

Without that number, any correction of the result is a guess. With it, the correction is arithmetic.

When the count starts

Your first backtest is already trial 1. The count of adjustments starts the moment you look at a result, it doesn't convince you, and you change something. Three indicators, the result isn't what you expected, you swap one or add one: trial 2. You run the backtest again, it still doesn't work, you add another indicator, move another parameter: trial 3. Every change made after seeing a result adds one.

The rule in one line: the count doesn't measure changes, it measures results looked at. Every distinct configuration you saw a result for is a trial, whether you kept it or not.

What counts and what doesn't

Counts:

• Changing or adding an indicator.
• Moving a parameter. Every value tested is a trial; an optimizer's grid too: ten values of one parameter by ten of another are a hundred trials, not one.
• Adding or removing a filter, a rule, an exception.
• Testing the same idea on another timeframe, another session, another instrument.
• Running it again "just to see" after moving something.
• Every time you used a reserved period to choose between versions. An out-of-sample period stops being out-of-sample the moment it decides.

Doesn't count:

• Fixing a code bug and re-running the same configuration: it is the same trial, computed correctly.
• Changing the report format or the metric you look at on the same trades.
• An idea you discarded without running it. What produced no result doesn't count.

When in doubt, count more, not less. A short count inflates the result in your favor; a long count only makes it more demanding.

The trial log

Five columns are enough. They are filled in at the time, not from memory.

• Trial number.
• Date.
• What changed from the previous one.
• What you saw in the previous result that led you to change it.
• The result you looked at, and whether you kept or discarded it.

One row reads, for example: trial 3 · September 12 · added a volatility filter · trial 2 lost in sideways markets · better one year, worse the next · kept. The number of rows is your number of trials. If the strategy already exists and you never kept count, rebuild it backwards from what you remember and round up: what you forgot happened too.

What the count does to your result

Counting does not prevent data mining: it makes it visible and discounts it. Search long enough and sooner or later you find a pattern in the past with wonderful numbers; the count is what lets you know how much of that the search alone could have produced. That is the noise bar: the best Sharpe that statistical noise reaches with your number of trials and your amount of data. Thirty trials on two years of daily data put it at ≈ 1.47; a hundred, at ≈ 1.80.

The count's discount (the Deflated Sharpe Ratio; in the calculator, "after discounting the search") leaves your result closer to what the evidence supports. Not closer to what the market will do: the discount predicts nothing and guarantees nothing; it removes from the number what the search gave it for free. Counting doesn't make a strategy good. It makes the number honest.

What to do with this

1. Open the log before the next backtest. The first row is the trial you already have.
2. Write down every distinct configuration you look at a result for, including the ones you discard.
3. Take the total to the Overfitting Calculator together with your sample. Clearing the bar certifies nothing and failing to clear it condemns nothing: it is the signal for deciding whether the strategy deserves a full validation.
4. When it does, that total is the number you declare. A short count doesn't fool the audit: it inflates your own result.