I took a large public collection of ready-made trading strategies — 68 files, the canonical freqtrade-strategies repository — and audited what the artifacts declare, without running a single backtest.
No data downloads. No re-implementation. No argument about whether I built someone's idea correctly. Just one question: if you download one of these and run it, what can you tell about the evidence behind it from what ships in the box?
The answer is less than you would think, and the missing piece is the same one every time.
What is in the box
25 of 68 are optimiser output — they carry tunable parameters with live search ranges still in the file. Median 5 tunable parameters, maximum 24.
12 of 68 declare a trial count somewhere, usually in a comment pasted straight from the optimiser. This is more honest than most of the field and worth saying plainly before anything else. The declared budgets:
declared budget files
40,000 1
~5,000 2
2,000 1
1,000 1
100 or fewer 7
14 of the 25 optimiser-derived strategies declare no trial count at all.
4 of 68 declare the sample window the result came from.
Exactly 1 of 68 declares both a trial count and a window — the two fields you need together before either means anything.
0 of 68 report a Sharpe ratio, or any other risk-adjusted number. Ten report a headline total profit percentage. A profit percentage cannot be deflated, cannot be compared against a null, and cannot be compared between two strategies run over different windows.
That last one is the finding. It is not that the numbers are wrong. It is that the number you would need in order to check them is not a field anybody fills in.
What a trial count costs
If you search N configurations and keep the best, the best one has a good in-sample Sharpe whether or not anything real is there. That is not a flaw in anyone's method — it is what taking a maximum does.
The expected best-of-N Sharpe under pure noise has a closed form (Bailey and López de Prado), so you can put a floor under what a result has to beat before it means anything other than "I looked at N things and kept the luckiest".
Annualised Sharpe the winner must exceed:
declared trials 1 yr 2 yr 3 yr 5 yr
10 1.57 1.11 0.91 0.70
100 2.53 1.79 1.46 1.13
1,000 3.26 2.30 1.88 1.46
5,000 3.69 2.61 2.13 1.65
40,000 4.19 2.96 2.42 1.87
Read the bottom row. A strategy selected from 40,000 trials on one year of data needs an annualised Sharpe above 4.19 before its backtest says anything at all. That is not a high bar in the sense of ambitious. It is a bar almost nothing clears.
The single file in the corpus that declares both fields declares exactly that: 40,000 trials, one year of data. Its reported result is a total profit figure. So the bar is computable, the result is not comparable to it, and the gap is invisible to anyone who downloads the file and runs it.
The one that declares its own multiple-comparisons problem
One file ships the results of eight optimiser runs across six different loss functions — Sharpe, Sharpe-daily, Sortino, Sortino-daily, profit-only, trade-duration — pasted in as a block of comments, one result line each, and then one set of parameters below.
Publishing all eight is more transparent than publishing one, and it should be said. But the arithmetic does not care about intent: trying six objectives and keeping the configuration you liked best multiplies the search. The declared budget across those runs sums to a little over 5,000. Nothing downstream of that file knows it.
The precision tell
25 of 68 carry a take-profit or stoploss constant specified to three or more decimal places. Some go to five: 0.21275, 0.04093, 0.27058.
Nobody chooses to take profit at 21.275%. That number is the output of a search, and its precision is the fingerprint. It is the cheapest thing to look for in any strategy artifact you are handed, and it takes about four seconds.
What is not wrong here, and one thing that is
It is worth being precise about what this collection does correctly, because the usual complaint does not land.
Fees are applied. Freqtrade applies the exchange's fees automatically during backtesting, so a strategy file has no reason to mention them and their absence from the source is not evidence of anything. If you were about to write "they ignore costs" — I checked, and they do not.
Slippage is not modelled. The documentation is explicit: orders fill at the requested price as long as it is within the candle's range. For a strategy trading a handful of times a month, fine. The file declaring 40,000 trials also declares 30,918 trades at an average of 0.39% each, which breaks even at 19.5 basis points of slippage per side, and loses a quarter of its edge at 5bp. At that trade count this is not a modelling nitpick; it is most of the result.
Credit where it is due: the repository also ships a folder of four strategies deliberately flagged as containing lookahead bias, with the specific bug in each one written up as an exercise. That is a more honest artifact than most commercial products manage, and it is the reason this collection is worth auditing rather than dismissing.
And mine
I ran the same check over my own retired research artifacts before writing this. The trial-count field is missing there too — not because I did not search, but because I searched across sessions over two years and never wrote a count down anywhere. Reconstructing it afterwards is not possible.
That is the same defect measured above, and it is the reason I think this is a convention problem rather than a carelessness problem.
The check
Whatever you are running, from wherever you got it, three fields:
How many configurations were tried before this one was kept?
How long was the sample?
What was the risk-adjusted result — not the total profit?
If you have all three, one function tells you whether the first number ate the third:
import math
from scipy.stats import norm
GAMMA = 0.5772156649015329
def sharpe_floor(n_trials, years):
"""Annualised Sharpe a best-of-N winner must beat to mean anything."""
a = norm.ppf(1 - 1 / n_trials)
b = norm.ppf(1 - 1 / (n_trials * math.e))
return ((1 - GAMMA) * a + GAMMA * b) / math.sqrt(years)
sharpe_floor(40000, 1) # -> 4.19
sharpe_floor(100, 3) # -> 1.46
sharpe_floor(10, 5) # -> 0.70
If you cannot answer the first question, the honest reading of the backtest is that it is uninterpretable rather than good or bad. That is an uncomfortable place to leave it, and it is where the evidence actually leaves it.
The cheap fix is to write the count down while you search. Reconstructed afterwards it is always an underestimate, because the runs you do not remember are the ones that did not work.
What this does not claim
I did not run these strategies and I am not saying whether any of them makes or loses money — the audit is of the artifacts, not the returns, and every number above comes from what the files declare about themselves.
The floor assumes independent trials. Guided search such as TPE is not independent, and the argument for which direction that pushes the floor is genuinely unsettled, so treat the table as an order of magnitude rather than a threshold.
None of this is a recommendation to trade or not trade anything.
Reproducing it
The audit script reads only what each file declares and runs no backtest. It takes a few seconds against a fresh clone.
git clone --depth 1 https://github.com/freqtrade/freqtrade-strategies
curl -O https://straud.dev/audit.py
python audit.py freqtrade-strategies/user_data/strategies
Every figure on this page comes out of it.
Originally published at straud.dev.
