Sample Size and Statistical Significance
Sample size is how many trades a result is based on; small samples produce misleading statistics, and statistical significance asks whether a result could be luck.
Ten winning trades feel like proof, but they may be nothing more than luck. Sample size is the number of trades behind a result, and statistical significance asks whether that result is likely to be more than chance. Small samples make good and bad strategies look alike.
- Reading time
- 3 min
- Also called
- Number of trades, Trade sample, Statistical significance
- Updated
Sample size in plain words
Profits can come from skill — or just as easily from friendly market conditions or plain luck.1
Once researchers have tried many ideas on the same data, the usual tests of statistical significance are no longer good enough.2
A good win rate can still hide a losing strategy, so one number from a handful of trades proves little.3
How to think about sample size
Count trades, not days
Results depend on the number of independent trades.
Set a minimum first
Decide in advance how many trades a test needs before judging it.
Raise the bar for mined ideas
Set a higher bar for any new idea you've found by testing many variations.2
Cover different markets
Include up, down and sideways periods in Backtesting.
Watch for bias
CFA Institute flags survivorship bias and look-ahead bias in backtesting.4
Track live results
Keep adding trades in a Trading Journal and recheck the Expectancy.
Worked examples
Example 1 — Ten-trade hero
Too few tradesA hypothetical system wins 8 of its first 10 trades.
| Win rate | 80% |
| Trades | 10 |
With so few trades, the result might equally come from favourable conditions or simple luck.1
Example 2 — Larger sample
Truer pictureAfter 200 hypothetical trades, the same system wins 52%.
| Win rate | 52% |
| Expectancy | +0.1R |
A larger sample gives a truer picture, and a high early win rate alone did not show the real edge.3
Example 3 — Tested many ideas
Data-minedA hypothetical trader tests 50 rule variants and keeps the best one.
| Variants tested | 50 |
| Live result | −6R |
After heavy data mining, usual significance criteria do not make sense, so the 'best' variant was likely luck.2
Small vs large samples
| Aspect | Small sample | Large sample |
|---|---|---|
| Trades | A handful | Hundreds |
| Luck's role | Large | Smaller |
| Confidence in edge | Low | Higher |
| Risk of false discovery | High | Lower |
Where sample size matters
Backtests
Judging Backtesting results.
Win rates
Reading a Win Rate fairly.
Edge claims
Testing a supposed Trading Edge.
Performance stats
Trusting a Profit Factor or Sharpe Ratio.
Advantages and disadvantages
Advantages
- Protects against trusting luck.
- Makes strategy reviews fairer.
- Reduces system hopping.
- Sets clear test rules in advance.
Disadvantages
- Large samples take time to collect.
- Markets change during long tests.
- Trades are not always independent.
- No trade count guarantees an edge.
Going deeper
Read as far as you need. Each level builds on the one before it.
Luck vs skill
Beginner
Results can come from skill, good conditions or luck, and only a large enough number of trades starts to tell them apart.1
Significance hurdles
Intermediate
After heavy data mining, a new finding has to clear a much higher statistical hurdle to be believable.2
False findings
Advanced
Some researchers go as far as arguing that most claimed findings in financial economics are probably false.2 Traders testing many rule sets face the same risk.
Common mistakes
Judging on a few trades
Luck dominates small samples.
Testing many variants
The best one is often noise.
One market regime
Results may not travel.
Ignoring costs
Small edges vanish after costs.
Check your understanding
Test yourself before moving on. Each answer comes with an explanation.
Key points to remember
- Few trades prove little.
- Luck can look like skill.
- Testing many ideas raises the bar.
- Set sample rules in advance.
- Keep checking live results.
Frequently asked questions
How many trades are enough?
Does a high win rate prove an edge?
What is statistical significance?
Further learning
- Backtesting
Testing on history.
- Expectancy
The average result per trade.
- Trading Edge
What a sample tries to prove.
- Win Rate
Reading it fairly.
Sources
This article is written in our own words. The sources below support the specific facts, rules and figures marked with a number in the text. Links open the original publisher’s page.
- CMC Markets. Trading psychology explained: The mental side of trading (opens in a new tab) Checked .
- NBER. ... and the Cross-Section of Expected Returns (Working Paper 20592) (opens in a new tab) Checked .
- IG. Keeping track of your trading performance (opens in a new tab) Checked .
- CFA Institute. Backtesting and Simulation (opens in a new tab) Checked .
Educational content, not financial advice. Examples use hypothetical prices and simplified costs; actual fees, margin rules and regulations vary by broker, market and country. We are not affiliated with any publisher linked above. Last reviewed . Report an error or request a source credit.