Strategy Testing: Can You Trust the Results?

Key takeaways

1. Define the question, rules, data, and assumptions before examining the outcome. Keep enough information to reconstruct the test.

2. Check what was knowable at each decision and how orders were modeled. A simulated fill and an unresolved candle path require careful interpretation.

3. Retain the alternatives tried and distinguish development from evaluation. Reusing evaluation results to revise rules makes those observations part of development.

4. Examine the full record, execution-cost sensitivity, and forward observations. No single trade count, holdout period, or stable setting proves an edge.

Full transcript

How was the record produced?

A strategy report shows a profit.

The win rate, average result, and drawdown are all on the page.

But how was that record produced?

Did someone define the decisions first and then examine what happened?

Or did they keep changing the decisions until the past looked good?

The numbers alone cannot tell you.

In the previous lesson, we learned how to read several measurements together. Now we need to examine the process behind them.

Strategy testing means evaluating a defined approach using specified data, rules, and assumptions. The purpose is to learn what the evidence supports, where it is weak, and what deserves further investigation.

Define a testable question

It begins before the result.

Start with a question you can examine.

For example, suppose someone believes that a move beyond an earlier price range may continue for a time.

That is an idea. It is not yet a complete test.

Which market? Which session? How is the range measured? Does a touch count, or must price move beyond the boundary? Is a completed candle required?

When could an order actually be submitted? How would the position be closed? What happens if the intended order cannot fill?

For an invented example, imagine a rule that waits for a candle to close above a defined morning range, then attempts an entry at the next candle's opening price.

The test also needs a specific stop rule, exit rule, position size, trading window, and limit on repeated attempts.

These details are illustrations, not a strategy recommendation or a claim that this idea has an advantage.

Write the rules clearly enough that another person could apply them without seeing the later outcome and making different choices.

If a decision still requires judgment, describe that judgment and record how it was made. Do not silently turn a discretionary choice into a precise rule after seeing which answer worked.

Choose and check the data

Then define the record you will use.

Identify the data source, dates, time zone, session boundaries, and level of detail. Check for missing observations and values that do not make sense.

For futures, identify the contracts and how any change from one contract to the next is handled.

A chart assembled from several contracts is not automatically the same thing as a record of executable prices in each individual contract.

Keep enough information to reconstruct what was tested.

What a backtest models

Now we can distinguish a few types of evidence.

A historical backtest applies rules to past data using a simulation of decisions and execution.

It can help you examine many opportunities, locate mistakes, and compare results under stated assumptions.

But simulated orders are not actual fills. A backtest describes what the model produces. Its usefulness depends on whether the data, timing, and execution assumptions fit the question.

Manual testing can also be useful. So can code. Both can contain errors.

Automation makes a procedure repeatable. It does not make the procedure correct.

Information timing and look-ahead bias

One important check is whether every decision uses only information available at that moment.

Return to our invented rule. It requires a candle to close above the range.

The final closing price becomes known when that candle finishes. A simulation cannot use that completed close to justify an entry earlier inside the same candle.

That gives the decision information from its future. It is called look-ahead bias.

The same problem appears when a chart label marks a turning point using later candles, but the test acts as though that turning point was known immediately.

Ask two separate questions: when did the event occur, and when could the trader have known enough to act on it?

Execution assumptions

Next, examine how the test handles execution.

A price appearing on a chart does not, by itself, establish that your order would have filled at that price.

The model needs assumptions about order timing, available prices, trading costs, and slippage: the difference between the expected execution price and the price obtained.

Those assumptions should fit the instrument, order type, and conditions being studied.

What OHLC cannot tell you

There is another difficulty with candles.

Suppose a trade is already open before a candle begins. Both its stop level and target level fall inside that candle's high and low.

The candle shows that both prices were reached during the interval. Those four values alone may not tell you which was reached first.

If the test always awards the target first, it can manufacture a favorable result.

Use more detailed data when it can resolve the question. Otherwise, state how the ambiguity is handled and examine how much the conclusion depends on that choice.

An uncertain trade path should remain an uncertainty in the evidence.

Audit cases and record changes

Before interpreting the headline result, check individual cases.

Choose examples that include winners, losers, skipped opportunities, and awkward situations. Compare the recorded decisions with the written rules and the information available at the time.

If you fix an implementation error, document it and rerun the affected results. If you change the trading idea, record a new version.

That distinction matters when we reach development and evaluation.

Development and the search history

Development is where you explore an idea and make choices. You might compare entry definitions, stop distances, or trading windows.

The data used to make those choices is part of development. It is often called in-sample data.

Keep a record of the alternatives you tried, including the disappointing ones.

Suppose you compare four entry definitions, three stop settings, and two exit settings.

That creates twenty-four combinations.

Showing only the strongest result hides the search that selected it.

With more opportunities to choose, an attractive result can reflect an accidental fit to that particular history. This is part of the problem called overfitting.

The winner of the search still needs to be evaluated in light of the search.

Reserved evaluation data

One useful separation is to reserve a later period that has not guided those choices. Specify the version and evaluation method before examining its results.

That is the purpose of an out-of-sample evaluation: to examine how the chosen approach behaves on data outside the development process.

But calling a period out of sample does not make it independent of your decisions.

If you inspect the reserved results, change the rules to improve them, and test the same period again, those observations have now influenced development.

You can learn from them. You cannot honestly describe the revised result as a fresh evaluation untouched by those observations.

A separate period is also not a certificate of validity. Its length, conditions, prior familiarity, and the number of choices made still matter.

Read the full record

Now read the results using the measurements from Lesson Nine.

How many trades occurred? What was the average net result? What were the sizes of gains and losses? What drawdown occurred, and how was it measured?

Look beyond the total.

Were gains concentrated in a few unusual trades? Did most of the favorable result come from one short period? Were there long stretches with few opportunities?

Record the full period, including the unfavorable parts. A collection of selected examples cannot tell you how often the rules failed.

The number of trades matters, but there is no single trade count that makes every strategy trustworthy.

Many trades generated during one unusual episode may tell you less about other conditions than the count suggests. Trades can also be related to one another rather than independent repetitions.

The question is how much information the record provides for the conclusion being made.

Sensitivity to assumptions

Then ask how sensitive that conclusion is to reasonable changes in assumptions.

Suppose an invented backtest contains one hundred completed trades and shows a total gain of three hundred dollars after the costs already included.

Now suppose an alternative execution assumption adds four dollars of cost per completed trade.

That adds four hundred dollars across the record. Holding everything else fixed, the three hundred dollar gain becomes a one hundred dollar loss.

This does not establish that four dollars is the correct adjustment. It shows that the conclusion depends on an assumption that needs evidence.

You can also examine nearby rule settings. If a result collapses after a small change, investigate why.

Stability across nearby choices can be informative. It still does not establish an edge by itself, and choosing a new favorite after inspecting those comparisons is another development decision.

Forward observation and execution records

What happens after historical testing?

Forward observation means fixing the approach and recording what happens as new opportunities arrive, without knowing their outcomes in advance.

That can include paper trading or recording signals and intended orders under a documented procedure.

Record every qualifying opportunity, the information available, the intended decision, and what happened. Include missed actions, rejected orders, and differences between the plan and the recorded execution.

Paper trading helps examine process, but its fills and practical conditions may differ from real execution. Actual trading records add evidence about actual execution; they still do not remove uncertainty about future performance.

Plan reviews and preserve changes

Choose review points and reasons to pause before an upsetting result forces an improvised decision. If something appears broken, investigate it and preserve the record of what changed.

The goal is not to keep every version alive. A useful test may expose a data problem, an unrealistic assumption, an unstable result, or an idea that does not deserve more work.

Questions before accepting a report

Before accepting a strategy report, ask what was specified before the result, what information was available at each decision, how execution was modeled, and how many alternatives were tried.

Then ask which evidence helped build the approach, which evidence evaluated it, and what remains uncertain.

Testing earns its value by making those questions easier to answer.

A profitable curve is one result. A documented process helps you judge how much confidence that result deserves.

S&P 500 Futures Explained

Continue with Know the Contracts: connect S&P 500 futures price movement with dollar exposure, then explore costs, margin, expiration and execution.

ES, MES and NES · 10:41 · Video and full transcript