Why backtesting and simulation results are so different?

Research
1

In backtesting I obtain +50% average anual return but in simulation I get -3%

The settings are the same, I think.

Using same universe, ranking and buying rules. No hedging and no sell rules like in backtest.

2

The screener sells all stocks at the end of the rebalance period then buys a new set of stocks according to your buy criteria and ranking system. Often some of the new stocks you buy will be stocks you just sold. If you do not have any sell rules in your sim it will hold the stocks that it buys until the stock drops out of the universe.

You can start to make the sim more like the screener by putting Rank < 101 in the sell rules. Also in the general tab set “Allow sold holdings to be
re-bought at current rebalance” to yes. This will cause all of the stocks to be sold like in the screener but some of them will be bought again if they are ranked high and meet the buy rules criteria.

Later you can add or change your sell rules. For example if you want to hold stocks for a longer period you might change the sell rule to Rank < 90

3

If you do not have any sell rules in a sim, then you will never sell (unless there is merger or bankrutcy). In a screener your stocks are automatically revaluated every rebalance period but in a sim the stocks are not revaluated unless they are sold. So in the sim add a sell rule which could be “Rank<101” and set “Allow sold holdings to be re-bought at current rebalance” to “Yes”. Then your results will be closer.

Hope this helps.

KJ

P.S: I think Jim beat me to it.

4

Wrong thread, sorry

5

That helped a lot, thanks.

Another problem is about weighting, with backtesting, the rebalance makes the new stocks to be at same weight in the port. However in simulation weight is fixed.
is there any way to change this to backtesting approach?
My port is so optimized in backtesting that it holds already an average of 5 stocks each week.

6

Please, help to share the best sell rule for weekly rebalancing and avg 20 days holding; sell based on lowrank or profit taking.

Thanks
tibco

7

Tibco,

You can combine rules to achieve many combinations. Examples:

Rank < xx & NoDays > 20 or Rank < yy or GainPct > zz
This will sell a stock only if it is been held longer than 20 days and has a rank less than xx, or the rank falls even farther below yy, or has gained over zz%.

Rank < xx & PctFromHi < -yy or PctFromHi < zz
This will sell a stock if the rank is less than xx and the stock has lost yy % from the high since it was bought, or it has lost even more than zz%.

Denny :sunglasses:

8

Denny,

Thank you for the response on the sell rule formula and knowledge sharing.

I am getting different results for the same sell rule for different periods.

Please, kindly share with me
best optimized sell rule for small cap high beta stocks for last 15 years data and consistent result most of the year to year basis.

Thank you
tibco

9

First, if the rule were optimized to fit the past it would not be the best rule for the future. With P123 it is very easy to curve fit a strategy to fit the past and it has been shown that such strategies can completely break down going forward. Second, there is no one best sell rule. If there was and it was commonly known we would all be using it and then it wouldn’t work anymore.

With that said, I would say the best sell rule is rank < 99.5.

10

dwpeters/Denny/Marco,

< 99.5 will selloff all the holdings most of the time. it will lead to more transaction cost and slippage.

In screen back test it returns 50% with 30% drawdown for 15 years.
But in simulator returns -10% with -70% drawdown.
with 1% slippage and 10 dollar commission.
Believe, because of wrong sell rule.

it is for strong fundamental stocks in upward moment.
Would like to have avg held 20+ days,
run the gainers 20+ days until rank get degraded rank.
remove the lossers in 5 or 10 days ignoring ranks with weekly rebalancing.

Please, suggest any common rule to start fulfill above requirement.

also help with how to include screener rule with Pr2BookQ<9 and need to include Pr2BookQ = “-” or “NA”
and need avoid -negative values of Pr2BookQ .

thanks
tibco

11

Please, help to share your experience.

whether the simulation performance for ready to go model will be better than screen backtest performance in weekly rebalancing scenario ?

So, I can fine tune sell rule to achieve the result.

thanks
tibco

12

I use rank < 99.5 in a simulation that returns over 100% annualized, using variable slippage which adjusts slippage based on the market cap or trading volume. Yes it has a higher turnover, and only 6 stocks but it generally takes high turnover to achieve high performance … and you did ask for the optimum sell rule. Really there is no such thing and there is much more to a simulation than a sell rule. A good way to get experience with simulations is to copy one of the P123 strategies and try making changes to it.

Pr2BookQ < 9 and Pr2BookQ > 0
I don’t see any stocks with Pr2BookQ = “-” and it appears many stocks have a value of NA for this factor, and they are not filtered out by the above rule.

13

dwpeters,

Thank you the information, It helped.

tibco

14

Tibco,

The Screener sells all stocks every rebalance and buys the highest ranked stocks. That may include the stocks that were just sold if they are still included in the highest ranked. So in effect it only holds a stock for 7 days if rebalanced weekly. The Rank < 99.5 simulation sell rule will sell stocks less frequently than the Screener will. The screener also equalizes the dollar amount of the stocks every rebalance. That means that it may increase or decrease the number of shares by a small amount of any stocks that it re-buys. Because of this, the Screener is not the best backtest tool. The Simulation is far better since you can simulate real trading approaches with realistic sell rules.

If you want to compare the Screener results to the Simulation the in the Sim set “Allow sold holdings to be re-bought at current rebalance” on the first page to Yes, and use any sell rule that causes the Sim to sell all stocks like Rank < 101, or just use a 1 for a sell rule. That way the Sim will buy and sell stocks similar to the screener. However, that is not the best way to actually trade stocks.

You said; “I am getting different results for the same sell rule for different periods.”

That is to be expected. The economy and the markets are very different in different periods so it is impossible for a sell rule (or a set of sell rules) to give consistent real results in all periods. The best you will be able to do is consistently outperform the market baseline you choose to compare to.

Denny :sunglasses:

15

Denny,

Thank you for sharing your experience.

How to compare the simulation paper results with practical investment ?
100K capital in 15 years, in simulation it shows million/billions dollars.
In practical it is not possible to continue to buy 100M capital and $50K(liquidity).
then we need to move to different universe with high capital then return become 25%-30% annual return.

Being a long time successful investor, what is your suggestion on this;
How to interpret simulation results with practical investment. How was your transition from small cap to large capital investment.

thank you
tibco

16

my backtest has 80% stocks turnaround (4 stocks need to sell out of 5) in weekly rebalance and lead to 30% performance down in simulation with 1% slippage.

Please, share your experience.

In real investment/trading with interactive brokers or any other low cost transaction brokers,

to get 0% slipplage

what is the stock’s threshold for
(1) minimum avg daily volume in numbers,
(2) minimum avg capital in numbers.
(3) minimum avg price in numbers.
or
(4) any combination of above with numbers.

thanks
tibco

17

dwpeters,

Thank you for the R2G backtest document on your profile.
that helps to understand the slippage cost.

thanks
tibco