ETFReplay vs Portfolio 123, different backtest results

Research
3

Hi Brian,

Why specifically do you say that the result is so different? What is your Annual Return for this strategy on ETFReplay? What differences are you most concerned about?

I was just going to tell you that days and bars are different in P123 and that (generally) there are 20 bars in a month. However, as I am posting this I see that Michael (SpacemanJones) is also advised you regarding this issue at the same time, so I edited this answer. Let us know if this solution resolved your problem.

Chris

4

Slippage matters. Try this screen I made, which I think is what you want.

Try with 0 slippage vs 0.15 slippage.

https://www.portfolio123.com/app/screen/summary/203581?st=0&mt=9

5

thank you for all the reply.

I just revised my simulation based on your inputs. However, my results are completely different. It is almost like my ranking system runs in the opposite direction. I am looking to have the ETF, with the highest total returns 13 week. However, the output seems to find the ETF with the lowest total returns 13 weeks. Therefore, the model has chosen SHY (3 month treasury) every month, instead of the ETF with the highest returns.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

the below is the ranking system.

https://www.portfolio123.com/rank_about.jsp

I wonder what I have done wrong. Can you please shed some light on this?

thank you very much.

B

6

Sorry I forgot to unlock the ranking system. Please see the below the ranking system. I want to find the ETF with the highest 13 week total returns. but the model seems to come up with the etf with the lowest returns. Many thanks

https://www.portfolio123.com/rank_about.jsp?rankid=324563

7

One thing is that you are using IMW in buy rule …there is no such ETF…is that typo for IWM ?
Check your univ also

8

thanks, it is IWM, indeed.

I have corrected this.

still not sure what else I have done wrong. thanks

9

Sell…rule needs modification…once there is investment in SHY…how will it be sold?
Also …if once a particular stock ETF is bought …next period it is only compared with SHY…let’s say it’s not sold. Won’t it miss out on buying the highest ranked etf?

Look at transactions also…and try to find out…if they are as expected…and you will discover issues

10

Use this as your buy rule:
eval(close(0)/close(60) > close(0,#bench)/close(60,#bench),ticker(“IWM,MDY,QQQ,SPY”),ticker(“SHY”))

And “true” as your sell rule

Allow immediate buybacks and your ranking system should be 13 week price % change

11

HI PortfolioPerfection

thank you for your comments.

May I know what ‘true’ in the sell rule means? I did make changes as you suggested. the results are here: total returns: 96.76% with a max drawdown of -49% over the period from 31 Oct 2002 to 16 Feb 2018.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

However, according to ETFreplay, the attached are the results: the CAGR is 14.1% with the max drawdown as -18%. I assume no slippage or commissions in Portfolio123 so they re not the reasons.

I attached the screenshot below:

What may be the reason for such a wide difference?

ETFReplay.pdf (234 KB)

12

I’m joining this discussion because our separate discussion by email seems like duplicated effort.

True as a sell rule means sell/reevaluate at each rebalance. If that’s what you want, then it suggests that you might be better served with a screen (unless you need the transaction record for some reason, like trading with IB).

I would use this as a buy rule:

(Close(0)/Close(63))-1>=(Close(0,GetSeries("SHY"))/Close(63,GetSeries("SHY")))-1

Replace the 63s with however many trading days you’d like. The trick is that SHY will only ever be the top result in your ranking system when there’s nothing else to be bought. RankPos=1 isn’t actually doing anything in portfolio/simulation as long as you’ve set the position size to 100%. (It sounds like you’d like allow immediate buyback set to yes too, but that hammers results in my test.)

I would use

(Close(0)/Close(63))-1

as the node in your ranking system. This is for two reasons: First, it makes the rule above work; there’s no guarantee that Price13W%Chg will give the same result as that formula due to rounding on the back end (as I recall, it’s dividing 251 by 4 to come up with a single quarter, and is only doing so once a week).

In fact, I can guarantee that Pr13W%Chg is going to give a different result than Close(0)/Close(63)-1, because the close formula is taking dividends into account and Pr13W%Chg is only comparing price changes.

If you want dividends to be included, then you should use TotalReturn13W, or change your ranking node to the basic formula. If you don’t want dividends to be included, then use CloseExDiv instead of Close in the basic formula.

Again, I would suggest using Close(0)/Close(63)-1 consistently everywhere.

And the more that I’ve looked at this problem, the less that I think that you’re going to get us to match ETFReplay. I’m not getting anything even remotely as high as they’re reporting.

I have no idea why, but the difference that you’re reporting in drawdowns is probably the key: If they’re reporting an 18% drawdown and we’re reporting a 45% drawdown then we’ve selected different ETFs in 2008/2009.

13

Whether ETFs pays dividend, whether it is accounted for in P123.

Thanks
Kumar

14

ETF dividends are paid into the cash account in portfolios and simulations. We adjust the prices for future dividends in screener backtests to ensure total return outputs in that tool.

15

thank you for your reply.

I did notice that ‘allow immediate buyback’ seriously affects the results negatively.

Do you have any insight as to why “allow immediate buyback” can affect results so much?

the below is the new portfolio.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

In the ETFReplay , the CAGR is 14.1% with -18.4% drawndown (vs 9.8% with -55.2% drawdown of SP500). It outpersorms the SP500 by a large margin. Please see the screenshot in the previous reply.

However in POrtfolio 123, the CAGR is 9.26%, with max drawdown of -26.53%. It underperforms SP500. the annual difference of over 4% between ETFReplay and Portfolio123 is huge.

In the ETFReplay, it picks the best of the four (SPY, QQQ，MDY，IWM) based on the past 3 month returns. If the best is still worse than that of SHY, SHY will be selected instead.

Really wonder which model is doing the right calculation? or maybe I am still not replicating the ETFReplay correctly?

Many thanks for all the support.

B

16

brwg,

Please, post the transactions for last 5 years, for both ETFReplay and Portfolio 123.
We will compare transaction by transaction manually and figure out where it deviates.

ETFReplay and Portfolio123 are tools, small difference in buy or sell rules make a big difference over period of time.

Thanks
Kumar

17

thanks for your feedback. Kumar.

I just downloaded the transactions to the attached excel sheet. Yes, I found the major difference is the rebalance timing.

In ETFRepay, rebalance is set at the end of each months to be invested on the 1st business day of the calendar month.

However, in Portfolio123, rebalance frequency is every 4 weeks. Therefore, the rebalance timing is quite different, resulting in buying different ETFs. I think this is the major reason. However, as the difference in returns is quite large, I can’t help wondering whether rebalancing at the end of each calendar month is better than '4 weeks" used in the Portfolio123.

Does anyone know whether Portfolio123 can allow such adjustments for rebalance period?

I am trying to determine:

  1. whether one rebalance timing has advantage over the other
  2. whether there are any other underlying fundamental differences in the two models, potentially causing one model more accurate than the other.

Also, When I set Sell Rule as “True”, what does that mean? Does that mean, stocks will be all sold first before the Buy Rule is applied? If I don’t have “True” in the Sell rule, it seems no stock will be sold at all over the testing period.

Also, why “allow immediate buyback” will affect negatively the results?

Many thanks for your support.

ETFReplay vs Portfolio 123.xls (74.5 KB)

18

Nice to know, you have nail down the difference.

P123 mostly for fundamental data,
13 weeks quarterly FA report and 52 weeks annual FA report
and weekly rebalance and Monday only trading’s are
one kind of FA strategy relies on earning and weekly fundamentals.

ETFs don’t have any fundamental, it has price and volume, it is more towards technical. TA.

I have followed some TA experts, they have assumed end of the month,mutual funds Money will sell some ETFs for profit and switch another on by first week of every month.

In P123

  1. keep weekly rebalance and
  2. in sell rule Keep
    ((MonthDay>=1) & (MonthDay<=7))
    ============================
    One of the Monday will fall on this dates 1st week of every months, so it become monthly re-balance.

3.In buy rule if you keep
(LastSellDays > 30)
it won’t buy the same ETF just sold for 1 month (30 days) time.

Please, let us know the outcome with transaction history.

Thanks
Kumar :sunglasses:

19

Kumar

thanks for the reply. I am sorry I am still new to P123.

did you mean “NoBars >=1 and NoBars<=7” in the sell rule?

or “NoDays >=1 and NoDays<=7”?

I tried both and changed the rebalance to weekly. However, I see the trades are done weekly. The rebalance timing is not closer to 1st day of each month and the results not changed much.

Maybe I have misunderstood you?

thanks

20

sorry, I will try again. I seem to have misunderstood you. Let me try again.

21

As i am in office, my recent reply was from mobile with typo error and less info, then later i have added more details with exact rules.
I am good with p123 commands. I am curious to know P123 can replicate the performance of ETFReplay.

Thanks
Kumar :slight_smile:

22

Kumar

Interestingly, If I set the sell rule as:
(1)
MonthDay>=28
MonthDay<=5

the results are significantly improved, with the returns being 316.5% and drawdown dropping to only -19%. .

(2)
If I use the sell rule as
Month Day>1 and MonthDay<7, the returns dropped to 45.38% and drawdown -26.42% (yes, not sure why the returns dropped that much).

(LastSellDays > 30) also makes a huge difference.

If I remove this rule from (2), the returns goes back to 278.39% and drawdowns dropped to -28.96%.

In contrast, if I remove this rule (LastSellDays > 30) from case 1, the returns jumped to 360.8% but drawdown decreased to -37%.

I wonder what made you come up with this (LastSellDays > 30) in the first place? If we are to choose the best performer over the 3 months, why do we exclude it even if it has been bought within the last 30days?

thanks

MonthDay comparison.xltx (108 KB)