I’m with Walter on this one. A private forum for subscribers would allow the exchange of ideas, including backtest results, without the general public being exposed to them. After all, backtesting is at the core of P123. To not allow subscibers to post backtest results, IMO, would stymie collective thought and idea generation.
Most of us, I’m guessing, know intuitively that historical performance rarely repeats and therefore create systems with that in mind.
A private subscriber forum could relieve P123 of some of the liability created by exposing backtests. If you go to “who’s online” at any given time the guests outnumber the subscribers by a very large number…
I think everyone is mixing up two very different things.
Knowing how to make money
Knowing how to produce great break tests.
These two things have nothing in common. If I am going to invest real money in someone’s Designer models I want to see 5 years OS. At this point in time there are very few models who have beat the S&P over five years and even fewer who have beat the market since 2013. Congratulations Marc and anyone I missed. The odds for professionals are now less than 5% so this is no surprise. In the past I thought 2 years but it’s not good enough. The reason these 2 things have nothing in common is evident.
Designers models when failing are removed. This kills your credibility and starts the clock over for anyone who wants to invest in your models.
The designer models that are left the majority are not beating the index.
If a designer has 1 out of ten designer models that are beating the index but the other 9 are not this makes me think the one that is beating the market is a fluke.
If I were a Designer I would focus on ETF and trying to get a simple edge. I have posted about this before and have not been proven wrong yet. If I had to pick one model that I think will outperform during the next recession it is Georg’s Best(SPY-SH) Gains for Up & Down Markets. It’s an ETF with marketing timing and as he said until there is a recession it probably won’t beat the index.
I think the community has to ask some hard questions and grow some tough skin because you might not like the answers:
Which designer has the most credibility at making money and why?
Would you invest with any designer that does not have a proven track record of at least 5 years?
Would you invest with a designer who has a pile of graveyard models?
Would the designer community like to be publicly evaluated on these questions?
I don’t think the community is ready for these tough questions. Any time someone gets publicly criticized about their OS performance it starts a war of words that solves nothing.
Maybe there should be a forum for those who want to make money and one for those who want to backtest.
Thank you P123 and everyone else who contributes to these forums.
I actually like seeing the backtests and factor discussions. I learned a lot from those. if we move these to a separate group, i fear newer folk will not be able to learn by observation.
I would also like it the way it was, with free discussions and posting of ideas, charts and results.
Much to learn for everybody. Would be a shame to change that.
But I believe Yuval’s Crazy Returns Mico-cap Model (his only public micro-cap model) says much more about micro-cap strategies than any backtested micro-cap model I would try to present on the forum.
If I present a great backtested model and try to convince you to switch from Yuval’s model, switching would be a mistake.
Does not mean you shouldn’t take a look at one of my backtests—if you don’t like the Super Bowl commercials this week-end and have nothing else to do.
Uh… Honestly, you should look at the Super Bowl commercials.
A machine learning specialist would not argue against backtesting. They might do some things to limit how much overfitting you can do but within those constraints they would want you to backtest a lot and get the best backtest possible.
They would always validate that, however, even if that meant paper-trading the model for while.
We see how that works with designer models. I suspect every one of the Designer Models backtested well.
Anyway, backtesting is just a tool–like a hand gun–that can be misused.
A discussion of backtesting in isolation does not advance the discussion much IMHO.
But it is fair to say everyone—including de Prado–backtests and tries to get the best backtest possible (within any constraints).
Be a cold blooded investor.
Create, backtest and invest in your own rule-based strategies.
There would not be a need for backtest if there was a perfect theory to select stocks based on functions, ranking, etc. and always make profit on every trade. If such a perfect rule based theory existed, and only a few insiders know it, they would be the richest people on earth.
If it’s not used to sell a DM, I like seeing backtest results, too. It motivates me to explore techniques that I would otherwise overlook. I gotta thank Georg for the use of ETFs and recession indicators in portfolio construction.
And I am very exited about mixing stock trend and recession indicator models. I would have posted the rules and backtest here,
but unfortunately the community will not be able to see them under the new forum regulations.
But all is not lost, I will post the article on Seeking Alpha shortly.
Backtesting doesn’t tell you a model whether a model will work, but it does tell you what hasn’t worked. I like backtesting, but I try to just throw something together without tweaking values to the upteenth degree. I think Marc Gerstein mentioned on his backtesting in the design guide that he’s not looking for perfection in his backtests but rather does an idea yield some alpha. He tried to take top percentiles like take top 30% value and apply a core quality + core sentiment ranking.
Not all Marc’s strategies have performed well but on the whole they are the better performing strategies.
It’s better to be in vaguely right than perfectly wrong (Marc’s quote). Investing is not a precise endeavor so I think we need to avoid precision in our backtests to get better trading strategies. Easier said than done. It’s natural to want to improve.
I would also never trust my money with one strategy even if it OOS tested well. 3 or 4 strategies at least. Then at least if a couple flunk you can maintain capital.
But do we know that Marc is beating the benchmarks? I think we do not despite the fact the there is much to like about his methods.
Are all of his sims and ports meant for serious investing?. Was Piotroski meant for real money?
Which ports are serious?
So if you take the serious ports (whichever ones those are) which ones should we be in? Which should we be out of now? I do not know.
I like Marc and his ideas. But I make no claim whatsoever about having an accurate idea of how all of that might work for me.
Maybe someone has seen more evidence than I have one way or the other.
The only evidence I have is that on average his ports are underperforming the benchmarks I think last time I checked. Saying some of those models were not serious is only an argument that I know nothing.
Okay I know nothing on this. But I do not find convincing evidence for much that is said—or any evidence at all for that matter.
I do think the last two years have been a difficult investment environment. There’s a lot of strategies that just haven’t worked or underperformed. I have a couple strategies that have outperformed in backtesting but they are either momentum or quality based. Definitely doesn’t include value or traditional growth. I can get a value strategy to perform if I compare within industry, but there is a lot of value in financials and energy now which the market has been avoiding. Even still that value strategy has underperformed the benchmark the last two years. This is a bond market and quality names serve as bond proxies. Value performs best in high growth, high rate environments. We are not in that domain right now. Also consider that most stock market appreciation has been due to market Goliath Faang and Microsoft stocks.
So I think you are going to find it nearly impossible to develop a viable value strategy that has beat the market the last couple years. That might be a good thing because the more value gets beat down the more it will be coiled for a rebound. Depending on how much the market corrects and whether a recession occurs this change might happen sooner than later.
Maybe look for a value strategy that has shown outperformance in the past including before 2008 and accept that now is not values time but it could be poised to spring back at any time. I think that’s as good as we can do. Not every factor is going to outperform all the time.
