Don’t forget better looking!
That sounds like an interesting idea.
Perhaps we need not ban or delete posts featuring backtests if we confine them to a separate forum that one can enter only after clicking to accept and agree to understanding that portfolio123 does not support the use of backtesting as an indicator of what might happen in the future and possibly an amendment to our Terms of Use to the effect publication of p123 screen shots showing backtest results be accompanied not just by a credit to p123 and the data provider but also a statement that p123 backtests depict that past and that it cannot be used to infer future performance . . . something like that.
P123 users have the right to work as they wish. But P123 has the right to separate itself from practices we believe cast the company in a negative light. Failure have been diligent about this back when DMs (initially known as Ready to Go) launched caused that very promising business to blow up, possibly beyond repair (by the time we corrected it, the cat was out of the bag, the horse was gone from the barn, and the subscribers were gone from the great i-s/lousy o-o-s models). We can’t afford more mis-steps like that, especially as we seek to attract more professional subscribers (an important area where s&P had been badly blocking us). And individuals want us to succeed in this; it will help us keep your prices affordable. We can’t have pros publishing white papers and reports citing and sourcing to portfolio123 and having clients say “You mean you built your stuff THERE? Good bye. Good luck.”
I hope Georg and other backtest devotees find this a reasonable compromise. We’ve been good to you; and Georg, you have no idea how good we’ve been to you as S&P constantly kept looking at our subscriber list panting and dreaming about how they were going to unmask all these professionals who would be kicked off p123 unless they spent big bucks for S&P data licenses. In fact, a major reason why we’re taking the trouble to switch data providers is so that we don’t have to throw folks like Georg to the hungry wolf. Bottom line: We’ve been damn good to you and went through a lot of behind the scenes ugliness to protect your ability to continue to do your thing on p123 and run your site as you do. Now, it’s time for you to do some good for us, even if you don’t agree with what we ask; to agree to stop posting stuff in ways we believe will hurt our business. I think the suggested compromise is dam good idea, and a lot better for you and others like you than Marco’s thought of simply taking down posts that show backtests.
Frankly Georg, considering the WHOLE picture, there are only two words I want to hear from you (And others who value Georg’s site and and are happy we busted our buts and didn’t let S&P blow it into oblivion): “Thank you.”
And as a professional courtesy, I am asking you to stop posting your backtest results until we can implement some arrangement along the lines discussed here. (If you ignore me and if Marco decides to simply pull down those posts, when he calls me to get my opinion, I won’t argue; I’ll just say “Go ahead, take them down. “)
This sounds perfectly reasonable to me and is a Good Idea. I have always, and will continue like others to say: “Thank You” P123 !!!
In the meantime I will not post any backtests on P123. I do intend to write a Seeking Alpha article, something in the line of: “The Use of Recession Indicators in Stock Market Timing”, which will show how to use the tools on P123 to do this. I will provide you with a preliminary copy for review. The proposed article could bring a lot of potential subscribers to P123 who may want to run models based on the business cycle themselves on the P123 platform.
If P123 is concerned about sharing sim results (market timing ideas, ML, whatever), maybe taking it off the public forum and into a private Portfolio123 Group would work. As per its description - “Portfolio123 Groups allow you to create smaller groups of members with whom you can collaborate and exchange ideas.”
Walter
OK. Looks like we’re in agreement. Thank you for your cooperation.
Don’t know when Marco will see this so I’ll IM him to get his attention.
I’m with Walter on this one. A private forum for subscribers would allow the exchange of ideas, including backtest results, without the general public being exposed to them. After all, backtesting is at the core of P123. To not allow subscibers to post backtest results, IMO, would stymie collective thought and idea generation.
Most of us, I’m guessing, know intuitively that historical performance rarely repeats and therefore create systems with that in mind.
A private subscriber forum could relieve P123 of some of the liability created by exposing backtests. If you go to “who’s online” at any given time the guests outnumber the subscribers by a very large number…
Yes, this would be implicit in the compromise solution to which Georg and I agreed.
I think everyone is mixing up two very different things.
- Knowing how to make money
- Knowing how to produce great break tests.
These two things have nothing in common. If I am going to invest real money in someone’s Designer models I want to see 5 years OS. At this point in time there are very few models who have beat the S&P over five years and even fewer who have beat the market since 2013. Congratulations Marc and anyone I missed. The odds for professionals are now less than 5% so this is no surprise. In the past I thought 2 years but it’s not good enough. The reason these 2 things have nothing in common is evident.
- Designers models when failing are removed. This kills your credibility and starts the clock over for anyone who wants to invest in your models.
- The designer models that are left the majority are not beating the index.
- If a designer has 1 out of ten designer models that are beating the index but the other 9 are not this makes me think the one that is beating the market is a fluke.
If I were a Designer I would focus on ETF and trying to get a simple edge. I have posted about this before and have not been proven wrong yet. If I had to pick one model that I think will outperform during the next recession it is Georg’s Best(SPY-SH) Gains for Up & Down Markets. It’s an ETF with marketing timing and as he said until there is a recession it probably won’t beat the index.
I think the community has to ask some hard questions and grow some tough skin because you might not like the answers:
Which designer has the most credibility at making money and why?
Would you invest with any designer that does not have a proven track record of at least 5 years?
Would you invest with a designer who has a pile of graveyard models?
Would the designer community like to be publicly evaluated on these questions?
I don’t think the community is ready for these tough questions. Any time someone gets publicly criticized about their OS performance it starts a war of words that solves nothing.
Maybe there should be a forum for those who want to make money and one for those who want to backtest.
Thank you P123 and everyone else who contributes to these forums.
Cheers
Mark V.
I’ve posted below the out-of-sample results of my delisted designer model Crazy Returns, Large Caps, by Jim’s request.
Thank you Yuval.
And in purely mathematical terms: pretty darn good.
-Jim
5/24/2017 to 1/31/2019 is a tough period for the market. I don’t think that there are many models that did well over this period.
I actually like seeing the backtests and factor discussions. I learned a lot from those. if we move these to a separate group, i fear newer folk will not be able to learn by observation.
I would also like it the way it was, with free discussions and posting of ideas, charts and results.
Much to learn for everybody. Would be a shame to change that.
RT,
I am okay with some backtests myself.
But I believe Yuval’s Crazy Returns Mico-cap Model (his only public micro-cap model) says much more about micro-cap strategies than any backtested micro-cap model I would try to present on the forum.
If I present a great backtested model and try to convince you to switch from Yuval’s model, switching would be a mistake.
Does not mean you shouldn’t take a look at one of my backtests—if you don’t like the Super Bowl commercials this week-end and have nothing else to do.
Uh… Honestly, you should look at the Super Bowl commercials.
-Jim
I completely agree with the above.
I also particularly like the back tests and charts that George posts.
I now own a domain address for this, just in case.
P123BACKTESTS.COM
If the P123 community wants a forum away from P123, please let me know so I can get somebody to set it up for us.
Just complete the Golden Cross & Recession Timer (Stocks-Bonds) 1999-2020, but I can’t show it to you under the new forum regulations.
Not that it matters to this discussion but:
A machine learning specialist would not argue against backtesting. They might do some things to limit how much overfitting you can do but within those constraints they would want you to backtest a lot and get the best backtest possible.
They would always validate that, however, even if that meant paper-trading the model for while.
We see how that works with designer models. I suspect every one of the Designer Models backtested well.
Anyway, backtesting is just a tool–like a hand gun–that can be misused.
A discussion of backtesting in isolation does not advance the discussion much IMHO.
But it is fair to say everyone—including de Prado–backtests and tries to get the best backtest possible (within any constraints).
AND AT LEAST IT IS SOME EVIDENCE.
-Jim
Also here is what P123 has on their front page:
Be a cold blooded investor.
Create, backtest and invest in your own rule-based strategies.
There would not be a need for backtest if there was a perfect theory to select stocks based on functions, ranking, etc. and always make profit on every trade. If such a perfect rule based theory existed, and only a few insiders know it, they would be the richest people on earth.
If it’s not used to sell a DM, I like seeing backtest results, too. It motivates me to explore techniques that I would otherwise overlook. I gotta thank Georg for the use of ETFs and recession indicators in portfolio construction.
Walter
Maybe that should be the deciding factor.
Also I would not mind if P123 actually took some concrete steps to help us identify over-fitting.
I mean some of the methods the AI specialist P123 hired used.
And the steps Yuval took: no survivorship bias in his models. DOCUMENTED FACTS OF HOW IDEAS ARE WORKING OUT-OF SAMPLE.
-Jim