Anyways I wasn’t implying anything sinister, just that I tend to use a wide range of factors on P123, some of them less traveled with higher potential for bugs to show up.
BTW - You will be glad to hear that I’m currently reading a valuation doc by Aswath Damodaran.
It is very well written and explains in great detail many of the concepts you (Marc) are trying to educate us on. It is so thorough … my only complaint is that there is so much info presented it is hard to absorb it all. I recommend this document to everyone.
The primary measure between OOS and IS performance is alpha stability. OOS alpha should be at least 50% of IS alpha.
Sharpe is not correct measure. Market performance has significant influence on it. If you decompose sharpe = alpha/sigma +correl x Market Ret / Market sigma = a/s+p*Smarket. It means that if market has higher sharpe and you system higher correlation to market (you want lower correlation usually) you will get higher sharpe for your system all else being equal.
Anyways I wasn’t implying anything sinister, just that I tend to use a wide range of factors on P123, some of them less traveled with higher potential for bugs to show up.
So much info? Heck yes. Actually, calling it a doc is only a save-as technicality. It looks like a bootleg copy of one of his textbooks!
More specifically, it’s a bootleg copy of one of the books I recommended in the Help section “Recommended Reading” list. So yes, I would be very happy to see people tackling it.
Little? The link that Steve posted produces a 600+ page textbook (a well regarded textbook, but one you’re not going to read in one sitting).
He also has a separate book called “The Dark Side of Valuation” (I beleive a lot of this material is included in the textbook you can access vbia Steve’s link. The Dark Side text was originally written to show how to approach internet stocks, and there is a later edition. Damodoran maintains that there is never an excuse to avoid considering value. But somesituations do call for different and inb some cases more relaxed appraches. I don;t recall if he said this in a book, but at a presentation I atended, he said the goal is not to be right but to be “a little less wrong than everybody else.”
Steve - R2Gs underperform OOS vs IS during the bull market, there is not much difference in market trend since QE3 until 2015Q2. So no difference in launch date for all R2G with launch period >360 days till 2015-02-28. Value, growth, momentum, whatever, everything underperforms.
Denny - what “difficult last year market”? Even last 2-3 month is nothing comparing to IS 2000 and 2008. Looking at market since 1999 till most R2Gs launch during 2013 I’d say this period on average was much less favorable than last 2-3 years. Note my previous note, I am actually comparing OOS to Combined (IS+OOS), so IS is already somewhat adjusted to last year difficulties. More, stats as of 2015-02-28 looks not much better than as of today.
What about those outperforming FREE R2Gs? There is their QE impact, “another more subtle reason” and “difficult last year market”?
For me it looks like Russian anecdote: First I was drinking champagne, after switched to wine and then had vodka and beer… and then, you won’t believe, poisoned with cookies!
As for Steve question, the fact I am presenting is not about R2G selection, it is about filtering. If backtest fails to “predict” (at least to some degree) the future model performance, then see you in 10 years with OOS. I am not expecting very close consistency OOS vs. IS. Monitoring this stats weekly for some time, it is quite volatile, no wonder for short OOS and reasons mentioned, still average 3 year of bull market R2Gs OOS underperformance of >50% and almost no outperformance is really worrisome.
I would probably leave P123 at this point if not those outperforming R2Gs. If P123/Marc would launch design class for subscribed investors I would, probably, get into design by myself, at least to get rid of overoptimization caused mostly, I think, by marketing efforts.
Strategy design class? Sounds good to me if anybody is interested.
I have been trying to share my approach and would love to continue doing so. (Actually, it’s not really “my” approach. I’m not that smart. The approach I’m trying to share is based on well-established and very widespread knowledge and practices in finance.)
The problem is that too much of it has been in dribs and drabs in response to topics that arise in the forums, rather than as a systematic effort to start at the basics and build up – and then, over time, things get lost except to the very few (if any – not including me) who are exceptionally good at searching for old forum posts. And also, I’m human – I get fatigued constantly having to argue and debate the same points over and over agains with those who disagree with the approach I use. It’s perfectly fie for people to disagree and criticize and whoever is game for that is welcome to continue. What’s not OK is for disagreement to drown out the things I’d like to teach for the benefit of those who are interested in learning them and for disagreement to become a distraction that drains my time.
So i have an idea – if there’s interest.
How about I use the “Collaborative Groups” portion of the forum. If you aren’t familiar with it, click Community and hunt around. Perhaps I could establish a Group called “P123 Strategy Design Class.” This group, like any other formed by any other users, would have its own message board and the capacity for screens, ranks, sims, etc, that are enabled for and shared among members of the group.
I envision that I would open a topic by starting a clearly labeled (and numbered?) thread with a post and/or a link to content posted on p123 (a pdf, perhaps) or elsewhere. (I try as best I can to say things in a setting that can be seen by a broader investing public with the aim of attracting new members, either among those who want to design strategies and also those who could become interested in coming to R2G and subscribing to strategies.) For example, Topic one might be: “How stocks are priced” and be seeded with links to sa four-part series I published on TalkMarkets.com. The rest of the thread would be for questions and discussion on that topic.
In addition to substantive topic threads, there would be another thread entitled “General questions, comments, discussion.”
I would make it an open group, meaning any p123 member could join. I also think there would be two rules for continuing membership:
Stay on topic. in other words, in a topic thread, don;t post anything that isn’t connected with that topic. If you aren’t sure or want to go off topic, use the General questions, comments, discussion" thread. I think this is very important because for this to work, it;'s going to have to be easy for users to locate information a=on any topic and to be able to do so at any time – if they join late, if they want to review later, etc.
Respect the purpose of the group. That means that this would not be the place to express disagreement or alternative views. That’s for the general forum, and/or for any other collaborative groups any other users might wish to join. For a strategy design class to work, it’s important that those who want to participate be given an opportunity to do so without being distracted should they so choose. And it’s important that I be able to invest my time in working with the group and not debating those with opposing views (something I can and have done in the open forum and would continue doing as time permits).
If there’s interest, say so and I’ll set it up today in the group portion of the forum – whether it’s one person five people, 50 people or 300.
Marc, I support this idea, especially rules you propose, and I will definitely participate. I would propose one more rule - simplicity, not only to let beginners start, but for some “advanced” users with unrealistic expectations to get back to reality. I would very welcome case studies and practicum, at least one collectively developed model from ground zero will give to newbies much more than reading all the forum and P123 guides content. And I am ready to share all my newbie designer questions to shape the topics structure.
Yury, since 2015-02-28 to 2015-10-10 R2Gs subscribers count fall by ~13% and sales fall by ~17%. For sure, users panic on any correction and august impacted, but… the fact is really “fun”.
Konstantin/Yury - Of the 4 free models that are outperforming IS, there are two largecaps: (1) Cherrypicking the Blue Chips Standard; and (2) Folio. I assume that the Folio version is a minor variant of the standard version (so there are really 3 not 4 outperforming models). I have put them on a graph with Stitts Wealth Creation, starting from Sept 2013 when the Stitts model was launched. Now what is the difference in performance between the Stitts model and Cherrypicking models? Stitts outperformed for the first 6 months, then under-performed from March 2014 until about June 2015 when Stiits was revised, and is now picking up the pace again. The difference in performance between these models is noise. Under some market conditions one model will outperform the others and this is why you want to diversify with multiple models/providers.
You are dwelling way too much on past performance (or simulation as the case may be). You need to focus on the model strategy / developer and whether they are right for you going forward.
Konstantin, interesting statistics. I think this process looks like that. New users come to R2G, trying trading 3M-6M, don’t get what they see on graphs, leave it and never come back, new users come so overall number stays more or less stable. I suppose new rules should be established for R2G. Maybe it won’t be impressive but achievable in real life. If systems show 50%+ alpha you should count on least 25% (roughly for example, OOS two times worse than IS)) in my view. But if realized alpha is -5% or 5% it is not ok at all. Also current R2G numbers create inflated expectation from the beginning to ordinary users. I think P123 team should take more attention to all this stuff, maybe evaluate models somehow from their own side and experience, disclose more information about systems and it’s ranking. Or maybe to make their own robust models, launch copy trading through IB based on it, promote it externally and attract broader public finally.
I’m curious about how this would work in practice. The alpha displayed in R2Gs are annnalized from inception and cover many simulated years. Wouldn’t alpha measured from launch be much more volatile? For example, the rolling tests with a one year holding period show alphas bouncing around. It seems that if you want to compare alphas, you’ll need to use comparable timeframes or fairly long timeframes. How do you see this working?
Yury, I think you are right about come-and-go process of R2G subscribers. Subscribers asking for mind blowing performance, so designers supply pretty looking IS, 3M-6M of OOS underperformance and subscriber gone realizing this is just another crap selling. There is also R2Gs price factor and under-supply of realistic models. We had a discussions on both with designers with no results. Evolution fails.
No wonder, most of the retail investment industry is build upon this crap selling model. I just interested if it suits P123 strategy and if no, then is there any will to brake this circle.
What you propose on over-optimization disclosure was already discussed and much awaited with new R2G presentation screen. But I am not sure if real disclosure will take place as looking on OSS vs. IS stats presented it is not hard to suppose there are massive over-optimization in R2Gs supply at the moment. So the new presentation with over-optimization disclosure could just blow out R2G project. This way it would be wise just to keep everything as is and fail Revolution.
Yes, right now R2G is a game where designers put up spectacular simulation results, charge a lot of $/mo., and the real-time performance is all over the place.
I had subscribed to a high AR model, and in real time, it bombed and I lost a lot of real money. In this case the particular R2G is still available. In other cases, I know of R2G models that have since been taken down, so any study you do right now of R2G has survivorship bias.
However, I am also trading 2 other high AR R2G models. One is flat and one is up ~50% from when I started trading. Overall I am up on R2G, but a lot of it is due to luck (I put a small amount into the one that bombed, and put a large amount in the one that is up.) I could have easily chosen different R2G or different allocations, and I would be down and I would not be happy.
In the past I was also guilty of putting up R2G with spectacular looking AR but in real-time the performance was disappointing. Recently, I’ve been releasing a number of R2G that are low turnover, low AR, and I believe they will be much less disappointing in real-time. However, it is hard to attract subscribers to low AR models.
Steve, we are not comparing models performance. As I said, my point is not about selection, but about filtering. If model “lies” I don’t need it even with 100% return, because next month this could be -100% return. So, the difference between models in your comparison is HUGE and it is trust.
Unfortunately, no single R2G model fully enough disclosed for me to “focus on the model strategy / developer and whether they are right for you going forward”. What’s why my choice and priority are OSS vs. IS consistency and self-developed models.
MisterChang, there were a lot said about under-subscription and lack of interest to low return models. No wonder, this is very different retail investment market segment and you won’t win it with marketing efforts tailored for other segment. Wealthfront like model could fit, but first we need to have the quality supply and competitive pricing policy.
First, they change R2G simulations to 1 year rolling tests. You should be able to see all parameters year by year. Then you see alpha stability. It should be more or less stable within market cycles/similar conditions. Then you compare it to OOS parameters year by year again and the average for the whole.
Second they should introduce ranking reliability parameter based on it’s correlation to future stock returns within specific time frames based on specific history periods.
For example, you probably buy lower ranked stocks (means less forecast return) with higher reliability (means past correlation) than vice versa. R2G shows very high returns but with very low reliability. Ranking have to work stable within at least >3 years. Less tested period the less is ranking correlation, therefore you have to base your models on much larger numbers of holdings than 5 stocks. In that case you expecting to achieve averaged result, not lucky or unlucky realization.
There are a lot of other important things I will post soon, I have to check it before.
About P123 strategy and proposed revolution:). I think they should start from it. Anyway if they continue to overlook arose problems they reduce their progress. Critical mass of factors used in ranking (they still spend time on it?) is already achieved long time ago. It is time for more important things now.
Fun quote from Charlie Munger to Jason Zweig (via Tren’s book)
“I know one guy, he’s extremely smart and a very capable investor. I asked him, ‘What returns do you tell your institutional clients you will earn for them?’ He said, ‘20%.’ I couldn’t believe it, because he knows that’s impossible. But he said, ‘Charlie, if I gave them a lower number, they wouldn’t give me any money to invest!’ The investment-management business is insane.” http://mebfaber.com/2015/10/20/what-isnt-possible/
Yury, the main question here is what is the P123 strategy and how it fits into insane investment-management business? What we see could be just a part of some strategy, then why worry?
Marco/Marc - are there any resolution after summer strategy brainstorming?
Also forgot to mention one thing about ranking results presentation. Why don’t to show alpha instead of returns on ranking histogram?
It is very inconvenient to compare performance visually on different time frames because of very different market returns.
I worry because I need it for my future small hedge fund investment management process. And now at presentation preparation stage (I can’t mislead my future investors more than two times:) Promise 20%, have to get at least 10%. In other case I’m dead:). I can’t afford 500K annual budget for the powefull S&P systems and a team of programmers now.
