The idea that many of us have is that investing a small portfolio means you could harness mispricings that larger investors can’t bother to exploit. Its intuitive to me that the only remaining “edges” retail investors could have is 1. disciplined and informed approach to microcaps or 2. patience / time arbitrage.
It doesn’t seem possible to generate 50% returns in the real world, even with an iron stomach.
What returns do you think one can generate using P123 in microcaps? in larger stocks?
I have two micro/small cap strategies that have +5 years of out-of-sample, live data. The annualized returns are 35% and 45%. The DDs are similar to market DDs.
In addition, I’ve been experimenting with using investment research buy/sell recommendations as the basis for a universe. Early results look promising with 65% annualized returns.
Model turnover is about 350%. I’m a discretionary, manual trader. My models form the basis for some of my picks. One process I’ve settled on is to select several models, include their holdings in a screen (via Portfolio() statement), and then rerank those holdings to get the top 30. From that point on, it’s discretionary.
I’ve garnered an eight-year CAGR of 44%, if that helps. That’s for my entire portfolio, which is pretty large now.
The much smaller accounts I’ve been managing for my kids have substantially outperformed my overall portfolio due to more concentrated positions, lower transaction costs, and the ability to buy super tiny stocks. Those accounts have gone up 232% in the last three years compared to my 158%.
That really is incredible. One other concern I have (for outside of microcaps) - P123 seems great for fundamental and technical factor testing. Heavy duty quant shops like AQR are doing things like using machine learning to compare the transcripts of earnings calls to detect the tone of management in a more quantifiable way. They also have access to a lot of other alternative data. That example is why I worry a bit about competing anywhere groups like that can play.
I used a crazy amount of alternative data at WorldQuant. My personal take on it is this…fundamental data is king. It stands the test of time. I can’t tell you how many different data sets looked amazing with gorgeous backtests and after a couple years it just sort of pooped out and went down hard. I found that instead of looking for the latest alt data, it was more profitable to try and look at existing fundamental data in a unique way.
Over the last 4 years, I’ve managed a CAGR of about 37%. 2 main strategies (“GUTS”), US & Canada (4 strategies total), mostly microcaps.
That said, during my “learning phase”, the first few years were very volatile (2016-2019), and it took time to develop my systems (they were much too over-fit in the beginning).
It is my understanding that @marco and @Riki37 are working hard to provide at least a couple methods of cross-validation for P123. Namely time-series validation and k-fold validation. Perhaps they would agree that it can be helpful. They have spent a lot of money and time to provide it–resulting in significant delay of the AI/ML release.
Independently, a bunch of people using open-source libraries at Sklearn would agree that cross-validation can prevent overfitting. And provide answers as to what to expect out-of-sample which I think was your first question.
Anyway, cross-validation (train/test splits) would suggest that 50% returns may be possible. Here is a time-series validation supporting what people have said above well as providing one answer to your question about overfitting (average 54%):
Thank you @marco and @Riki37 for striving to provide a real AI/ML tool. Even delaying the release to do so.
The over-fitting was clear after the fact as out-of-sample fell apart after 1 year. Some simple considerations:
very few positions (only 5 holdings, less data points) - prefer to have 15-20 now
using only the simulations as backtest performance tool - ranking decile charts can provide much more information on how effective a factor has been
had very specific buy rules (too many) - now prefer to keep to a minimum, capture most rules/criteria in the ranking system
factor weights - assigning weights to improve sim performance only - now prefer to test each factor individually, look at different time periods & universes
I would like to contribute to the discussion how to more trust your ml model (reduce variance).
For p123 users who uses tree-based model, I would recommend to use ‘Monotonic Constraints’ ( monotonic_cst - has been added to scikit recently). This constraint ‘enforces’ positive relationship between a future and target during learning process.
Below you can see chart of a decision tree with simulated data where, I imposed positive constraint on ROE and negative on Accruals, then the model would use this guidance and use ROE as a first split (most important). If I would impose positive constraint on Accruals then this feature would not be selected at all in this toy example.
This is of course simplification. Nice explanation can be found here: https://xgboost.readthedocs.io/en/lat est/tutorials/monotonic.html. You can use this feature using Random Forest or XGBoost. Be careful with imposing monotonicity constraints to all features (e.g., sales growth).
Constraints: ROE positive, Accruals negative, can be read as if ROE <= 50, and Accruals <= 15 then average return within two samples = -10%.
I will perhaps edit this later. But about your use monotonic constraints. It takes a very knowledgeable person to just know about this. Let alone have understood it and used it.
Thank you for your contributions.
BTW, I ran into an extremely well-qualified profile on linked in. The person bore a striking resemblance to you.
I wonder if you would want to link to that. I would but wanted you permission first. My profile. Uh, not so pertinent to this discussion. This person’s profile (or training and experience) was pertinent do this discussion.
Edit (my apologies for being a little repetitive of what Pitmaster posted. My link works when I try it so will not edit it out): Monotonic Constraints is also a part of XGBoost. I am not sure of the implementation of XGBoost but it may be available within the P123 platform soon. Perhaps @marco or @Riki37 will want to post on the possible implementation of this feature and its usefulness.
In XGBoost you can set a boolean to turn monotonic constraints on or off for each factor. Cross-validate (with a GridSearch) to determine whether it is helpful for each factor if you don’t think you know ahead of time. @Pistmaster has introduced something worth trying, I believe.
I really appreciate the answer. The learning on this community has been great. Honestly, I have to learn a bit more before I can understand the reply fully- I’m not familiar with k-fold.
