Do we think that the best models are actually put forward by the designers? Would you put your go to best model up as a designer model?
My main purpose was to determine if “Past Performance Predicts Future Returns” and how that might relate to my own models. I did not test your question with this study.
Great idea using Claude for this! Here’s the full dataset for anyone else to use attached below if they need (I pulled using Fable).
The good news is that models reaching closer to around 100% annualized return do appear to outperform their benchmarks.
Here’s Claude’s analysis:
dm_is_oos_analysis.csv (14.5 KB)
designer_models_full.csv (84.0 KB)
5 Likes
Thank you, Quantopian! Good stuff!!!
Adding to what you have done, I used your spreadsheet to look at this with time-weighting of the models. Where a model that has 13 months of out-of-sample data would not have the same weight as a model with over 5 years of data in the regression. Fable 5 helped me with this again.. Here are the results:
The point to the far right has a lot of leverage. With it removed here is the result:
Clearly, the better and worse performing models are the ones with shorter out-of-sample windows. The asymmetry possibly being due to poorly performing models being removed quickly (the survivorship bias). Or maybe due to recency bias. Or a little of both.
As previously noted, this mirrors Kahneman’s 'small schools' observation from Thinking, Fast and Slow—small sample sizes produce extreme outcomes (good and bad). As time goes on and the out-of-sample record grows, the sample size increases, luck washes out, and regression toward the mean takes over.
2 Likes
Also a good point. I’m sure there’s lots of models with significantly higher in-sample and OOS returns not posted here.
There is another thing (besides survivorship bias) that makes comparing sims to out-of-sample results difficult. That is revisions. Are we even comparing the same models now?
I see some models that did very well when they went live. And for a good period of time. But they have gone on to underperformed for the last 5 years. Recency bias? Mean reversion? Something else? Or maybe they just are not the same model anymore (after revisions)?
3 Likes
Assuming no model revisions, I think definitely either market regime changes, or perhaps its also possible that the edge the model identified was subsequently found by enough market participants to where the edge was arbitraged away. I’ve read a few times that that most strategies will eventually stop performing well, it’s just a matter of when not if.
1 Like
I agree. Regime change seems like an important factor. I would just add that recency bias is the same thing. After a while--with no revisions--we end up in a new regime that is not part of the backtest. Presumably a model not trained on the new regime is less predictive.
A new model or a revision would update its training to the new regime—essentially resetting the clock on recency bias and perhaps perform better. This makes revision a complex topic. For sure, I would want a Designer to continue to have the option of updating the same model with new data.
But we used to think 100 features was a lot. Now we hear of models with 300 or 600 features. This seems to be a general trend in the forum that may or may not be reflected in the Designer models. I do not know if it is the case but if Designers are adding that many new features, then I have to wonder if that always helps or if it is the same model as when it was first introduced.
1 Like
Designers keep adding more features (sometimes going up to 300 or 600) and using more complex AI approaches. The issue is that most of these new features, new formulaa, linear or not, are still derived from the same core data (Factset for example...). They rarely add meaningful new information, and the extra complexity tends to make the model more prone to overfitting past regimes.Instead of just adding more and more features when performance drops, a better approach in high-dimensional settings is often to use techniques like PCA (as Marcos López de Prado has discussed) or just use ortogonality matrices. This can help reduce redundancy and multicollinearity among features without simply increasing model complexity. In many cases, this can be more effective than continuously expanding the feature set.Another important factor is lack of patience. Some models can underperform for long periods and then recover strongly when market conditions change again. A good example is the momentum factor — it looked largely dead in many studies between 2010 and 2020 (you can take a look at book Momentum for example to expand the info), yet it came back quite strongly afterward.
Constantly revising or overcomplicating a model during a long drawdown can actually destroy its long-term edge.Regarding funds like Medallion, they didn’t become top performers mainly by building extremely complex formulas on traditional market data that everyone else already had access to. From what’s publicly known, a significant part of their edge came from working with much larger and more diverse datasets, combined with very disciplined statistical processes. This is quite different from just adding more derived features on standard data.
In the end, when models start struggling after a few years, it’s worth asking whether the degradation is due to a regime shift, alpha decay, too much people in the same boat, lack of patience during a temporary drawdown, or simply because the model has become too complex and overfitted to previous market conditions through constant revisions and feature additions.
6 Likes
For those who has not experimented with PCA, try it out, it's quite interesting.
1 Like
It's also a matter of for how long. Many strategies will stop performing well for a while and then start performing well again.
1 Like
Designers can address this themselves by pre-registering their models at OCF.io
This discussion got me interested in Designer models and what we can infer from them. I thought I would add to the discussion by submitting 2 of my own models and pre-registering them: Two Pre-Registered Designer Models on OSF
2 Likes