Does Past Performance Predict Future Returns: A look at designer models

[Jrinne]

To a large extent this is what was easy with the designer model downloads and not necessarily the best method for predicting returns. Still…

There is an interesting discussion about Backtests here: Why the poor live results versus backtesting?

This is related but a slight different topic. So I started a new thread. Summarizing the previous thread, and maybe over-simplifying when doing so, it may be fair to say there is a complex and not fully understood relationship between in-sample and out-of-sample results. Some have suggested that backtested results are not important for predicting future returns. I tend to agree with that.

In any case, that leaves open the question: How do we pick a model? Maybe we want to look at the relative performance of backtests, use validation sample results,, test sample results or even just look at the out-of-sample results of a live port for picking a model?

So lets try the last. I had Claude Fable 5 help me with this. But with download from the designer models of the 1-year and 5-year returns we already have the 1-year returns and the previous 4-years can be calculated from that data. And a clean graph and R^2 or coefficient of determination can be obtained.

Conclusion: Somewhat surprising to me. Enough to be without words at this time. Maybe someone else has comments. Maybe the possibility of survivorship bias is worth mentioning, but otherwise without comment:

[regallow]

My uneducated / unvalidated guess is that the comparison shows that, taken as a group on average, the designer models tended to strongly match the overall market in the latest year’s return. During today’s market, SPY is up 20.26%, RSP is up 16.24% over one year.

It might be helpful to break the models into groups defined by the index they compared themselves to. if there are enough models in any group to make a valid comparison to their index, that analysis might show how difficult or easy it is to improve on stock returns in that segment of the market. Just my thoughts…

[Jrinne]

Great point!!! I agree that excess returns are better. That is hard to get from the Designer model downloads, however. For 5 years (or longer) that is not possible. But the models are being compared to themselves here, making correlations of one model to itself over different periods valid, I think.

I find zero correlation and zero predictive power from one period to the next in this study interesting.

I would really like to look at the correlation of backtest results to out-of-sample results too, but the designer model dowload does not have enough data for that.

[yuvaltaylor]

@Jrinne

How do we pick a model? Maybe we want to look at the relative performance of backtests, use validation sample results,, test sample results or even just look at the out-of-sample results of a live port for picking a model?

You left the most important thing out. You pick a model by reading the designer's description of it, combining a healthy dose of skepticism with your own opinions about what's likely to work.

[ars7777]

@yuvaltaylor

You left the most important thing out. You pick a model by reading the designer's description of it, combining a healthy dose of skepticism with your own opinions about what's likely to work.

I spend a lot of time perusing the Designer Models market.

I'd argue that even more important than reading the description is getting to know the designer — through forum posts, direct messages, email, maybe a call. What I'm gauging is how well they grasp key concepts and how open-minded they are to competing, conflicting, or new ideas and factors. I spend an inordinate amount of time noodling on portfolio construction, hedging, and related concepts — probably to an unhealthy extent — and I always seek out designers who are similarly borderline obsessive about factors and model construction.

Two semi-related thoughts:

1. Can we do away with the 3-month incubation period?

I get the intent, but three months doesn't vet or prove anything — and the live record accrues from launch date, whether or not anyone can subscribe, so the gate adds no informational value. It just delays willing subscribers. There are at least two incubating models right now, from designers I know and trust, that fit a need I have; I'd subscribe to both today. I often subscribe to a model just to get a feel for its holdings, turnover, etc., and see how livable it would be for me — sometimes I trade it, sometimes I watch for a while. Put a warning on incubating models if you must, but let us subscribe. If the real purpose is to keep designers from launching many models and promoting only the winners, there are better ways to get there (e.g., the previously discussed idea of perpetually tracking all of a designer's models, including deleted ones).

2. Why not let us license ranking systems the way we license Designer Models?

Opt-in for designers, priced however they like: locked, nodes hidden, usable in a subscriber's own screens, sims, and books — with whatever safeguards designers' IP requires. It seems like a natural extension of the DM trust model.

I may be misunderstanding how some of this works behind the scenes. If so, apologies in advance — I'd welcome the correction.