Method 1: Full Universe, Oversized Backtest Portfolio
How it works: Run every simulation on the full 2,500-stock universe. The backtest portfolio is set to hold the top 100 ranked stocks at each rebalance. Optimization selects factor weights (LGBM surrogate + CMA-ES) that maximize performance of that 100-stock portfolio. Live trading will then take only the top 25 from the same ranking.
Why I considered it: With 100 names, idiosyncratic noise is heavily diversified away, so the fitness landscape is much smoother across thousands of simulations. The idea is to let the optimizer learn robust factor signals, not get lucky on 25-stock draws.
Obvious problem: Backtest and live have different portfolio sizes. The backtest will overstate Sharpe and understate drawdowns because variance scales with 1/sqrt(N).
Method 2: Split Universe, Size-Matched Backtest
How it works: Split the 2,500-stock universe into 4 roughly equal sub-universes (∼625 stocks each). Run the same optimizer, but each simulation holds exactly 25 stocks, matching live size. I run the optimization independently in each of the 4 buckets, then aggregate the results (sum of fitness) to choose the final factor weights.
Why I considered it: Portfolio size, turnover, and idiosyncratic risk now match live trading exactly. No size mismatch bias.
I would go with method 2 if given the choice. But there's a third choice, which is hold the top 100 in the 4 split universes. I would favor that one, for sure. It's the most robust.
Thanks, Yuval, I know you're good at this. The idea of splitting it up into 4 sub-universes I got from some of your posts, but..are you not afraid that by including as many as 4 times the number of stocks in each simulation as you would retain in a live strategy, that other parts of your portfolio—not the top 25—are driving much of your returns?
And why so high? What added value do you gain by holding 4 times more than the live strategy? Wouldn't much of the idiosyncratic risk be reduced when you are above 25 stocks?
When you're backtesting, you're trying to figure out what factors have driven returns in the past. If you use too few stocks, you're adding a lot of idiosyncratic returns that may have little relationship to what comes forward. If you use too many, you're losing the idiosyncracies of your portfolio management system. There's a balance. In my correlation tests (correlating backtest results with future performance) I've found it to be 5 universes and 3.75X the number of stocks you'll actually be holding, which means including, in total, about 19X the number of stocks you have in your portfolio. That just makes things more robust.
That said, I don't think this all makes a huge difference. There's no one right or wrong way to do this stuff, and out-of-sample returns are going to be pretty idiosyncratic no matter how you backtest. Far more important than getting the right weights for your factors is factor design. If you're using straight-out-of-the-box factors you're never going to get the kind of performance you'll get with factors that you've thought through and designed yourself, based on serious and thoughtful consideration of how they actually work. Remember that accounting measures are not scientific--they're human artifices--so only thinking deeply about how they work, or how they don't work, is going to give you an edge.
This study of yours was very interesting. Please tell me more If you are not using sub-universes, but just one universe, the ratio will still be the same, meaning if you have 10 stocks in the live strategy, should you have 190 stocks in the backtest optimization? And as I understand you, even if you test with so many stocks in the backtest, such a large portfolio will still, in an optimization, provide stock criteria that assist in good selection of stocks among the top 10-25 stock in your live strategy?
In the study, if I remember correctly, I took about 100 strategies (none of which I developed myself, so they hadn't been backtested) and looked at how they performed over various in-sample and out-of-sample periods depending on the ratio of stocks in the in-sample to the stocks in the out-of-sample periods, using 5 subuniverses for the in-sample periods and none for the out-of-sample period. And I found that using a multiple around 3.5 or 4 was best.
I can't swear to the accuracy of that study, but it's the general drift.
I do not recommend using only one subuniverse for testing because you lose the advantage of averaging several different systems after optimizing on each subuniverse. That advantage is pretty important to me. It allows for a great deal of variation in each ranking system and makes things more comprehensive in the end.
I have 1500 stocks in my EU universe and 2500 in USA/Canada. If I run with 5 sub-universes, it means at least 100 stocks out of a total universe of 300, which is almost 1/3 of the sub-universe. What did you do there? Did you test with nearly 100 stocks, even though it constituted as much as 1/3 of the sub-universe?
Hi Yuval. Just a quick question about your statement here:
“That advantage is pretty important to me. It allows for a great deal of variation in each ranking system and makes things more comprehensive in the end.”
Are you talking about averaging things at the factor weight level or the portfolio level? So I mean, are you averaging the weights of the ranking system from the 5 different subuniverses into a final system, or do you run all 5 ranking systems (which is just the same ranking system with different weights), and the averaging is meant at the portfolio level?
Okay, thank you. What would be the maximum limit you would extend to for the total size of the stock portfolio within the sub-universe? For example, 5% of the sub-universe can be used as a portfolio in each simulation?
If you are not using sub-universes, but just one universe, the ratio will still be the same, meaning if you have 10 stocks in the live strategy, should you have 190 stocks in the backtest optimization? And as I understand you, even if you test with so many stocks in the backtest, such a large portfolio will still, in an optimization, provide stock criteria that assist in good selection of stocks among the top 10-25 stock in your live strategy?