Reoptimizing a Multi-Nodal Ranking Sytem

General
1

I would be interested in learning what methods others are using to re-optimize multi-node ranking systems. By “multi-node ranking system,” I mean systems like the Small & Micro Cap Focus ranking system, as one example.

In the interest of both giving and taking, I’ll share my current process. I would be interested in hearing what others are doing, as well as any suggested improvements to the method below.

At a high level, I use a stepped, node-by-node optimization process. It is essentially a practical coordinate-search method rather than a full global optimization.

The total weights in my ranking system add up to 60, recognizing Portfolio123’s standard note that weights are automatically normalized to 100%. Because of that normalization, I am focused on the relative weights between nodes rather than the absolute total.

For each candidate weighting, I run both longer-term and shorter-term simulations/backtests, generally using both 20-year and 8-year periods. I then average the results of those runs as my baseline comparison. The intent is to give some additional weight to newer data without completely ignoring older data.

I also run the backtests using a rolling 4-week frequency. I use the rolling tests to reduce starting-month bias, so the results are not overly dependent on one arbitrary rebalance start date.

The optimization itself is performed in batches. For example, if the ranking system has 20 nodes, each initially assigned a weight of 3, the total system weight is 60. I then add +1 to the first node, increasing the total system weight to 61, run the backtests, and save the results. I then reset that node and add +1 to the second node, again producing a candidate system with a total weight of 61. I repeat this for each node, producing 20 separate runs, each with a total weight of 61.

I then compare those 20 candidate systems against the baseline. If one or more candidates improve on the baseline, I select the highest-performing candidate and make that the new baseline weighting. I then repeat the process by adding +1 to each node from the new baseline, producing candidate systems with a total weight of 62. If one or more of those candidates improve the result, I again select the highest-performing candidate as the new baseline, and continue the process from there.

I use the same general process in reverse by subtracting weight from individual nodes. For example, from a baseline with a total weight of 60, I can run 20 separate -1 tests, each with a total weight of 59. This helps identify nodes that may be over-weighted rather than under-weighted.

I typically continue the process until I no longer see improvement for several consecutive iterations. Improvement is not always linear. For example, there may be no improvement at total weights of 75–77, but an improvement may appear at 78, so I generally keep going until I have multiple no-improvement passes.

Finally, I usually run the process from more than one starting point. I will start from equal node weights, but I will also start from the last optimized weighting. The process can be path dependent, so using multiple starting points gives me more confidence that I am not simply accepting the first local improvement path.

It is a long and tedious process, but I have had good results with it. It also gives me more confidence that the ranking system is adapting to changing market conditions rather than remaining frozen around stale factor weights.

I would be interested in hearing whether others are using similar re-optimization methods, whether there are better objective functions to use, or whether there are more efficient ways to approach this in Portfolio123.

1 Like
2

I assign weights to nodes in multiples of 2 (max # of nodes would be 50). I take a ranking system that has worked well in the past and randomly increase 7 nodes by 2 and 3 nodes by 4, then randomly subtract 2 or 4 from nodes until it gets back to 100. I do that 27 times and test it on 5 random universes. Then I take the winners and do it again.

I do the same with nodes weighted by multiples of 2.5, increasing 7 nodes by 2.5 and 3 nodes by 5.

So it's also a long and tedious iterative process, sort of like yours.

The funny thing is that the 2 and the 2.5 end results are usually extremely different. So I use each one along with the the average of both.

3 Likes
3

no wonder Yuval outperforms.

most people complain about overfitting.

nobody talks about underfitting.

1 Like
4

I think almost everyone I know now subsamples or bootstraps rows (time-periods) and subsamples columns (features) for at least some of their models. Maybe not for linear models although some may be bagging and/or aggregating linear models–with different features–too. I did both when I funded linear models.

This applies to machine learning and many methods for finding P123 classic rank weights it seems.

2 Likes
5

Yuval, thanks for sharing your process. I have a question about the testing window you use when doing this kind of ranking-system reoptimization.

When you generate the 27 randomized variants and test them on 5 random universes, what lookback period are you typically using for the tests? Are you still mostly using a 10-year backtest window for this type of optimization, or have you moved toward using both 10-year and 20-year periods?

I ask because in your recent blog post on how far back to backtest, you mentioned the value of looking at both shorter and longer periods. So I’m wondering whether, in practice, you now optimize using a hybrid objective; for example, some combination of 10-year and 20-year results; or whether you still prefer one primary lookback period when selecting the winning ranking-system variants.

Also, when the 10-year and 20-year results disagree, how do you usually handle that? Do you favor the more recent 10-year performance, the full 20-year robustness, or do you average the results?

6

Great question! For simplicity's sake, I look at the ten-year and the 25-year results simultaneously, weighting them so that the 25-year results have 1.6 times the weight as the ten-year results. So it's a weighted average of the two.

When backtesting only non-US stocks, it's different because the pre-mid-2009 data is not terribly reliable: I think there's some survivorship bias prior to then due to the fact that FactSet hadn't added a lot of stocks to their European database yet. So for that I simply average the maximum (about 17 years back to mid-2009) with a ten-year lookback.

These are pretty much seat-of-the-pants decisions. I think any course of action that you suggested in your question would be fine, including using both periods separately and then averaging the resultant ranking systems, or using both ranking systems.

Lastly, if you're backtesting for going short in the US universe instead of going long, I strongly advise a separate backtest on only the period from 3/15/2020 to today. That period was dominated by extremely strong participation of retail investors, and is really quite different from prior periods (or other countries), in my view. Failing to take this into account could be disastrous for shorts.

5 Likes