Thank you, Yuval for your contribution. I definitely had some of what you are writing in mind when I started this topic. Note also my mention of “you now want to rank those ‘direct’ factors (so excluding marketcap and accrual type factors that have more of a augmenting nature)”, which was pretty much a direct reference to your classification of direct factors vs augmenting factors (as mentioned here: The 3 Types Of Factors | Seeking Alpha.
I wasn’t planning on sharing this, but as this discussion is going a bit further now, I will do so anyway.
What I’m working on currently (and what I think is related to your reply) is answering the question of: “Should this factor be added to my multi factor ranking systems?”.
The process in my mind goes as follows:
(1) Gather a whole bunch of factors that I think make financial sense. Of all those factors (including the augmenting ones) run single factors ranking tests and test them all based on one or more metrics (the topic of this discussion). In this step, the augmenting ones will of course not have a good chance to rank high, so let’s keep those in mind. Next, (2) rank them all and (3) create a correlation matrix of all factors using the historical time series data similar to what you described here: non correlated factors - #2 by yuvaltaylor (again, maybe taking an other metric into account as opposed to just the slope to determine the correlations) . After that (4) use a clustering algorithm (without going too much in depth, I’m currently looking at something simple and basic like this: Create clusters using correlation matrix in Python - Stack Overflow) to make x different clusters and pick the y top factors from each ‘correlation cluster’. You now have x*y amount of factors in your ranking system. The correlation makes sure that you, let’s say, at least somewhat take into account the dynamics of the forrest as a whole instead of staring blindly at the biggest trees in the forrest the whole day.
(5) After that - and this brings me to your input - add back, one by one, any other factors that you started with and made financial sense to you (these are the ‘least effective’ direct factors, some potentially augmenting factors like marketcap and accruels and of course, some factors that maybe seemed to make financial sense to you but don’t show a lot of promise in practice. The augmenting factors will pop up - or at least that is my idea - and will augment the other factors such that the ranking system as a whole will score higher on the metric as described under step (2).
With such a system in place (or a similar but better one), whenever I find a new factors, I just add the new factors to the starting step (1) and follow the other steps to see if it adds value. Of course assuming the testing takes place with multiple randomly generated universes and over different time periods.
As I have some ocd-nerd qualities myself as well, I will most likely do this using the portfolio123 API.
Apart from your process of determining metrics to ‘judge’ a single factor ranking system, I would be very interested to hear your thoughts on researching the changing dynamics of a ranking system as a whole. I was hoping that the results of the process I described above would give me at least some insights into that as well, and won’t be just a big framework of optimisation and data-mining, which is an easy trap to fall into.