I haven't become entirely comfortable using AI factors in live training yet. I've tested a number of different features and systems; sometimes they yield very different results without me being able to fully understand why, and other times I get the impression that some of it is simply timing luck.
But has anyone tried merging different AI factor runs together, for example:
Take three AI factor runs, extract the 50 best features based on prediction importance.
Will this work, or is there something I don't quite understand about how feature importance works and what would happen if you just took, for instance, the 50 best from each run and then combined these features in a new, consolidated AI factor run?
Yes, I have been developing a system like you describe;
Instead of developing a set of features for each model I have separated features that work on trees from those that only work on linear regressions. I run all the models using either the tree or linear feature sets and rank their results. Rank the overall weighted results into one ensemble score. Tree models have slightly better results; Linear models tend to lower the turnover in the ensemble. This is a longer-term project, and I too have not put any money into it yet. I am trying to improve the risk return ratio for lower volatility.
High importance does not directly translate to better features. It just means that the features were more present in the tree structures.
If you have a very noisy feature like a fast shifting macro feature, it will get high importance but might be directly damaging to your training or overfitting to noise in-sample.
Feature A might only have high importance because it interacted well with Feature B in Run 1. Stripping Feature A out and putting it into a new model without Feature B breaks that interaction.
High feature importance often goes to redundant or highly correlated features that split the signal. So doing what you say you do might throw you in the Multicollinearity pitfall and generate an unstable model with high turnover. The risk of overfitting is overwhelming as well if you reuse the same training data when chasing importance.
When stripping out low importance features with no reasoning why, you risk loosing highly valuable features. Example, R&D features does not mean anything for most companies, but for a few it is very important and might be the tie breaker if the company should be ranked high. This feature will show low importance.
Yeah, grabbing the top features by importance from a few different runs can be a decent starting point for finding ideas you might have missed. But it’s really just a noisy shortlist — not a finished strategy.
You still need to clean it up: knock out the highly correlated ones, check which features stay important across different periods or seeds, and properly test the whole thing out-of-sample.
If you just dump the combined top-50s into a new model without those steps, you’re more likely to end up with extra noise, multicollinearity, higher turnover, and overfitting than with something actually more robust.