Great discussion, many interesting ideas for further research. I have not yet even started to develop fully-fledged ML models for real money making.
There are couple reasons to be hesitant:
. ML models will be hard to interpret. We like to use asc/desc for each factor. This powerful weapon based on many years of research will be gone.
. ML models are mostly probabilistic incorporating randomness. Thus, the same set of features and initial settings will lead to different outputs.
. ML models will find a suboptimal solution - this is similar case to p123 optimisation. However with optimisation you can find global optimum (using brute force) be restricting range of weights.
On the other hand, ML models and more general ML framework provides some promising tools :
. modelling non-linear relations between feature(s) and a target → ability to uncover hidden patterns
. exploiting time-varying nature of financial markets. These complex, non-linear patterns may be profitable only for a short time. Decision needs to be made how much history use to train and when to re-train the model. These decisions can be conditioned on a separate model.
. using ensemble methods to reduce variance and potentially live results - one of the method is bagging (averaging) when one model is run multiple times to average suboptimal solutions to lower variance. Random Forrest is an example of ensembling decision trees. However this may be not enough. Why not to ensemble models with different training periods, different settings, different models, etc. There are multiple possibilities but there is one general rule: A perfect ensemble should contain highly accurate and diverse base-models which errors, and in result forecasts, are uncorrelated.