How about we look at models that we can access all details about, that delivered poor live results despite good backtests?
Then we can comment on how we could have spotted that in advance.
That might produce practical additional tests and approaches we can use, and we can see them live with our own eyes on real systems.
Any suggestions?
How are Designer AI models handled? There isn’t really a “sim” for AI models is there? If so, how is that generated? It would be interesting to see the correlation and decay for the validation or test-set vs live results for AI models (private or Designer).
I recently thought I could get away with using backtesting to create a linear model with low degrees of freedom and by using other safeguards to prevent overfitting. But the model was massively overfit. It may be just me, but I ended up having to use a validation method.