My only point is these debates are great fun but they should not stand in the way of progress at P123. We should not wait until we get some sort of consensus or some number of likes for a post before newer ideas can be implemented. Adoption of new (or somewhat new ideas in many cases) should be sped up not slowed down!!!
As an example, cross-validation was discussed at length and discounted by nearly everyone at P123 until it wasn't. That is until it became a cornerstone for P123’s AI/ML. Not sure how else to come at that one. And I don’t want to beat a dead horse.
Its almost as if we need an authority figure to tell use how to think. Marc Gersteing says equal weighting is quite the bom and we hear nothing else in the forum. Marco accepts cross-validation and quickly only a few have lingering doubts.
Here is an example of Marc telling us that adding noise to our backtest is a good idea. In truth this is quite insightful. Adding noise in a controlled manner can be a regularization method and he is also discussing permutation importances here. Brilliant!!! Truly remarkable: WTF!!! We're doomed. - #8 by mgerstein
But it took is over 10 years to develop some more formal, controllable and widely accepted regularization methods (elastic-net regression being just one example). And we still have not put permutation importances into action at P123 despite Marc’s endorsement in that thread.
Few, at the time, wondered if we could see if there were already published papers on the topic and maybe considering implementing what had already been done by others.
Hmmm.. Here is an idea from the 1990s we could could look into:
.One widely published method made as a feature request quite a while ago. It got 6 likes including one by Hemmerling who tends to be skeptical about ML methods: Add Support for LightGBM’s XENDCG Objective (Better Ranking) . Did anyone at P123 read it and consider it?
Don’t get me wrong. I have downloaded data and tried it. It is not a strong priority for me, but it is one clear answer to your question as to one way it is being done out in the real world.
I honestly think P123 is a bit ahead of the curve now and I don’t want to be an alarmist. Progress is being made. Adding LLMs will be like as steam-catipult on an aircraft carrier–keeping us ahead for awhile. But things change pretty fast these days.
One way to keep ahead is by opening up the platform. Again @marco is working on some good things to keep us ahead–including allowing members to contribute through Voila etc.