Great thread everyone. As an aside Google NotebooksLM also creates slide decks which seem to have more information than the infographics. The paid version, at least, will also create a nice video. The flash cards are in depth and one can learn from these.
So here is an example of what AlgoMan means using the Titanic dataset from Kaggle. I could not find the P123 dataset for insider information.
Maybe @AlgoMan and others (advanced ML members too numerous to mention now) could have a go at this and other challenges. Why should Kaggle have all the fun?
I also want to mention a related ML/LLM tree method mentioned previously by @pitmaster that may be less prone to overfitting. As everyone probably already knows the reason we have random forests with many aggregated decision trees is because single decision trees tend to overfit. Pitmaster addresses that by using more than one tree and also having the LLM put the aggregate tree results into words. Content by the Judge on X - #10 by pitmaster
So ideally we could construct short trees that are great for interactions and create an AI factor with multiple short AI factors in the ranking system.
At present XGBoost and other models allow you to specify interactions with arrays. This is very clunky as it turns off interactions (lots to turn off). An LLM could speed that up possibly. But I think we may not have access to that in P123’s AI at the present time in any case.
The other problem is that a random forest will find thousand of interactions. Many of them will be spurious–finding interactions in past market regimes that no longer exist. With so many interactions there will be many false interactions.
A solution might be focused interactions as P123 has already done here. Great idea @marco that could be expanded using @AlgoMan’s suggestion!!!
Here is a Sklearn generated tree telling you who was likely to survive on the Titanic (classification decision tree):
