I have a better answer now. Here are the walk-forward backtest results of the last 10 years. So I can say, "Yes, this is completely out of sample." And without any caveats about possible information leakage with k-fold validation:
Keeping with the main point of this thread, I just told Claude 3 that instead of a hold-out test set I wanted to do this as a walk-forward backtest.
Discussion:
This is like a screen at the close with weekly rebalance and no transaction costs.
There is some drop-off in the models performance over the last 10 years. People can make their own judgement as to whether it is still okay with out-of-sample results.
But I stipulate ahead of time that I would benefit from hiring a full-time feature engineer for my little family office. And that others might do better for a variety of reasons.
Jim