When running a model with multiple folds how are the sub models weighted for the testing-training and testing-holdout runs? Similarly, when creating a final predictor are models estimated in the validation fold process considered?
Thank you
1 Like
The multiple folds are not weighted in the result calculations. There is no need for that because the holdout periods do not overlap. For example in your screenshot it shows fold 1 had an out of sample period of approximately 2009-2012, fold 2 was 2012-late 2014, etc. So the returns for any single date are from a single fold.
When you create a final predictor it does not use anything related to the periods from the folds. The folds are a way to run multiple out of sample tests while replicating what you plan to do when you go live with the strategy. When I say "what you plan to do", I mean if my plan is that the predictor for my live strategy will be trained using all available data then I would use the Time Series CV or KFold validation methods in my testing. If my live strategy will be trained using the latest rolling n years of data then use Rolling Time Series CV.
The only things being reused by the Predictor from your work up to that point are the dataset, feature list and the algorithm. You specify the period to use for training the Predictor. Usually the period you select would be based on your choice of Validation method used in your earlier testing as I mentioned above, but that is not required.