Marco: “To be clear: an ensemble inside an AI Factor would train each model of the ensemble at each fold, then use the average predictions to generate the results. Is that what you want? . . .The downside of the ranking system ensemble, is that it will be pretty slow to run, and somewhat complicated to setup: you have to set up.”
I was envisioning a much simpler approach. Your current system runs through all the folds and ranks the models performance across all the folds. At this point during the testing option one could have the option of doing an ensemble performance of the top few of the models that have performed the best during the validation folds. A simple summation of the top ranks from each of the models would give a consensus (ensemble) rank. I’ve only sampled this method with one simulation, but the performance was better than the individual results.