One method mentioned by several people in the forum is bootstrapping. One could spend some time comparing and contrasting bootstrapping to rolling tests. They have some similarities (and differences) in my mind.
At P123—without the API or DataMiner—people use a similar method called subsampling in the literature. I don’t think it has a formal name at P123. Subsampleing has been found to be very similar to bootstrapping IF THE SUBSAMPLE IS NOT TOO SMALL. Most of the time you want the subsample be be at least 50% of the universe (according to any peer-reviews literature or implementations I have seen in Python anyway). This takes the form of using MOD() at P123 for the most part. It has been advocated by many people in the forum (including P123 staff I believe).
Hmmm…a Mod() subsample is usually less than 50% of the universe. Does that matter? Yes, No question about it.
A simple random seed for random() would literally add nearly infinite additional subsamples (with greater than or equal to 50% of the universe if desired) in addition the great idea of using MOD() universes. Mod() is a great idea, if EXTREMELY limited.
Also, real bootstrapping (not just subsampling) can be done with Python—without bothering P123 with a feature request (with or without a random seed BTW).
Yuval frequently advocates Bayesian methods. Putting it simply, I agree that this can be helpful sometimes.
Specifically Yuval has referenced this paper frequently: Is There a Replication Crisis in Finance?.
In this thread for example: 0 Vote Why such a large discrepancy between backtesting and real-life results?
Same question keeps coming up. Similar answers each time as there a finite number of reasonalbe answers.
Pretty much all of them available in Python. Most of those Python programs have a random seed option. Random seed would go a long way toward making some of those algorithms doable in a spreadsheet (or within the P123 platform) if that is desirable at all.
BTW, while I think random seed might give reasonable “bang for the buck” for anyone (requiring Python now), I love P123 just the way it is. I am happy with my ports just the way they are, they seem to be making me money based on my present out-of-sample data and P123 does many incredibly cool things!!! I.e., not a feature requests. A suggestion? Yeah, why not?