Georg,
My apologies for a short—and somewhat frustrated–answer previously. I do not seem to be very good at writing about some of this. If anyone is truly interested then people smarter (and better at writing) have written chapters about this, with editors. Better than a few posts from me if there is true interest.
Indeed, de Prado solidified this for me and could probably clarify any questions. He is a principle for a company that invests more than $217 Billion (AQR Capital Management). If you want wisdom from an immortal (a rich and well-trained one) that is truly where you should go (for $15.09 at Amazon). But that should not stop me from trying to give my best answer to a good question.
IID, as I posted before, means independent and identically distributed. The lack of independence gives a preference for how we cross-validate.
An example is best. Let us suppose you wanted to see how a regression strategy (or optimized P123 rank) works and you wanted to TEST the strategy from 2009 to 2010.
I was addressing one simple question. Should you TRAIN the system from 2004 to 2008 or should you TRAIN it from 2011 to 2015?
So depends on what you are looking for perhaps. But if you are interested in how well the system is likely to do LOOKING FORWARD then you should LOOK FORWARD. The training data should be before the test data. You should train from 2004 to 2008 and test using data from 2009 to 2010, for this example.
If you use the data from 2004 to 2008 you are training before the recession which may have changed the market. Maybe interest rates and a bunch of other things changed too. But it is a fair test. Because the regression (rank) was not trained on that information that was not available before the test period. This makes it a fair test of the system.
If you TRAIN with 2011 to 2015 you ARE training on a different market. You are training on data (and information) that you could not have possessed, at any price, in the 2009 to 2010 time period.
It could be that when you test after the recession you will do well but only because you had a set of data (and information) about the market after the recession to train with—which would have been impossible for you in 2009.
Using data that is not PIT is just an example of using INFORMATION that would not have been available at the time. Here you are using information and data to train your system. Data you could not have gotten through the SP500 (or anywhere for any price). It is also A LOT LIKE look-ahead bias.
Anyway, just wanted to give the best answer that I could. And, admittedly, I probably did not do as well as a principle in a 217 billion dollar investment firm (de Prado is a principle for AQR Capital Management). Someone who had several chapters to develop the idea. Also he probably did not have any errors in his math—which I may have done.
This mortal thanks you for your question. De Prado—who, obviously, communicates with the immortals—is a better source if you have a true interest.
And you could just use data that is forward in time if you want to see how your model is likely to perform going forward. This is common sense that a few mortals do posses.
Sorry if that still does not answer your question. But I will stop before I copy and paste an entire chapter and the references;-) You can do that on your own if you are interested.
I appreciate the question.
-Jim