If you believe that your ranking system should reflect the most recent developments in the market, then it would make sense to run your backtests covering a relatively recent period, say just the last five years. If you believe that some factors revert to the mean and others stick around, then it would make sense to run your backtests covering the entire seventeen years of data available at P123, or perhaps even exclude the last five years.
I did a little experiment. Using 30 factors that I know work in general, I developed a ranking system optimized for just the period 4/2006 to 4/2011 and another one optimized for the period 1/1999 to 4/2011. They were quite similar in many ways—each contained 16 or 17 factors, and the most heavily weighted one was growth in operating income; they also both used the same universe; but some of the factors were quite different between the two systems. I then tested both systems to see how well they functioned since 5/11. Both performed about half as well they had before that time, but the one optimized for just the prior five-year period outperformed the one optimized for the more-than-twelve-year period by almost 5%.
I was wondering if anyone else has performed similar backtests to ascertain whether it’s better to backtest just a few years in the recent past or whether it’s better to backtest over a longer time. I’ve noticed a huge difference in the efficacy of certain factors over the course of the last seventeen years. And a ranking system optimized for the last five years will look very different from one optimized for the last seventeen. What I worry about is that a factor that worked very well in, say, the 1999 to 2008 period and not very well since will, because of its efficacy in the earlier period, be a major part of a system that is optimized over the long run, and will simply fail to work in the near future since it hasn’t worked in the near past. After all, the market today is in many ways more similar to the market of 2013 than it is to the market of 2001, considering the rise in ETFs (especially leveraged and inverse ones), the number of investors using quantitative approaches rather than relying on analysts and brokers, the ready availability of market data, the ease of making quick and cheap transactions, the increase in volatility, the shorter-term investment horizon, and so on. Perhaps most significantly, I think market behavior has fundamentally changed because of whatever lessons folks have taken away from the dot-com and housing bubbles.