No, I understand what you are talking about, Andreas, but maybe I didn’t communicate clearly enough.
What I’m saying is that if—for whatever reason—P123 chooses a different stock or ETF selected for your model on day ‘N,’ that singular difference of a change at the beginning, with the power of compounding, can result in an enormous difference in performance on day N+10 years or N+20 years.
I was just offering the concept of starting your model on different days as a way to prove this point to yourself. It’s also a way that P123 members can test for robustness. If the previous iteration of your model selected stock X on day N and the new iteration selects stock Y on day N, there can be a huge difference in the results after 10 or 20 years.
Another possibility is that perhaps the PRICE for the same stocks (both are stock ‘X’) are slightly different at the beginning of each iteration. I have seen big differences in performance by testing the three pricing options of Next Close, Next Open, or Next Avg of Hi and Low. If this robustness test of price alternatives results in a significant difference in performance, it indicates your model is not robust.
Again, I’m not suggesting this happened to you, but I’m providing examples that can be tested by other readers who want to check their models for robustness. If there are any changes at the beginning of a Sim, it can result in huge differences in the results at the end.
That said, I again state that I agree 100% with you regarding PIT data. If the data is always moving, even slightly, it is impossible to develop strategies to exploit market inefficiencies. However, in the world that we live in in 2020, this subject is one of the biggest challenges that humanity faces. We are well on our way to a world driven by data run on ubiquitous microprocessors.
For the world society in general, data accuracy is an enormous problem. How to solve for these inaccuracies is not an easy challenge to resolve. I think I agree with Yuval that the data should be fixed when an error is identified. I would prefer to have the most accurate possible data than to have a stable environment that doesn’t fix errors.
That said, I believe that the optimum solution would be for P123 to offer a stable, ‘As-initially reported’ option—and ever-subtly changing ‘refreshed’ data. However, this might be an impossible challenge for P123 because of all the potential iterations of the data that would need to be maintained.
Back to my original post, have you compared the initial transactions of your first version (if you have them) vs. the transactions in the latest version? Is there a difference in the stocks selected or a difference in opening prices? That could be causing your problems. As I pointed out initially, a single change in a stock price could cause enormous differences in performance after spans of years.
I understand your concern with the data - by a factor of infinity. But I doubt there is an easy solution that will satisfy all users.
Chris
