I have a sim that I last ran 6 months ago. I wanted to re-run to see recent performance so I made a copy and changed the Period to Max. Both sims had the same start date and universe (S&P 1500), yet the results, almost immediately, were quite different and I am trying to understand how this can be. Obviously nothing changed as far as the rules are concerned which seems to imply the data has changed but how can that be??
This is a short term, relatively high frequency, traditional (no AI) model. Here is a chart of the original:
Impossible to tell with these data. Just be aware that very few positions, very volatile stocks, portfolio simulation (instead of something like a rolling screen) => extreme sensitivity to rankings.
Interesting. I can see how this could explain the differences but I don’t understand how this could happen. In my mind, the past data is set in stone, so any calculations, entry/exit rules, etc, should produce identical histories every time a sim runs. Perhaps a small change to the data made by Factset could ripple through a sim, resulting in wildly different results? If so, then this would seem to raise some questions about robustness…..
Maybe I should stop looking at results with quantiles set at 100 and instead set them to 10. I don’t know, kind of lost on this. I have noticed that not many of the public sims achieve returns close to the returns realized prior to the sim going public. Perhaps they are suffering from the same issue. I think most of us are going to gravitate towards setting at 50 or 100 as the results look sooo much better than set to anything lower.
I should know this but don’t: does P123 use the actual price data and credits a sim with the dividend, rather than back-adjusting the price data so that the dividend is embedded in the price? (back-adjusting will cause the entire price series to be re-computed and any ratios, formulas, etc, you might use that use some combination of, for example, sales and price, could have important before/after back-adjusting created differences)
There are tiny updates to past data + tiny rounding errors. You should not expect past ranks to change much.
Fortunately tiny changes to the training data should not lead to large changes in the trained model, but (and this is the critical point): tiny changes in past data can give large changes in a simulation!
Imagine you running a sim with a sell rule: rank<95. You run the sim and get a good result. Then the database gets some tiny fixes, perturbing the ranks in a small way. Some of the stocks in your simulation might not be sold at different points in time: a stock that was sold in week N with a rank of 94.95 might now end up being held a couple of weeks more, which means that your sim will not buy the same stock at this time since there were no free positions. These changes will propagate for the remainder of the history of the sim, changing (some times dramatically) when and what you buy.
If your sim has only a small set of stocks, and if they are very volatile, and especially if your sim is even mildly overfit (=outperformance coming from a small set of the stocks with extreme returns having to be bought at the right moment in time), then tiny changes in the historical data will give large changes in the simulated result! Meaning that basing a model only on a single simulation will not give a robust out-of-sample model.
I have bad experiences with trying to optimize models using only rank performance (but I have no idea why), rolling screeners or sets of simulations have worked great.
Is it possible you changed the ranking system? The differences are definitely caused by ranks being very different. Here are the first transactions by the original system
The changes that have happened since 9/7/25 do not go back to 2004. The transition to FactSet mktcap for example only affects data starting around 2016.
The only thing left is then a change in the ranking system. Unfortunately we do not track revisions (it's a project in the to-do list). But we do record the last time it was updated, and for your ranking system it was updated on 10/17/2025, so about a month after your original simulation. Is it possible that you made a substantial change to the ranking system after you ran your original simulation?
It is possible but I try to be careful about this and put all components of Live sims in folders named Production. Anything in a Production folder is supposed to be hands off so as to avoid this potential problem. The ranking system used is basically Core Combo minus 2 nodes, plus 2 nodes I added. Maybe I changed a weight…. really I can’t say for sure at this point. Sounds like I probably screwed it up somehow. Thanks
