Having run a lot of traditional ranking systems over the last few years I've observed a lot of changes in backtested screen holdings, despite making no screen adjustments. This can also be reflected in the fact that the backtested CAGR for any given fixed set of dates tends to change slightly every day. For example, I might buy everything the screen tells me to this week, only to run a backtest next week, check the holdings for 1 week ago and see a different holding or two to what I actually bought.
From a previous post, it seems like using WeeksIntoQ/WeeksToQ in screens/ranking systems might be responsible, with rankings being retroactively adjusted when earnings are processed late (e.g. a company reports last Monday, Factset only processes it a week later, which then retroactively changes the rankings/screen holdings on a backtest.
This is just speculation of course. I can't identify what exactly causes these issues. It's also hard to say how much they may be overstating results. That said, now that the AI factors are here, it seems like they could be particularly good at finding and exploiting any PIT vulnerabilities in the features.
While WeeksToQ/WeeksIntoQ might be an important feature for the models, would they be best to avoid given possible PIT issues? I'm not sure if BarsSince(LatestFilingDate) could be problematic as well? Are there any other features that may pose issues? Any feedback would be appreciated.