Live vs Simulated - AI Factors Rank

AI Factor Discussions
1

Hi everyone,

I am surprised to see that I am getting different rank values for the same AI factors when comparing live portfolios vs. backtests over the same period.

My AI factors were trained up to the end of 2024, and the model is not being retrained. However, when I compare the live portfolios from May/June with the backtest covering the exact same period, I notice significant differences in results, which seem to stem from discrepancies in stock rank values (and not just rounding!).

How do you reconcile this? I was not expecting any difference, given the model is static.

This makes me question the reliability of the backtest, as it no longer seems to reflect what actually happened live...

Thanks in advance.

2

To provide more detail, please find attached the Live and Backtest transaction reports.

You will notice the difference in the Rank values between the two:

Live
Live811×728 45.6 KB

Backtest
Backtest802×750 46.8 KB

3

I’d like to highlight a discrepancy observed during yesterday’s rebalancing, using VNCE as a case study.:

ticker VNCE

In the Live Strategy, the rank for VNCE at the time of rebalancing (and even now from the Holdings tab) is 98.1 for 07/05/2025:

VNCE1
VNCE1855×152 7.76 KB

However, when I check the same stock in a Backtest as well as in a Screen as of 07/05/2025, the rank is 97.1:

Screen: VNCE_Screen
Backtest: VNCE_BT

Could you kindly investigate why this inconsistency is occurring?

I would expect consistent rank values across Live, Backtest, and Screen when referencing the same date. These differences raise concerns about the reliability and consistency of backtesting results.

Thanks in advance.

4

We have not made any changes to the engine lately. Therefore there's only one explanation for ranks changing: the underlying data changed. Also the fact that ranks are relative, a change in one security can affect the rank of a different security.

There are two main causes for data changing:

  • Using preliminary data
  • Vendor fixing errors, backfilling or revising

We're preparing a KB article detailing the above. In short, FactSet is not a true point in time database (if such thing even exists), but it's good enough to develop strategies. The reason is because a backtest is nothing more that one possible outcome of many of your ideas. That's the reality no matter which data vendor you have.

Also worth noting: most academic papers lag the data a certain fixed amount after the filing date. Something like 6 months for annual reports. And thousands of papers are written this way. All these papers are nothing more than an approximation of what real results would have been for a system.

Hope this helps

3 Likes