IMPORTANT: Fix for StdDev of split-sensitive estimates has been released

Latest Release
1

Dear All,

The fix is for the StdDev of split-sensitive consensus estimates is now live. It affects estimates of EPS and Price Target. The problem was uncovered in this thread by @philjoe

In short, the StdDev of the estimates for EPS and PriceTarget, when evaluated in the past, did not have the splits reversed out. This means that, for example, the StdDev of the EPS estimates in the past was lower by the factor of all future splits. For the present it was always presenting the correct value.

There's no sugar coating this. If you developed a strategy using this metric, please retest your system. This has a potential to make simulations much better than what they really were since stocks that split do so because the price increases.

We totally missed this in the tests. This bug has been lurking since we switched to Factset.

We're very sorry about this.

Regards.

NOTE: we also released a new function ConsEstRSD() that returns the relative SD. Relative SD is the normal use for SD of estimates case since it normalizes the SD to a percentage.

4 Likes
2

You can recreate the problem before this fix using FutureSplitFactor. For example if you were ranking using this formula

CurFYEPSStdDev / CurFYEPSMean

Replace it with

(CurFYEPSStdDev / FutureSplitFactor) / CurFYEPSMean

NOTE: FutureSplitFactor is something that should never be used in developing a system. It's just there for debugging.

3

Thank you for the fast fix Marco. It sucks as I thought a combo of these was pretty much the holy grail. But better to find out now than to have it fail in live trading. And I found a combo factors that work well to replace my old system.

Onward and forward. Thank you for addressing this quickly when you discovered it.

1 Like
4

If I'm understanding this, EPSSUE would have been affected, too. Right?

https://www.portfolio123.com/doc/doc_detail.jsp?factor=EPSSUE

5

SUE is ok. Internally it's a ratio of fully split-adjusted items, so the split factor cancels out.

SUE = (Actual - Est) / SD

The problem is when you combine factors externally, like in a formula, where everything should be point-in-time with regards to splits and dividends.

2 Likes
6

I did backtested 5 effected (micro/small cap) strategies before the update, and today after the update, 4 out of 5 is backtesting higher returns after the update.

7

It shouldn't improve your backtest, as how it was working before there was look-ahead bias or future leak or whatever you'd like to call it.

2 Likes
8

I've learned to assume nothing, and test everything.

But #1 sanity check will always be: if results are too good to be true, it's probably because of curve-fitting or future-bias.

And, as a side note, AI Factor could be used to spot problems in your Ranking System factors and formulas. Just follow these steps:

  1. Create a new AI Factor
  2. Import the features from your ranking system
  3. Load the dataset
  4. Train a predictor that supports Feature Importance (ExtraTrees, RandomForest, LightGBM, XGBoost, and Linear)
  5. Look for Importance outliers. These may have a problem
1 Like