I think for FactSet there is not much difference between the “All Stocks” and the “All Fundamental” universes.
Ranks determined with the legacy engine using the “All Fundamental” universe will be different when the FactSet engine is used, because the FactSet “All Fundamental” universe is much larger than the old “All Fundamental” universe.
Obviously sims will show different performance (usually lower) with FactSet because they were not designed for it. My guess is that if a sim was designed with the old “All Stocks” universe it will perform similarly under FactSet and Compustat.
“All Stocks” has 9,784 stocks
FactSet “All Fundamental” has 8,245 stocks
Compustat “All Fundamental” has 6,519 stocks
Here’s what to expect. We have changed the code for CashEquiv, which was the source of the problem.
Compustat gives us numbers for CashEquiv for banks that are about twice as high as what they report on the CapitalIQ service. I’m not sure where they’re getting the numbers, but they’re kind of absurd. Banks do not report CashEquiv on their financial statements.
We have come up with a workaround that allows us to report CashEquiv for banks. It’s pretty close to the CapitalIQ numbers.
For other stocks, we’ve come up with a formula that will give far fewer N/As than it does right now. But for some insurance companies and healthcare providers, unfortunately, our CashEquiv is going to be basically cash only.
Here’s the actual formula. If it’s a bank, we use the maximum of cash due from banks and other investments. If it’s not a bank we use the maximum of cash and short-term equivalents and cash only. For FactSet, most financial companies give N/A for cash and short-term equivalents.
Yuval - are we able to see what changes P123 implemented yesterday afternoon, 6.3.20, that would affect simulations utilizing Current>FactSet>Prelim ?
For over 1 week I kept dozens of browser tabs open of the identical simulation and would open new tabs and re-run to see if there were any changes. Yesterday afternoon I noted significant changes in excess of 10% annual. I read that a reported issue regarding MACD was resolved yesterday, but there was brief and unclear discussion around what P123 changed and how it would affect simulations that incorporated technical indicators in buy & sell rules. I specifically use RSI in both buy & sell and would like to know if yesterday’s P123 update would also affect sims utilizing RSI.
This feels like trying to catch grasshoppers from a helicopter.
Joey, the problem was effectively evaluating price-sensitive rules on two different dates (including RSI).
It is as if the rule ‘RSI(9) > 90’ were replaced with ‘RSI(9) > 90 & RSI(9, -1) > 90’, pretending that -1 is permitted.
The fact is that the FactSet engine produces lower returns for ranking systems which uses Greenblatt ranking.
For example using Stock Factor input for VDIGX for which I have holdings data from 2014, and selecting the highest 10 ranked stocks and using universe S&P500, then the annualized return from Jul-2014:
with legacy: 18.19%
Compustat: 18.19%
FactSet: 15.23%
That is a huge difference which can only arise because the Greenblatt ranking system is evaluated differently for FactSet and Compustat.
Aaron - I’m trying to better understand the pro & con of this change. Are you suggesting that a simulation ‘could’ print a buy or sell on day #1, and because of the 2nd pass, and perhaps pricing updates, the simulation ‘could’ then invalidate/remove the signal on day #2 and if I were to observe on day #2 I might see different results than observing on day #1 ?
Further, if the above is true, then it’s fair to assume that on day #2 I might see a signal that occurred on day #1, but wouldn’t have been available to act upon on day #1 because it was caused from the 2nd pass. This would inflate simulation results because in real-time that signal wouldn’t have been available.
I have seen something similar to this with certain indicators on charting software (TradeStation) that people coin “re-painting bars”.
This is challenging for an objective mind to understand. The way I see it is the simulation utilizes Friday’s closing price and applies the RSI indicator to that close and prior supporting closes and if a True condition exists and I have the simulation utilizing Next Open price for transactions then it should record that transaction. It’ll initially use Friday’s close as the transaction date and price and then update to Monday’s open price after the servers update Monday night.
The Greenblatt ranking system needs to exclude financial companies. Greenblatt himself said so. See if the differences are as big if you exclude those.
I thought the point of Portfolio123 was to help us optimize investment performance. Are you now telling us differently? I can’t quite believe I’m reading this.
And as for historical accuracy, Greenblatt didn’t quite say that. He did exclude financials in his Magic Formula, but he’s gone on to describe the formula in the book as the “not trying very hard approach”. He himself has gone beyond his basic formula with the funds he manages.
Joey, the behavior I described was a mistake which has been fixed. It should have only been considering ‘RSI(9) > 90’ for a single observation date, not two observation dates.
FactSet is averaging estimates from fourteen analysts. Compustat is averaging estimates from four analysts. Considering Go Daddy’s recent earnings, Compustat’s estimates seem way off to me.
Yuval,
I receive a “ownership error” when I try to click on your screen, this has happened before on other screens you posted, can someone please look into it
Please, folks, disregard this advice. I’m getting great performance stats from my multi-factor ranking systems. That’s because multi-factor ranking systems tend to be robust. If you use fifty or a hundred factors, each one has little weight, and if one goes wrong, another will compensate.
Here’s a backtest I ran on the ranking systems I use to buy and sell stocks. This is a simple screen backtest that buys the top 30 stocks every week over the last ten years; slippage is 0.25. The first backtest is FactSet Current use prelims with RBICS; the second is Legacy. The ranking systems are very slightly different, but not much.
Hi Marco,
It’s a very simple screen. Another very similar screen had a small drop in performance. The main difference between the two is that the failed screen uses LTGrthMean. In a previous post I cast doubt on the accuracy of LTGrthMean for BHF in Factset. Yuval gave the following reply:
“BHF is a weird one. Both Compustat and FactSet record two analysts who give long-term growth estimates. One of them estimates the long-term growth at 8%. The other one estimates it at 4% for Compustat and 41.29% for FactSet. I suspect the latter is a data error. I expanded the earlier screen to show you how I figured that out. See https://www.portfolio123.com/app/screen/summary/242582?st=0&mt=1”
My fear is that one possible data error is being used to “correct” all of the Factset long-term growth data. Since the other screen drops a bit, perhaps Sales and EPS estimates have also been “corrected”.
Thanks,
David
Could you show the backtest excluding the Prelim or explain why you use the prelim?
I supposed that the best way to avoid any Look-ahed bias (be as “PIT” as possible) was to exclude the prelims.
When the change from Compustat to Factset occurs, what happens to “LIVE Strategies”?
I assume nothing, and they will continue to work but obviously as is being discussed in this thread, future performance may be much different and may warrant updating the rule of even Live Strategies.
