Yes, the plan is to get all FactSet data loaded onto the beta site. But it’s a lot of work and it’s taking some time. And we don’t know yet whether or not we’ll be able to expose short interest data. It likely won’t be from FactSet, but we’re looking into other data providers for it.
Marco/Yuval - If I use Exclude Prelim from FactSet data how much is this a problem compared to Compustat data? As of right now my sims are actually doing better if using Exclude Prelim option with FactSet. There are probably two reasons why this might be the case. Either there is a strong look-ahead bias in my sims and/or the FactSet data is actually better data.
I was actually expecting somewhat improved performance in the sims when I heard we were switching to FactSet giving the data is not point in time anymore. I am just wondering how much is this real and how is it not.
OpIncQ, OpInc(1,QTR), OpIncTTM, CashEquivQ, EBITTTM are not coming through on factset for smaller? financials?
e.g. NWLI,LNC
But they are coming through for SCHW.
Please keep us posted here as the bugs are fixed. I just tried to run a diffetent sim on the beta site, and got a different error on this one: UnhandledExceptionHandler: EXCEPTION_ACCESS_VIOLATION
No apparent execution problems so far for me with stock screens and ETF sims on the beta site, but consistently get “Invalid uid in PriceHistory” error re-running existing stock sims. Made a new, simple stock sim and it worked.
When I duplicated a non-beta stock sim’s settings and rules without using the sim copy method, the error “In Rule 2: ReqContext: UserUid not initialized” happened. There was no Rule 2 by name. The error disappeared when custom series used in a buy and sell rule were updated and saved on the beta site.
I see some very significant differences in ranking performance results between Compustat and FactSet but haven’t delved deeply into that concern yet because of sim failures.
We are working on fixing all the bugs. There are lots of them, so please be patient.
Once the bugs are fixed there will still be some minor differences between the regular site and the beta site with Compustat and “Use Prelim” checked. We realized when setting up the beta site that we had one flag for all preliminary data. We should have had two: one for interim data and one for annual data. So an annual may have been incorrectly flagged as preliminary during the first three quarters and erroneously fallen back. That has been fixed on the beta site and has not yet been fixed on the regular site. It was important for us to see how the beta site was working before applying the fix to the regular site.
As you can see, this is a work in progress. So please be patient with us.
Thank you very much for your update. Let me make sure I understand what you are saying.
Is there a difference in meaning between preliminary data and interim data?
Does p123’s error mean that preliminary annual data was backdated in p123 so that simulations and backtests believed this data was available up to one year before it really was?
Are these backdating errors limited to annual CompuStat data only or do they affect such p123 functions as “ttm” and/or other non-CompuStat data such as estimates, short interest or institutional ownership?
If it was up to me, I’d focus first on exposing all the data before trying to extinguish the bugs. That’s how it was done last time during the switch from Reuters and it was an amazing job.
Yes. Interim data is quarterly data. Quarters are interim periods between annual data releases. Preliminary data is released by the company prior to the SEC filing.
No. When a statement is flagged as preliminary, some NAs will be filled by going back to the prior period. Since the annual was flagged incorrectly as prelim, NA is the correct value, and falling back could get you a false positive.
These errors are limited to Compustat fundamentals only and do not affect estimates, short interest, institutional ownership, etc. Those are all completely different datasets.
By the way, a few of the bugs have now been fixed. Not all of them, but issues relating to preliminaries and invalid benchmark notices should be OK now.
Possible to share some kind of bugtracking with the community in order to improve the bugreporting in this thread? E.g. by editing the first post with something like this:
#001: The quick brown fox jumps over the lazy dog.
STATUS: Closed #002: The five boxing wizards jump quickly.
STATUS: Open
etc.
It sounds as if interim, meaning quarterly, preliminary data has always been handled correctly. Is that so?
If quarterly preliminary data has always been handled correctly, then “ttm” data points should be unaffected too. Is that also so?
I wonder if you could provide some concrete examples, meaning before the correction and after, of how annual data points may have changed?
I ask because I’m not hearing enough of a cataclysmic difference regarding what p123 has corrected to explain the huge differences in simulation results I am seeing.
Therefore, either…
I am misunderstanding, and underestimating, just how significant this correction to the annual data must be.
There are other changes to or bugs affecting the beta site w/preliminary data that need attention as well.
Unfortunately, I don’t believe it’s possible to evaluate the transition to the FactSet data without fully understanding whatever issues are affecting the CompuStat w/preliminary data – old or new.
In addition, and in part subject to whether “ttm” data is affected, I’m not sure I even use that many annual data points in my screens and models. If I (and hopefully other users) can understand exactly which data points are at issue, it will be much easier to correct, discard or recalibrate our existing models.
At that point we can turn our attention to evaluating the FactSet data with more certainty - and while there is still time.
I understand p123 is striving to get things right. I appreciate your help with this.
To reiterate what Yuval said, the previous version of the beta server (before 2020-04-20 6PM CDT) was rife with mistakes relating to handling preliminary fallbacks. The latest beta version will more closely match current behavior in production for handling preliminary data, but it will not match exactly since intentional changes are still present. It is recommended that you run backtests again that were wildly different before the fixes.
To give a sense of the magnitude if the issues, here is a list of the affected factors and functions I could find. These issues also impacted any industry factors derived from them.
Now that some of the bugs have been fixed, please report specific differences you’re seeing between the regular site and the beta Compustat Use Prelims site. Try to isolate exactly what the differences are. For example, if you’re running a simulation on both sites, look at the transactions record; if you’re doing a screen, use the ShowVar and Ticker commands to see what the differences might be due to; if you have differences in rank performance, try to figure out which node is causing it. All of these will help us sort out any remaining bugs in the site. After all the bugs have been fixed, the beta site will not be identical to the regular site because of the difference in how preliminaries are handled, but the results should not vary too widely.
Also see Aaron’s comment above and the list of bugs that have not yet been resolved on the Trello board I posted.
A few of my sims work now, and all of them have been consistent to have BETTER results (higher Sharpe ratio, better performance) with FactSet + Exclude Prelim than with Compustat (with or without Prelim).
However, many sims are still unable to run, all of them with the same error: Invalid uid in PriceHistory
Clearly these revisions involve a LOT more than just annual datapoint corrections.
Running the sims post 2020-04-20 at 6PM CDT made virtually no difference to the results I am seeing.
Have all of the intentional changes been posted on the bug-tracking site?
Sim results are apparently unchanged on the old site.
The consequences of what you are saying are horrifying. It appears that every serious fundamentals-based formula, ranking system, screen, simulation and/or designer model that relied upon CompuStat data populating p123 before yesterday evening was constructed using a significant amount of data unavailable at the time the data was purported to have been available.