FactSet beta site v1.0, NOW LIVE

[yuvaltaylor]

Happy to start some tests. My first 2 questions:
1/ SI%float is still available when choosing Factset provider. Where does it come from? Have you found a solution or is it still Compustat data that will disappear in the transition?
2/ My CEF sims don’t change at all when switching to Factset. Do CEF fundamental data really come from Factset? (Yield, NAVDisc, NAVDiscAvg(), NAV%Chg12M etc…)

SI%Float is not a fundamentals item so we’re still getting Compustat numbers for that. Ditto with the CEF data. The only things on the website that are different between Compustat and FactSet are the fundamental line items and things derived from those.

Thanks for your answer Yuval.
Maybe you don’t see a stock’s SI%float and a CEF’s NAV growth or discount as fundamentals, but they are not technical either. Is there a plan to get data about short interest and CEFs before Q3 or must we already throw away models in which they are used? The sooner we know it the better if we must go out at best of such trading strategies. Your migration and your communication should be driven by use cases and user’s risk, not database constraints.

Yes, the plan is to get all FactSet data loaded onto the beta site. But it’s a lot of work and it’s taking some time. And we don’t know yet whether or not we’ll be able to expose short interest data. It likely won’t be from FactSet, but we’re looking into other data providers for it.

[EbenezerScrooge]

Any sign of these line items in the factset datastream?

SpcItems
ExpNonOp

Could you publish a complete list of line items from compustat and factset and how you are mapping p123 functions to each?

The corollary being, which p123 functions are we currently using that will map to nothing (no data for an line item) in the FactSet data?

If factset is missing a line item is the beta site mapping to the compustat line item? Don’t do that (re: SI%Float @Yuval above), so we can see what we will have when compustat goes away.

FactSet gives a lot more NAs for SpcItems than Compustat does. I suggest adding IsNA to all your SpcItems rules. That’s what I’m doing. Instead of SpcItemsTTM, use IsNA(SpcItemsTTM,0).

It looks like ExpNonOp is missing. We’ll fix that. That’s a bug.

We’ll be making a spreadsheet available so that users can see the Compustat to FactSet mapping.

The goal is to have NO current P123 functions that will map to nothing in the FactSet data. Whether that goal is achievable or not remains to be seen.

All fundamental line items are FactSet if you use FactSet on the beta site. SI%Float is not a fundamental line item, so the beta site defaults to Compustat, like it does with all non-fundamental line items. As additional FactSet data gets incorporated into the beta site, that will change.

You da’ bomb Yuval. Keep up the good work.
Scrooge

[atw]

Could you say more, please, about the errors you have discovered on the existing site?

A number of my simulations are showing staggeringly different results on the existing site vs. ComputStat with preliminary data. It’s very concerning.

Thank you.

Hugh

[enisbe]

Luckily Factset does give us two key dates for past filings: when the preliminary data was first announced (“News Date”) and when it was filed with the SEC (“File Date”). When you choose “Use Prelim” we expose the final data on the News Date + one day. This means that some data points may suffer from look-ahead biases. Depending on your strategy, this may not matter much since many important items are usually always reported in press releases.

Marco/Yuval - If I use Exclude Prelim from FactSet data how much is this a problem compared to Compustat data? As of right now my sims are actually doing better if using Exclude Prelim option with FactSet. There are probably two reasons why this might be the case. Either there is a strong look-ahead bias in my sims and/or the FactSet data is actually better data.

I was actually expecting somewhat improved performance in the sims when I heard we were switching to FactSet giving the data is not point in time anymore. I am just wondering how much is this real and how is it not.

[EbenezerScrooge]

OpIncQ, OpInc(1,QTR), OpIncTTM, CashEquivQ, EBITTTM are not coming through on factset for smaller? financials?
e.g. NWLI,LNC
But they are coming through for SCHW.

[rodbarcv2]

Hi Yuval,

Please keep us posted here as the bugs are fixed. I just tried to run a diffetent sim on the beta site, and got a different error on this one: UnhandledExceptionHandler: EXCEPTION_ACCESS_VIOLATION

Sim:

https://beta.portfolio123.com/port_summary.jsp?portid=1490690

Thanks

[regallow]

No apparent execution problems so far for me with stock screens and ETF sims on the beta site, but consistently get “Invalid uid in PriceHistory” error re-running existing stock sims. Made a new, simple stock sim and it worked.

When I duplicated a non-beta stock sim’s settings and rules without using the sim copy method, the error “In Rule 2: ReqContext: UserUid not initialized” happened. There was no Rule 2 by name. The error disappeared when custom series used in a buy and sell rule were updated and saved on the beta site.

I see some very significant differences in ranking performance results between Compustat and FactSet but haven’t delved deeply into that concern yet because of sim failures.

[dnevin123]

Any update on the “invalid uid in PriceHistory” error? It was one of the first errors mentioned on this thread and hasn’t been commented on.

I have had some luck recreating simulations with said error from scratch and getting them to run on the beta site.

[yuvaltaylor]

We are working on fixing all the bugs. There are lots of them, so please be patient.

Once the bugs are fixed there will still be some minor differences between the regular site and the beta site with Compustat and “Use Prelim” checked. We realized when setting up the beta site that we had one flag for all preliminary data. We should have had two: one for interim data and one for annual data. So an annual may have been incorrectly flagged as preliminary during the first three quarters and erroneously fallen back. That has been fixed on the beta site and has not yet been fixed on the regular site. It was important for us to see how the beta site was working before applying the fix to the regular site.

As you can see, this is a work in progress. So please be patient with us.

[atw]

Hi Yuval,

Thank you very much for your update. Let me make sure I understand what you are saying.

1. Is there a difference in meaning between preliminary data and interim data?

2. Does p123’s error mean that preliminary annual data was backdated in p123 so that simulations and backtests believed this data was available up to one year before it really was?

3. Are these backdating errors limited to annual CompuStat data only or do they affect such p123 functions as “ttm” and/or other non-CompuStat data such as estimates, short interest or institutional ownership?

Thank you.

Hugh

[Chipper6]

If it was up to me, I’d focus first on exposing all the data before trying to extinguish the bugs. That’s how it was done last time during the switch from Reuters and it was an amazing job.

[yuvaltaylor]

Hi Yuval,

Thank you very much for your update. Let me make sure I understand what you are saying.

1. Is there a difference in meaning between preliminary data and interim data?

Yes. Interim data is quarterly data. Quarters are interim periods between annual data releases. Preliminary data is released by the company prior to the SEC filing.

2. Does p123’s error mean that preliminary annual data was backdated in p123 so that simulations and backtests believed this data was available up to one year before it really was?

No. When a statement is flagged as preliminary, some NAs will be filled by going back to the prior period. Since the annual was flagged incorrectly as prelim, NA is the correct value, and falling back could get you a false positive.

3. Are these backdating errors limited to annual CompuStat data only or do they affect such p123 functions as “ttm” and/or other non-CompuStat data such as estimates, short interest or institutional ownership?

These errors are limited to Compustat fundamentals only and do not affect estimates, short interest, institutional ownership, etc. Those are all completely different datasets.

[yuvaltaylor]

By the way, a few of the bugs have now been fixed. Not all of them, but issues relating to preliminaries and invalid benchmark notices should be OK now.

[Hedgehog]

Possible to share some kind of bugtracking with the community in order to improve the bugreporting in this thread? E.g. by editing the first post with something like this:

#001: The quick brown fox jumps over the lazy dog.
STATUS: Closed
#002: The five boxing wizards jump quickly.
STATUS: Open
etc.

[atw]

Hi Yuval,

Thank you for your detailed explanation.

It sounds as if interim, meaning quarterly, preliminary data has always been handled correctly. Is that so?

If quarterly preliminary data has always been handled correctly, then “ttm” data points should be unaffected too. Is that also so?

I wonder if you could provide some concrete examples, meaning before the correction and after, of how annual data points may have changed?

I ask because I’m not hearing enough of a cataclysmic difference regarding what p123 has corrected to explain the huge differences in simulation results I am seeing.

Therefore, either…

1. I am misunderstanding, and underestimating, just how significant this correction to the annual data must be.

2. There are other changes to or bugs affecting the beta site w/preliminary data that need attention as well.

Unfortunately, I don’t believe it’s possible to evaluate the transition to the FactSet data without fully understanding whatever issues are affecting the CompuStat w/preliminary data – old or new.

In addition, and in part subject to whether “ttm” data is affected, I’m not sure I even use that many annual data points in my screens and models. If I (and hopefully other users) can understand exactly which data points are at issue, it will be much easier to correct, discard or recalibrate our existing models.

At that point we can turn our attention to evaluating the FactSet data with more certainty - and while there is still time.

I understand p123 is striving to get things right. I appreciate your help with this.

Hugh

[aschiff]

To reiterate what Yuval said, the previous version of the beta server (before 2020-04-20 6PM CDT) was rife with mistakes relating to handling preliminary fallbacks. The latest beta version will more closely match current behavior in production for handling preliminary data, but it will not match exactly since intentional changes are still present. It is recommended that you run backtests again that were wildly different before the fixes.

To give a sense of the magnitude if the issues, here is a list of the affected factors and functions I could find. These issues also impacted any industry factors derived from them.

Standalone factors: DbtS2NIQ DbtS2NITTM EBITDMgn%5YAvg EBITDMgn%TTM EBITDPSTTM EPS#Positive EPS%ChgA EPS%ChgPQ EPS%ChgPYQ EPS%ChgTTM EPS3YCGr% EPS5YCGr% EPS10YCGr% EPSPExclXorA EPSPExclXorQ EPSPExclXorTTM EPSPInclXorA EPSPInclXorQ EPSPInclXorTTM EPSStableQ GMgn%5YAvg IncPerEmpQ IncPerEmpTTM IntCovQ IntCovTTM NI%ChgPQ NI%ChgPYQ NI3YCGr% NI5YCGr% NoIncP4YN2Y NoPosEBITDA5Y NoPosEPS5Q OCFPSQ OCFPSTTM PayRatio5YAvg PEExclXorTTM PEG PEGY PEInclXorTTM PMgn5YCGr% Pr2SalesQ Pr2SalesTTM Prc2SalesIncDebt Retn%Q Retn%TTM ROA%5YAvg ROE%5YAvg ROI%5YAvg Sales%ChgA Sales%ChgPQ Sales%ChgPYQ Sales%ChgTTM Sales3YCGr% Sales5YCGr% Sales10YCGr% SalesPerEmpQ SalesPerEmpTTM SGA2SalesQ SGA2SalesTTM
Functions (including their A, Q, and TTM factors): AstTurn CapEx2Sales Cash2AstTot CurRatio DbtLT2Ast DbtLT2Cap DbtLT2Eq DbtTot2Ast DbtTot2Cap DbtTot2Eq EBITDAMgn% FCFLMgn InvTurn OPMgn% PayRatio PEGInclXor PTMgn% QuickRatio RecTurn ROA% ROE% ROI% SGA2Sales%

[yuvaltaylor]

Possible to share some kind of bugtracking with the community in order to improve the bugreporting in this thread? E.g. by editing the first post with something like this:

#001: The quick brown fox jumps over the lazy dog.
STATUS: Closed
#002: The five boxing wizards jump quickly.
STATUS: Open
etc.

We thought this was a very good suggestion so we have a bug-tracking site now open for visits. See it here:

If you see bugs that are not on this site, please comment below.

[yuvaltaylor]

Hi Yuval,

Thank you for your detailed explanation.

It sounds as if interim, meaning quarterly, preliminary data has always been handled correctly. Is that so?

If quarterly preliminary data has always been handled correctly, then “ttm” data points should be unaffected too. Is that also so?

I wonder if you could provide some concrete examples, meaning before the correction and after, of how annual data points may have changed?

I ask because I’m not hearing enough of a cataclysmic difference regarding what p123 has corrected to explain the huge differences in simulation results I am seeing.

Therefore, either…

1. I am misunderstanding, and underestimating, just how significant this correction to the annual data must be.

2. There are other changes to or bugs affecting the beta site w/preliminary data that need attention as well.

Unfortunately, I don’t believe it’s possible to evaluate the transition to the FactSet data without fully understanding whatever issues are affecting the CompuStat w/preliminary data – old or new.

In addition, and in part subject to whether “ttm” data is affected, I’m not sure I even use that many annual data points in my screens and models. If I (and hopefully other users) can understand exactly which data points are at issue, it will be much easier to correct, discard or recalibrate our existing models.

At that point we can turn our attention to evaluating the FactSet data with more certainty - and while there is still time.

I understand p123 is striving to get things right. I appreciate your help with this.

Hugh

Hugh,

Now that some of the bugs have been fixed, please report specific differences you’re seeing between the regular site and the beta Compustat Use Prelims site. Try to isolate exactly what the differences are. For example, if you’re running a simulation on both sites, look at the transactions record; if you’re doing a screen, use the ShowVar and Ticker commands to see what the differences might be due to; if you have differences in rank performance, try to figure out which node is causing it. All of these will help us sort out any remaining bugs in the site. After all the bugs have been fixed, the beta site will not be identical to the regular site because of the difference in how preliminaries are handled, but the results should not vary too widely.

Also see Aaron’s comment above and the list of bugs that have not yet been resolved on the Trello board I posted.

[rodbarcv2]

A few of my sims work now, and all of them have been consistent to have BETTER results (higher Sharpe ratio, better performance) with FactSet + Exclude Prelim than with Compustat (with or without Prelim).

However, many sims are still unable to run, all of them with the same error: Invalid uid in PriceHistory

One of the sim with error for reference: https://beta.portfolio123.com/port_summary.jsp?portid=1531521

[atw]

Dear Aaron,

Thank you for your post.

1. Clearly these revisions involve a LOT more than just annual datapoint corrections.

2. Running the sims post 2020-04-20 at 6PM CDT made virtually no difference to the results I am seeing.

3. Have all of the intentional changes been posted on the bug-tracking site?

4. Sim results are apparently unchanged on the old site.

The consequences of what you are saying are horrifying. It appears that every serious fundamentals-based formula, ranking system, screen, simulation and/or designer model that relied upon CompuStat data populating p123 before yesterday evening was constructed using a significant amount of data unavailable at the time the data was purported to have been available.

Am I wrong?

Hugh