Does the PIT method account for FactSet delay in updating for earnings reports?

Research
21

I am heavily using estimates.

  1. FactSet (and the implementation from p123) is 10 times better then with the old data provider. As Yuval states the estimates get much faster then with the old provider.

  2. While I find some differences between sims and ports the ones I have found where positive, so $METC was picked in the live port but not in the sim I ran aftwerwards

  3. PIT is the holy grail, no doubt about it. If factset has a more PIT Version P123 should check it out, absolutly.
    Nevertheless PIT will be always something we might not hit 100%, no matter what, we should make peace with that fact.

One solution to capture the positive revisions: I use screens that I run daily (could do with the ports too, since they update daily (which sims do not) to get hold of stocks that have a positive estimate revision, check their rank and buy discreationary
if the rank > 90.

  1. the systems using earnings estimates run since beginning of may 2021 OOS. Bets of them made >30% since then.

  2. I do not use earnings estimates in the rank part, but in buy (and sometimes sell) rules. Stuff like CurQEPS1WkAgo < CurQEPSMean, that could
    make things a bit more robust (maybe thats why I have not much deveations, not sure)

As I stated, estimate revisions data was never better then now, not saying it can not be improved. If you compare the implementation of p123 for
example to Zacks etc. the implementation of p123 is much, much better.
Try to use them in the buy and sell rules maybe
that helps.

But as I said, if there is even a better solution from the provider, p123 should have a look at it.

22

Andreas,

I agree. And if it is okay to say it here: Zacks sucks. I continue to follow my port at P123 and have never considered going back to Zacks.

Andreas,

I agree on this also. Has P123 not looked at this already? Maybe they just haven’t told us what they found.

Best,

Jim

23

FactSet’s PIT consensus estimate data only goes back to 2009 while the consensus estimate data we use goes back to 2000. In addition, the consensus estimate data is updated in real time, while the PIT consensus estimate data is updated only at midnight, so the updates wouldn’t be available for our initial evening load. Lastly, the consensus estimate data includes many estimated items that the PIT consensus estimate data does not include. We think it’s better to use the consensus estimate data for those reasons.

Also, the best way to compare a live strategy to ranks in the past is to rerun the strategy as a simulation, then look at the historical holdings on specific dates. Usually the differences will be minor (one or two tickers), but it all depends on how many stocks the simulation holds and how sensitive your ranking system is. Also keep in mind that this will only apply to post-August 2020 data, since before that we used Compustat; in addition, you should only do this with strategies with high turnover, or else you’ll be retaining stocks that Compustat chose.

Lastly, I apologize for my last post, which was incorrect. 7/31 screen ranks should be similar to 8/3 ranks.

24

Ouch! You mean we wouldn’t be able to rebalance our ports at 1:00 a.m. in the morning?

[b]JUST ME I GUESS, BUT I WOULD LIKE TO HAVE THE LATEST DATA at 12 MIDNIGHT. Or later if there are analysts up past midnight and I could get it before the market opening. THAT WOULD BE GREAT!!!

WHY WOULD I WANT TO WAIT OVER 24 HOURS TO GET THAT DATA!!![/b]

Fine. I am not in charge of finding the best business model for P123. And I understand there are some serious considerations. I did not hear the cost for example. But just wow.

I have no problems with any of the rest of this and I appreciate the response.

Not a disagreement I think: I can document that it can make HUGE difference. I don’t have to show an image of my port/sim again I hope. Every model is different and I would have no disagreement if someone said a micro-cap strategy (tickers with few or no analysts) was not affected at all, for example. Perhaps, I would wonder what the true slippage of a stock with $10,000 ADT is, however. But that is for a different thread.

I think a lot of sims fail to predict the port at P123. The vast majority of Designer Models for example. Maybe this is just one of the reasons. Maybe not a significant one at that.

But nothing new here. Just a better understanding about what is making my sim less predictive than I would hope.

And my personal apology on his part. Marc did not think we were very smart doing a lot of sims, rank performance tests or machine learning at P123. None of these work well when there is a PIT problem.

Maybe Marc knew something about the data that I did not know. He was more savvy about quantitative methods than he let on.

But Marc was right about what sims and machine learning can tell us at P123, I think. For sure I will make less use of them and trust them less going forward. I will follow my port out-of-sample for now–ignoring the sims and not using the API for machine learning as Marc told us we should do.

IMHO, P123 will need better data to do serious machine learning. Quantopian took steps to get rid of look-ahead bias. So, I would not be the first to see a problem worth considering and alleviating where possible.

Thank you Marc Gerstein.

Best,

Jim

25

Can someone point me in the right direction to see how the FactSet estimate data works that we are using? I can’t find it.

Also, are ALL variables updated daily for screens? Including fundamental and estimate data?

26

Also a new experiment for Jim - if you change the Current Estimate data with the “1wkago” data, your sim and ports should match. That is, if this estimate look-ahead problem is the only source of difference.

OR try removing the estimate data completely just for fun and see if it matches perfectly.

By the way the fact that the sims always outperform the ports OOS suggests there is look-ahead bias and there is something quite problematic.

27

Yes.

So I get that some people use different models. Don’t use estimates at all. That is excellent and I hope these members have continued success. They are not concerned with this: clearly.

But why would anyone be surprised that the sooner you get any information (including an analyst’s revision) the better. Whether it is in a port or a sim. The problem does not even HAVE to involve look-ahead bias (but probably does for the sim).

The sooner you get information for the port the better. How long do you think it takes RT to arbitrage away new information now?

Why do they use news readers for example?

I think I do not want to buy a stock that had a major analyst downgrade because I am unaware of the downgrade. Unaware because my data provider thinks it is not convent to tell me right away. More convenient to let me know 24 hours later while my stock takes a hit the during trading the day.

I do not want to use the word problematic but it clearly limits what you can do with sims and also how well the port will perform with delayed data.

Some sims are useless and some ports could do better. Some people would not want to make it that way intentionally. It’s just common sense.

It is what it is. The main purpose for all of this has been to learn about the limitations of my sim: which I have accomplished. P123 should do whatever it wants on this.

Thank you Philip. Thank you everyone. I have learned a lot.

Jim

28

Before I do a deep dive into the look-ahead problem in estimates I just want to confirm that the fundamentals / pricing data is perfect PIT? Ie. sim vs. port would be the same if I only use fundamental / pricing data. Has anyone confirmed this or should I?

29

Regarding look-ahead bias, do we throw the baby out with the bathwater? As I argued earlier, some look-ahead bias is absolutely inevitable simply because FactSet didn’t have the data it now has and Portfolio123 wasn’t always in existence. But we have gone to tremendous lengths to eliminate it as much as possible. Show me another backtesting engine that has less look-ahead bias than we have.

Information about the FactSet estimate data that we’re using is here: https://www.factset.com/marketplace/catalog/product/factset-estimates-consensus

All estimate and fundamental data is updated daily except for EPS revisions, which are updated weekly. By EPS revisions I mean the CurQUpRev, CurQDnRev, CurFYUpRev, CurFYDnRev, NextQUpRev, NextQDnRev, NextFYUpRev, NextFYDnRev, and TotRevisions factors.

Philip writes, “the fact that the sims always outperform the ports OOS suggests there is look-ahead bias and there is something quite problematic.” This is absolutely not the case. I took 20 live portfolios, none of which were updated in the last two years, and ran them as sims over the last two years, and compared the 2021 performance of each live portfolio to the corresponding sim. In twelve cases, the simulation did better, and in eight cases the portfolio did better. Statistically, this is no better than random.

Philip also wrote, “I just want to confirm that the fundamentals / pricing data is perfect PIT? Ie. sim vs. port would be the same if I only use fundamental / pricing data.” Certainly not! This was never the case with Compustat, and is certainly not the case with FactSet. Perfect PIT exists only in your imagination. Live strategies have never matched their corresponding simulations perfectly, whether relying on estimates or not. There are always going to be differences in stocks chosen simply because the mathematics of ranking systems are extremely sensitive to very tiny changes.

I’d like to sum up this issue as follows. Financial data is extremely imprecise. What do we measure when we use financial data? We measure what companies decide their data is. We measure subjective things. As I put it in my webinar, “In most sciences—and even a lot of sports—you can work with completely objective statistics such as height, weight, and distance—or field goal percentage, rebound percentage, and number of points scored per game. But not in finance. Almost every single number on a financial statement is an interpretation by the company’s accountants and officers.”

These subjective things are then put into ranking systems, which depend on extremely minute differences to calculate ranks. Many nodes, for instance, are dependent on industry composition; if one stock in the industry disappears from or gets added to that industry, the node rank will change; even more frequent would be a value in one stock changing from an N/A to an actual number, which will affect all other stocks in the industry. I’m sure you can come up with dozens of other examples where even the flap of a butterfly wing will change a rank.

I think it’s a huge mistake to expect precision in anything to do with finance. “It is better to be vaguely right than exactly wrong,” said Carveth Read. That is a fundamental rule of logic, but it applies especially to finance, which is as inexact as any system of data can possibly be. (If there is a more inexact system of data, I’d like to hear about it.)

I am a firm believer in backtesting. But I know that my backtested results are mutable, approximate, and chimerical, and that the correlation between in-sample and out-of-sample performance is always going to be very low (Best-case scenario? About 0.1 IMHO). Why? Not because of any flaws in the backtesting system. Backtesting using Portfolio123’s simulation and ranking tools is by far the best available. Simply because of the nature of markets and the nature of financial data.

Sorry to spout on at length, but this discussion seems to have gotten away from the fundamental facts about the way Portfolio123 works. I have never once seen completely exact replication of results, and don’t expect to. I have seen good approximations of results, and I think users should be happy with that.

30

Yuval, as I stated there nobody has perfect PIT and the “almost PIT” of P123 is from what I know not matched by other services. The length P123 goes for clean data is simply unmatched. And again, my models improved a ton by going to FactSet, especially due to the faster and better handling of estimates (stuff like CurQEPS1WkAgo).

I do not want to give too much away from my systems, but in small caps there is a huge edge trading stocks that had earnings estimate up revision with great fundamentals, momentum and industry momentum. Just look at the charts of SBOW (04/26) and METC (05/17). Big caps get a huge gap up (after earnings), small caps usually have super nice drift (depending on the fundamentals) after they have an eps upward revision that can be exploited beautifully.

My best systems simulated around 45% without the estimate revisions, my best systems now are north of 75% (even with less DD).
Also, my best systems (with earnings estimates on buy and sell filter) seem to have an implicit market timing: if they do not find small caps with upward estimate revisions, they go (step by step) to cash (or with the hedge into TLT). That “not finding” resonates very, very well with market phases that are not so hot (or really bad), that in turn produces very nice capital curves with much less DDs.

So all good, I am really happy how P123 improved and if it stays like it is fine!!!

But please, if we would have a chance to load the estimate data during the night to get even faster and it would be ready before the market, I am sure my (an others) models would get even much, much better.

Thank you!!!

31

So I will look at this further but Marc Gerstein was there when the decisions were made about CompuStat and FactSet data. He said we should not be using sims or doing machine learning the way many of us do. I take that as good, informed advice now.

Looking at the Designer Models it is clear that a sim doing well is generally a predictor that a port will underperform. If someone likes to focus on anecdotal examples knock yourself out.

I had thought the problem was overfitting. Maybe it sometimes, but not always I find out.

Any sim or machine learning data has a big question mark after it now.

I find no rational reason to risk any money based on sim data or machine learning data at this time. Especially, knowing that the only guarantee I have is that the data is not PIT and P123 does not think it is a problem that it wants to address or even quantify.

Marco, I would at least say that you do not like non-PIT data in principle if you want to proceed with machine learning at P123.

Maybe look at the data more generally than I can. We have a lot of anecdotal stories here (including mine).

Maybe abandon the machine learning if your only guarantee is going to be the the data is non-PIT. That some people are reporting problems but you have no plan to identify the cause of any problems you identify, or to quantify them or find solutions where warranted. Or that you have already done what you are going to do about making it more PIT. Figure it out for yourself it you think there is a problem.

Already doing something about it? Seems it was I who has brought to light the possibility that earnings estimates may be a problem. Maybe the only major problem. And I pointed out a potential solution if it is, in fact, a significant problem. BTW, I would be more flexible on the download schedule if this does turn out to be a significant problem.

If that is the only problem: good for P123. You should find out. Quantify it. Just let us know.

Big long posts never making any attempt at identifying what problems there are (if any) and stating your intention to not quantify any problems or look for any solutions where problems exist: bad for P123 IMHO.

Jim

32

Yes we all know there is no such thing as perfect PIT I’d just like to understand how non-perfect it is.

33

THIS.

I don’t expect perfect data. We can do quite well with imperfect data. But I do need transparency.

You have a excellent platform. Arguably because of unparalleled transparency. We have been allowed to have open discussions about issues. Known, fixable data issues were fixed promptly. Known, but unfixable issues were not swept under the rug, but were put into perspective. Issues that were suspected but not confirmed were investigated either by portfolio123 people or by users and discussed openly.

It’s fine to put data issues in perspective, but I do need transparency.

34

I would be happy to help in this understanding. As Chaim said, we are and have always been quite transparent about our processes and our data. If you have specific questions about PIT-ness, fire away! I’d be happy to answer them, and if I don’t have the answers, to ask others who do!

35

Yuval,

I suggest we start with what we know. FactSet says the earnings estimates are not PIT. I would suggest that you quantify that.

Since this seems to be new information to you, asking others that know seems appropriate.

I am sure that FactSet would be happy to talk to you about the advantages of their PIT version. Why it is better and in so doing make you aware of the problems with the non-PIT version that the PIT version solves.

If you do not already have someone to contact at FactSet here is a link for: "one-on-one demo with a FactSet specialist "

I do not know pricing etc. That would be a business decision for P123. But the information about what we are using now that this would give would be useful for members, I think.

Plus, the best thing for P123 would be if this were found to be the sole problem or almost exclusively the problem. I suspect that may be the case. If true it would be a huge advantage for everyone involved to learn this. It is better for everyone involved including P123 if this is almost exclusively the problem with fundamentals having few PIT issues.

Of course, I do not know what you will learn in advance.

Jim

36

Yuval,

Can you please explain how the FactSet estimates work? I went to the link you sent earlier and can’t find anything on how / when the database is updated as new estiamtes come out.

37

Philip,

I hope Yuval and others will respond but here is the problem with what we have in 2 sentences (according to FactSet):

The traditional FactSet Estimates Database uses the “Research Date,” or the market date of the broker contribution, to calculate the consensus. This exposes a look-ahead bias because most research dates are T-1 day of the broker contribution’s “Input Date,”

So let’s use ticker TALO as a random example with our present data at P123. Suppose an analyst changes his earnings estimate for TALO on August 2 at noon so that his estimate lowers the consensus earnings estimate.

You rebalance your portfolio in the morning on August 2 and the analyst has not even change his estimate yet when you rebalance your port so you buy TALO. You could not possibly know about it for your port because it has not even happened yet!

So with your port you have bought TALO and the analyst changes his estimate later in the day. The change in earnings estimates enters the FactSet database on August 3.

But guess what: the traditional FactSet database (that P123 uses) says it happened on August 2–which it did I guess. When you run the sim the change shows up on August 2 rebalance and the sim is able to use that information in the sim (even though you did not have that information for the port). You do not buy TALO in the sim because the sim knows the consensus earnings estimate is lowered that day.

The sim avoids buying a stock that will have its consensus earnings estimate lowered later that day. The sim avoids buying TALO while the port does buy TALO.

A clear lookahead bias. A clear lookahead bias that we have now with our present FactSet data.

Source: ACCURATELY BACKTESTING FINANCIAL MODELS THROUGH POINT-IN-TIME CONSENSUS ESTIMATES

This lookahead bias is clearly helpful for the sim over the port. Clearly better not to have it that way if you want to have any idea how the port–with no lookahead bias–will do.

And this is significant: the sim is always a trading day ahead of the port. If it is rebalanced every day with trading on the open, the sim is usually trading with data before even D.E. Shaw & Co. and Renaissance Technologies (RT) get the data.

A nice thing for the sim and pretty cool if one could do that in real life.

You (your port) always trade with data a day (or more) after RT gets it.

Now, if I could just get a handle on why my sim seems almost RT-like :wink:

Best,

Jim

38

Usually I’m not a fan of reading walls of text but your example is clear about a problem I been seeing and couldn’t pinpoint the issue. Thanks for the explanation!

39

Francois,

Thank you. Glad you found my wall-of-text helpful.

Going forward we can probably shorten it with this: sims with earnings estimates are fiction.

Best,

Jim

40

Jim, Yuval, now I am confused!

from what I have understood your statments contradict.

Yuval you pointed out that from August 2000 there is no lookahead bias.

Also what I understood is, that p123 takes data from F. and makes it pit and it takes around 24 hours to update the DB → so its pit but lets call it “PIT +24h” (which I could live with, but a look ahead should be really avoided!)

Yuval, please explain. Thank you!!!