Does the PIT method account for the fact that it will take the FactSet crew a few days to update for earnings reports for small caps?
If not, has anyone estimated how many days on average it takes for FactSet to update the database after an earnings report for small caps?
Generally, the news date we give for dates prior to July 2020 is one day after the date FactSet gives for the earnings report, though there are a few exceptions. If FactSet doesn’t give us a date for the earnings report (which isn’t very often), we use a complex checklist to determine a reasonable news date for PIT data.
In July 2020, when we switched from Compustat to FactSet, we started keeping track of when FactSet actually gave us the data, and we use those dates.
I’ve been observing this ever since we switched from Compustat to FactSet, though not systematically. With small caps it’s usually less than 24 hours. With microcaps and Canadian F shares FactSet can take a little longer than 24 hours, but I have rarely seen FactSet take more than 48 hours. I’m not sure, but overseas F shares may take longer than that.
In general, FactSet is much faster than Compustat, who on occasion will take three weeks to process an earnings report.
So its possible if dealing with Canadian F Shares or microcaps that the simulations suffer from “lookahead” bias, assuming you are using hte FactSet database going forward.
Yes, if you’re testing prior to August 2020. Post-August 2020 there should be very little look-ahead bias.
And the look-ahead bias won’t be limited to Canadian F Shares and microcaps. Remember that FactSet was a tiny company back in 2000 and that they didn’t acquire the Thomson Reuters Worldscope database until 2008 (this became FactSet Fundamentals). A huge number, if not the majority, of quarterly reports in our database prior to 2008 were likely not reported by FactSet at all at that time but were backfilled later.
To complicate matters even more, Portfolio123 did not even exist back in 2000, so if you wanted to completely avoid look-ahead bias, you’d want to exclude all data prior to our launch in 2004, and only include that data once we actually started receiving it, and you’d want to use Thomson-Reuters data for the first few years, then Compustat for the next dozen or so years, and only use FactSet data once we switched in 2020.
Our database is pretty point-in-time-ish if you pretend that FactSet always operated the way they do now, that Portfolio123 has always used FactSet data, and that Portfolio123 was in existence back in 1999. If you don’t pretend, then yes, look-ahead bias is present.
Let me agree with this probably being the general case. And even add that I can show some sims and ports that look pretty much the same.
But I just reduced the amount that I fund this port by about 80% because I no longer trust that the sim tells me anything.
I am considering that this may be a special sim/port that has found the lookahead bias in the data. After all, having a crystal ball into the future is one way to make money (even if it is a simulation). And we should be able to find that in the data (when it exists).
Overfitting is not a problem with this sim. The out-of-sample sim does as well as the in-sample sim. It is the port not performing like the sim that is a problem. Images: sim then port.
I am gathering more data. The only point I would want to make is that there might be something specific about this sim. But maybe I made a mistake. Or maybe just a statistical fluke where the sim and port will look more equal with more data. Like I said, I am gathering more data.
My working hypothesis is that maybe machine learning found what works the very best: having a crystal ball into the future with lookahead bias in the sim.
For sure, I am getting less confidence in the sim with each passing day. With the sim providing no reliable evidence as to whether the port is any good I have made MAJOR changes in my asset allocation. Essentially paper trading this port to get some data that I can trust.
Hi Jrinne,
I think, if you dig and analyze the data (manually) a little, the sim will tell you some. Both graphs are rather similar, only the performance differs a lot.
I suspect the following: Your strategy is a high turnover with small- or microcaps, and a low average win per position. This gives the slippage a lot of weight. Compare the slippage of your sim to that of your (life) port: Do you use realistic slippage in your sim?
A second reason for the difference I can imagine, are different positions in sim and port. You know that the sim does not use the data of the monday morning update. This, in the long run, should not give you worse results with the port.
A third possible reason: the sim may eventually reset the weight of the positions at rebalancing. Have you done this with your port in the same way?
Matthias
Matthias,
Thank you very much for your comments. I am trying to sort this out and you have some great ideas.
Slippage is not the answer, however. My slippage is not NEARLY as high as the variable slippage used in the sim here.
In addition, the sim and port usually buy the same ticker (but not always). The port has the slippage that I actually get when I trade (always edited to the real price I get with my broker). So the port includes my slippage. In those instances where the sim buys the same ticker it adds ADDITIONAL, UNREALISTIC slippage to the sim. This should put the sim at a disadvantage yet the sim is doing better. It must be something else.
With realistic slippage in the sim the difference gets ridiculous and I have no choice but to step back and take a look at this: to get a reality check from out-of-sample data.
My sim does not reset the weights. It sells based on RankPos or a few sell rules. The stock is purchased, the entire position is held and then the entire position is sold.
Yes!!!
So I am using the screener (with no buy rules) to sort this out. I have not been doing this for long, however (exactly 2 weeks now). The “Force Positions into Universe” in the sim and port cause too much chaos (literally) and confusion.
Some differences occur in the screener the next day. BUT, a this point (2 weeks of data) it is arguable how much effect this has.
First image, screenshot of the screen Monday morning which gave rank data as of 8/21/21 (I always check that the data is fully updated). This uses exclude preliminary data which I think will end up being important.
Second image, this morning looking back to 8/21/21.
This argues either way (with this limited data). There is a difference but does it really matter that GSKY and ZIM are switched? You can find some additional differences further-down in the ranks. You can use it to argue either way at this point, I think, and I will leave it at that for now.
Also, will there be more changes when I check this again in the future?
Matthias. Extremely well reasoned!!! I very much appreciate your help with this!!!
I am open to all ideas at this point including: no worries, I should put all my money into this port, there will be a little volatility but I will have my own personal Learjet in no time at all. But I think I will step back for a moment before committing to that. Get a little more data first.
So, just to be clear on what you are saying: It is possible that the port is getting more up-to-date data that–so far at least–is harming my port relative to the sim. Did I understand what you are saying correctly?
I think I might not understand why P123 would want it that way and would ask P123 to explain why they do it this way. And consider changing this if it is true (and possible to change).
Best,
Jim
We can immediately rule out slippage as the problem since these stocks are fairly liquid. Unless you’re turning over the entire portfolio each week.
Could you post all the transactions in the sim and in the port? I can go through them maybe?
I think we have stumbled upon the same problem.
Hi Philip,
Thank you for your interest. Is it convenient for you to send me an email through P123? I will reply with the attachments you requested if that works for you.
Jim
Jim what type of factors are used? Estimates, fundamentals, short interest, price and volume, all have different potential types of look ahead biases (if any).
Chaim,
Mine is a value and sentiment port.
It is, and always has been, a matter of HOW NOT-PIT the data is. There is no factual debate about this.
Clearly, it is FactSet’s opinion that their general offering is not PIT and I am assuming they know. Here is a link to some information about their PIT offering: AT A GLANCE: FACTSET ESTIMATES POINT-IN-TIME CONSENSUS
A quick and easy way to alleviate any confusion as to whether this is the data we use at P123 is to notice FactSet’s PIT offering only goes back to 2009.
Here is a link to: Request a one-on-one demo with a FactSet specialist
Rather than speculate further I wonder if a pro (who may get an answer back after filling out the form) can fill this out, talk to FactSet and provide a little more data on this than my port/sim provides.
I remain open to the idea that there are factors that are useful real-time in my port. Indeed, I paid a membership fee this year because I think it is possible. But I just want facts on this (relevant to my port). Fortunately, it is just a matter of simple accounting to get the facts I need whether anyone wants to (is able to) provide additional information or not. I have made no claims on that point based on the data I have now–either way–other than to say it is possible. Time will tell and any factual information that anyone wants to provide to quantitate this in the meantime would be greatly appreciated.
I am going to ask P123 to broaden the discussion on this and not focus on discrediting my port. P123 should provide additional information–after consulting with FactSet if necessary–on this topic and not attack what little information I am able to provide. I am not going to open-source my sim and one can debate whether the port does not follow the sim because of something I have done until the end of time.
Here is a white paper written by FactSet for some publicly available facts on this: ACCURATELY BACKTESTING FINANCIAL MODELS THROUGH POINT-IN-TIME CONSENSUS ESTIMATES
Best,
Jim
This means that sims starting in 2009 should show the same performance with “PIT Method - Prelim” USE or EXCLUDE.
I just tested this for a model using the S&P500 or the Russell1000 universe. Performance is identical with USE or EXCLUDE.
Georg,
This is just simply incorrect. Or actually just completely unrelated. Do you have a port to compare the sim to? But thank you for your near instant response on this.
One may have to do more than skim-over the FactSet information if they want the truth of this.
If one wants to hold-up their anecdotal models as Gosple and the final word on this (and ignore FactSet’s information) then readers should be aware of one thing. It might also help if they are addressing the right topic.
An overfit sim (one fit to noise by definition) and the resulting port should perform the same with very minor random differences. THIS IS EXPECTED FOR MANY MODELS AT P123. For the last 2 years only about 27% of the designer models are outperforming their benchmarks. The medial model has an excess return of of about -15%.
You should not expect to see any difference at all in the sim and port that most people will show you.
Only a sim that has found data that works out-of-sample has the potential to perform differently than the port–due to any factors present that are performing well due to a lookahead bias.
In any case:
Anyone who wants to focus on Georg’s sim results can ask him to make it available as a Designer port and get more insight into his anecdotal results.
My port and sim do not perform the same and Philip started this post wondering what is going on. No one claimed there is a problem with every port: certainly not me. But there will be some differences according to FactSet and some of us may notice more differences than others. Some may seek to quantitate this and will gather data on their sim/port out-of-sample for now.
FactSet has some data on this that may be objective too. Of course, they are torn between saying PIT is worth the extra money and saying those who cannot afford it do not need it. Any possible spin aside: I THINK I WILL TRUST FACTSET FOR SOME DATA ON THIS AND ACTUALLY READ SOME OF WHAT THEY SAY.
Thank you Georg for helping me understand why I will just have to collect my own data (which I already knew). Truly informative.
And again. You should not expect to see much difference in the sim and port that most people show you. If you want to take one person’s anecdotal evidence over what FactSet has to say: please do.
Best,
Jim
All,
So, for anyone interested in this, it is easier to sort this out than I realized: Thank you Philip for your request for the transactions (let me know if you did not get those). Your request got me thinking about easy ways to get some answers. Matthias, your comments were helpful also and much appreciated. Georg, thank you for helping us try to figure out what effect (if any) Exclude Preliminary Results has.
Below is a screenshot of my port holdings on 8/3/21 and their ranks as of 7/31/21.
To be sure, “Force Positions into Universe” can change this a little but it should not be causing huge changes. This can be compared to the ranks in the screener now, entering 7/31/21 as the date in the screener (same universe):
ARLP was 99.96 in the port. In the screener today it is 99.8364 (for 7/31/21)
CPG was 99.88 in the port. In the screener today it is 99.9182 (for 7/31/21)
TZOO was 99.84 in the port. In the screener today it is 91.1247 (for 7/31/21)[
MATX was 99.80 in the port. In the screener today it is 99.8773 (for 7/31/21)
ALK was 99.75 in the port. In the screener today it is 83.5583 (for 7/31/21)
I post this mostly as a tool that you may want to use. You can easily draw your own conclusions on your own sims and ports without my comments.
But am I on solid ground thinking I should stop expecting this port to look like the sim? Am I missing something? That is not to say anyone else’s port or sim will behave like mine.
It would probably take a pro to tell us anything more than what you can get from links above about FactSet PIT data–should you think that is pertinent.
I hope this might be helpful for anyone who shares my interest in this. My only reason for looking at this is to figure out how much I should invest in this port.
Thank you for allowing me to discuss this in the forum. People’s comments have been helpful and have undoubtably saved me some money. I thank you, my daughter thanks you and my wife–as always–is oblivious to this but she benefits also.
Best,
Jim
Jim -
Portfolio holdings on 8/3 should reflect the ranks as of 8/2, not 7/31. Can you redo this?
Thanks!
Yuval,
I do not think I can at my end. Am I unable to get the ranks the port was using for the tickers when the Monday morning port bought them?. That is a problem if I cannot, I think.
What is Rank>Holdings used for if not to give me this data? Is something just mislabelled?
I enter 8/3/21 and that is what I get: expanding the screenshot for my port. BTW entering 8/2/21 gives the same ranks but includes data on the tickers I sold on 8/2/21 which I found confusing but I will screenshot that too if you want. The ranks do not change when I do that.
Going to the next week I enter 8/10/21 and I get rank data as of 8/7/21 (expanded screenshot)…
…This week: rank data as if 8/21/21 will include an image if you want. Or for any other week you may be interested in.
Force weekly ranks has always been “No” if that helps (image).
What would you have me do on my end next?
So anyway, what can I show you next? And if this is not the way to do it, how can I find out how much P123 is changing the ranks later due to PIT issues or whatever. The only guarantee that I have from P123 so far is that there are some PIT issues and that this can be enough of a problem that FactSet has a separate PIT dataset available. Is there a way to see how much this is changing the ranks in my sim versus the port?
Can I request that you make that information available if this is not the way to get it and you do not have another way for me to get that information now?
If it is not a PIT issue and the ranks are not changing what can I or P123 do to make my sim somewhat similar to my Monday morning port so that I can go back to using it for guidance on how much to fund the port? Right now I have to wonder how much the sim says about the port and what is causing the large discrepancy.
A sim has 20 years of data that I would not mind being able to use for guidance if possible.
Jim
Yuval,
I am able to show you the ranks from 8/2 in my port as you requested: using the transactions page. These are the ranks that were on P123’s servers on 8/2/21 (fully updated) without question and without any labelling issues here. The values do not change from what I previously posted as you can see.
Thank you in advance for helping me determine the cause for the large discrepancy between my sim and port.
I am most interested in whether the ranks are changing (which still seems to be the case). And if so, why and by how much (or how often).
I think it is a reasonable question. One of critical important that addresses the usefulness of sims AND to whether the API provides useful data, IMHO. I have looked at this to the best of my ability without bothering anyone for several months now (since 1/22/21) and have simply provided a method for looking at this critical issue on the forum (in a thread that i did not start). You imply my method may not be a good one. I wonder if you have a better method (or any reliable method that we can use).
Best,
Jim
The ongoing PIT issue is very concerning. The extra noise added by an apparent look ahead bias overwhelms the signal in many cases. It drastically reduced the usefulness of backtesting, which is the point.
We need to look at the big difference in rankings ie. TZOO and ALK.
On July 31, their rankings in Jim’s sim vs. port were very different. They had earnings reports a few days earlier, July 21, and July 22. Given that FactSet only takes a couple days for microcaps, this can’t be the issue.
Jim has said he uses value and sentiment. I am assuming pricing data is correct and I’ve already ruled out financial statement data for these examples. That leaves Analyst Estimate data. Jim can you confirm whether you are using analyst estimate data in your models?
If it is a look-ahead problem with the estimate data it could be fixed (improved) in the sims by replacing any estimate data that is “current” with the “1wkago” version eg. switching CurFYEPSMean with CurFYEPS1WkAgo.
Also, just reading through the FactSet link that Jim provided and they really use the phrase PIT a lot, for something that apparently isn’t PIT? Does anyone know more about whether their estimate data is PIT?
Also just for Jim: In my opinion, I wouldn’t throw away your strategy - the performance OOS is still quite good despite these issue. Even though its not keeping up with the sim, the OOS performance of your port is still statistically significant.
AND just one more thought I had: I don’t think this is TRUE look-ahead bias. If the estimate data in the backtests had a true look ahead bias our backtest results would be god-like. We would be “trading” (in the backtest) on the analyst reaction to earnings reports before the earnings reports came out. That would result in annual results > 100% and sharpes in the 5+ range. So I don’t think that’s happening since I don’t think many people are seeing those kind of results in the backtests. My gut feeling is that FactSet is just a little slow to update their database going forward, but in the backtest that delay is not accounted for. So if an analyst estimate is released on July 31, it shows up in the backtest as Aug 1, but in reality FactSet didn’t update their data until Aug 2. Or something like that. Anyways just a hunch.
Philip,
Excellent assessment on my port I think. I will basically confirm each point below. But PLEASE UNDERSTAND WHAT I LINKED TO!!!
Please.
The FactSet link is not what P123 uses now. FactSet has more than one offering.
P123 uses data from FactSet that is not PIT at the present time. FactSet says the data P123 uses it is not PIT. P123 says it is not PIT. Everyone says it is not PIT. There is no factual debate on this. It is not PIT.
No one thinks the data at P123 is PIT and guess what: you found that my sim is not PIT (or at least that the ranks change after the fact). No surprises or inconsistencies. Just what everyone already knew and agreed upon.
The link I provided is to a different offering by FactSet that P123 is not using. P123 has said they will look into that offering (that P123 does not use now) but I have not seen anything in the Forum from P123 about this. I might have missed it however.
I am. Which ties in nicely to the above doesn’t it?
Assuming P123 looks into the PIT offering and finds the price is too high that could be a solution. But maybe I will ask again, considering P123 has already said they will look into this. Is the PIT version out of reach?
Thank you Philip. I am still following it. I have only adjusted the amount of funding for this port based on a change in my confidence in the port. I think I can no longer use the sim as any evidence of how well the port will preform out-of-sample in the future. If the port continues to do well with time I will allocate more resources to it again in the future.
Philip, I may disagree on this one point. While one can debate what “God-like” means this sim is not realistic. Image: full in-sample sim up until 1/22/21.
And if you look closely at the FactSet link, look-ahead bias is in the data we use. That is just an uncontested fact, I believe. That is why we might consider upgrading according to the white paper on FactSet’s offerings.
My sim is almost God-like due to look-ahead bias, IMHO. I am not claiming it is special as a port however. I do not have enough data to say much of anything about my port, IMHO.
In summary, I think machine learning found what gives the best returns: a look into the future.
Thank you Philip!
Best,
Jim
