P123,
Thank you for the above, very clear, explanations.
Best,
Jim
Jim,
I’m not sure which thread you mentioned this, but I can corroborate that I have experienced the same thing - I rebalance recommendations change in the middle of the day. Never seen it before / or maybe it happened and I never noticed.
I had a look at my live strats and sims.
Actually, live strats do in some cases (recently!) better then sim (sim often finds more stocks and sells earlier then the live strat). No Idea why.
On those strats that do better live I am using the following buy filters
Rank > 90
RankPrev(1)/RankPrev(0) < 0.90 or RankPrev(2)/RankPrev(0) < 0.8 or RankPrev(3)/RankPrev(0) < 0.8 or RankPrev(4)/RankPrev(0) < 0.8
CurQEPS1WkAgo < CurQEPSMean (would not work with prelim off, since I would loose weeks on buying the stock on small caps, same with
all other estimates).
CurFYSales1WkAgo < CurFYSalesMean
NextFYSales1WkAgo < NextFYSalesMean
In the ranking system I use cashflow (Quarter).
Most important question: are those buy rules effected by lookback bias (I think we discussed this by initiative and help of Jim, p123 found the problem and fixed it).
Also important: How big is the cash flow problem. Does get the preliminary cash flow overwritten (not sure if I understood this right) by the non preliminary data? If yes, that would mean my sim would have a lookahead biax (not huge, its only 5% of the ranking but still).
Not so important: any idea why some live strats doing better live?
Best Regards
Andreas
All,
I do not pretend to know what is going on. But I do have a simple and honest question: Is this supposed to happen? If it is, then I have some follow-up questions like does P123 snapshot all of the data since June 2020?
This is JUST ONE REBALANCE DATE b[/b]. Transactions for the port then the sim for the same ticker and same date. RANKS AT THE FAR RIGHT are different enough be be significant, I think.
Why and what might I be missing? I am not saying I know the answer, but not understanding how this could happen, I closed my port which was not doing as well as the sim. This port is closed and no longer funded. I closed it just after 10/11/21 with some of the transactions (and reasons for closing it) shown below.
To summarize, the port was not doing as well as the sim, and the port had different transactions than the sim, in part, because the ranks on the transactions were different for the port and sim. The universe and ranking system are the same as confirmed by this screenshot. Port first then sim as before.
Thank you for any understanding of how this could happen or why I should fund this port again with no concerns…
Best,
Jim
Jim -
Your ranking system is extremely susceptible to tiny changes in estimates, and we don’t have any way to track how FactSet is backfilling those. A-Mark, for instance, has only two analysts. I’ve attached FactSet’s estimate history for those two analysts on the three estimate measures that comprise most of your ranking system. You can see that between the end of August and mid-September–four weeks prior to your sell date–all the estimates changed drastically, some going up, some down, some going up and then down, some going down and then up. If FactSet just adjusted the date of one or two of those changes, or adjusted the EPS estimate, that would cause the rank to change. What’s more, FactSet might have added one analyst recently and then backfilled his/her estimates.
I would advise using a ranking system that relies on far more factors, preferably largely uncorrelated and unrelated.
- Yuval
Yuval,
What makes my ranking system so susceptible?
I guess you have looked at it.
It has VERY NORMAL VALUE FACTORS one being EBITDAQ/EV. And normal sentiment factors that I learned from Marc and have been posted numerous times over the years by many members.
Ultimately you may be right. My ranking system has only 6 COMPLETELY normal factors that everyone would recognize!!!
Only one buy and sell rule that basically ensures that there have been no recent analyst downgrades.
I would argue what I did worked to prevent overfitting. The sim continues to perform beyond my wildest dreams
So I think a lot of noise factors may drown out the signal-making the sim and port more closely aligned.
So I agree completely actually.
Take home points:
-
Is supposed to do this
-
P123 does not take a snapshot of all of the data.
Did I miss anything?
Thank you for expanding my knowledge of P123 and FactSet data.
Your comment are helpful and much appreciated.
Jim
Yuval,
As far as correlation is concerned, I performed PCA and factor analysis to reduce correlation between my nodes.
My sentiment and value nodes are completely uncorrelated.
Doing PCA and factor analysis stops me from adding a lot of noise factors which I say again: did prevent overfitting.
And one more question. The sentiment data is misleading in the sim no matter how many other factors I add, right? The misleading data just gets diluted by adding more factors. I don’t have to hire an AI expert for that right?
Extremely helpful. I did not fully understand FactSet’s earnings estimates data.
Honestly, I thought the lag recently added to that data had made it the most reliable data available at P123.
Jim
Yuval, please have a look at my question, thank you :-)))
Perhaps less is more? The difficulties we’re all having with prelims is why most research papers for factor analysis only use annual , final data that has been audited.
Preliminary data might just be completely useless for the type of analysis we do. During prelims data is incomplete, unstandardized, and can change. Some factors in your systems will fallback, others do not. Add to this that markets have knee jerk reactions creating value traps, and you have a complete mess during earnings. And having Compustat vs FactSet does not change the narrative much.
I’m not saying we’ll get rid of prelims. All I’m saying is that P123 should default to a much more stable mode. For Ex: when a user starts a new strategy it should default to avoid buying or selling during preliminary data (it’s possible to do this now but it’s a bit convoluted)
Marco,
Thank you very much for taking this seriously.
I do not use preliminary data: to the extent that checking the box to eliminate preliminary data completely eliminated it.
Yuval has just said earnings estimates are not at all reliable.
That being said is FactSet’s PIT offering of earnings estimates data not a possibility? Is it really PIT?
As it is, I think Yuval’s is right about the earnings estimate data.
That is why I have been importing Zacks rank data into P123 using InList.
Again, I am extremely grateful for the information you have provided and for addressing this in a serious manner.
Jim
FactSet estimates are quite reliable. Not sure what Yuval is referring to.
I apologize for being unclear and using the word “backfilling.” What I meant is that clearly something must have changed if Jim’s live strategy had a rank of 90.4 and the simulation had a rank of 39.2. Certainly from the data we have now it looks like the rank should have been 39.2. So FactSet must have done something to the data, but we don’t know what. Everything else in my post was a wild guess.
Yuval could have expressed it better.
But ultimately with regard to what he said he is correct all around isn’t he?
That was just 10/11/21.
Thank you Yuval for offering a plausible explanation for why the ranks are different.
I was investing over $250,000 on that port at one time when I thought it was credible (none now).
I did make money so no complaints.
Somthing is not right and I feel like a fool for taking any of it seriously with that many changes in the holdings and ranks.
As short as Yuval’s analysis was—and I prefer to use Zacks—it was an analysis.
Jim
Jrinne,
What time did you run the rebalance when you got the suspect ranks for AMRK & CRC? Can you give me the unique id or name of the live strategy?
I see you were doing rebalances on Saturday 10/13, Sunday 10/14 and Monday 10/15 starting at 4:30AM
Data is flowing in from Friday till Monday around 4AM. And after the fact it’s all lumped together as the “weekend update”.
Only rebalances after Monday 4Am should be trusted not to change.
Thanks
Marco,
I never do rebalances on Saturday, ever, with a live real port. I have been playing with InList from Zacks (which you cannot do with a sim) to see how many holdings it would have. I probably rebalance at all sorts or days and times with that.
I always check to make sure the data is up to date. I am never up at 4 am to do ports. My alarm goes off at 5:30 am except on surgery day.
If I I somehow made a weird error on some day that is just one day. I might have gotten busy and rebalanced on a Tuesday, like once. Would not be shocked if it was twice.
The port is 1Monday because I also ran it on (2)Tuesday, Wed, Thursday ETC and the number kept them in order. They do not do as well as the sim either, BTW.
It is in “Recent Archive” because I shut them all down (and put them into an archive). I do not know where to find the ID but would be happy to find it if you direct me.
Thank you for looking into this.
Jim
The lookahead bias that Jim pointed out to us a few months ago was corrected that week. We don’t know of any other lookahead bias besides the occasional cash flow numbers being filled in in prelims. This not an extremely common occurrence, since the large majority of companies announce and file on the same day, and some of those who announce early announce their cash flow along with their earnings.
The fact that it relies on only nine data points (I think). So if one of them shifts, it makes a huge difference in rank.
If you were using a ranking system like Core Combination or a variation thereof, there would be so many data points that if one or two shifted it would barely make a difference.
Yuval,
Edit: so I get your point if the goal is to match the sim and port. The image below was on unrelated subjects: the ideal number of factors for a port, noise factors, overfitting and optimizing a port to maximize profit. There is no reason to get into that here. I am sorry I cannot delete the image.
Thank you. You and Marco have been very helpful.
Jim
You are basically looking for changes in EPS estimates by comparing , for example, CurFYEPSMean with CurFYEPS4WkAgo
I plotted these for AMKR in the fundamental chart and maybe there’s something fishy with CurFYEPS4WkAgo around the rebalance date. The problem is further complicated because it’s around the same time NextFY become CurFY, but I don’t think that’s the issue.
It’s just a hunch right now. We’ll take a look again on how we calculate CurFYEPS4WkAgo to see if there’s any chance of it changing once the live value becomes part of the history. We do the date math using days, but FactSet timestamps these down to the second. Maybe when we go look for the value 4 weeks ago we’re off by a few hours due to a lower precision.
In any event this type of inconsistency should not invalidate your system in any way. The live portfolio may be looking at CurFYEPS4WkAgo but the simulation resolves to CurFYEPS4WkAgo give or take a few hours in some cases. That’s a meaningless difference. A live strategy with 100 positions should be very close to a simulation with 100 positions over the same period. But of course , if you compare a 5 stock live portfolio with a 5 stock simulation it could be completely different.
Makes sense?
Marco,
Thank you. Obviously, that would be ideal for me. And I appreciate what you have done.
You think the out-of-sample results of the sim are real then? The port performing at about 1/2 what the sim was doing (the excess return was roughly half) is just bad luck from random changes caused by an hour here or there 4 weeks ago.
Also I introduced some random changes like when I added or removed cash. I understand there will be some differences in the sim and port that are not a problem.
But long term just plug back in and I’ll be ready to start using summer as a verb in no time (e.g., I’ll be summering in the Hamptons).
I hope that is true. Absolutely hope that is true and I think it is possible although I would not bet on it here today. I will plug back in with some amount of money. Look at it again going forward.
I’ll show you the port and the sim in a few months no matter what happens with them, if you are interested.
Very, very much appreciated. I question my sim/port a little still but you looked at this and that is best information I have for now.
Best,
Jim