I started two live strategies on Aug 22 and I’ve noticed that if I re-run the exact same strategy as a simulation, starting Aug 22, there are differences in the stock rankings and performance.
The performance differences are consistently worse for the live strategy.
How can there be a difference between the two strategies? Does the database change after the fact?
EDIT: I just checked and neither the universe nor the ranking system had been updated since the day I created them, suggesting there is a look ahead bias in the data somehow.
Hi Phil, they are not the same: live one starts off 95% invested, the sim is 100% invested. Live one is also 97% invested the last few weeks. It’s very hard to determine if the cause is in fact a bias with those differences. Do you know why live one is not 100% invested all the time?
That makes it worse since the market is down over the past month. The reason is that sometimes there is not enough liquidity to buy all the shares I need.
The problem is not how much is invested, its much worse. The live strategy bought 5 stocks that the simulated did not. And the simulated strategy bought 9 stocks that the live strategy did not. Out of 100 stocks that’s a 14% difference…
The only way that could happen is if there is a difference in rankings from the database on the day of the transactions vs. today, ie. look ahead bias.
I found the cause of one difference: IBEX:USA. It’s domicile is now Bermuda so it no longer passes your rule Country(“USA”). FactSet must have updated and we do not keep track of changes in domicile. My suggestion is to use a broader filter that includes all tax havens like Country(“USA BER CYM BHS VGB PAN”)
We also launched Europe on that week. There might have been some other side effects to North America. I’ll have to get back to you .
OK that makes sense for IBEX but still 13 left… none of the other rules in my universe can be changed retroactively though ie. mediandailytot, mktcap, close price, etc?
One subtle change that we had to do with the Europe launch is to standardize the Beta factor to use holidays so it would work regardless of the benchmark used, and when the universe has stocks from different countries (with different holidays).
This change would cause slight differences in your ranks since you use Beta3Y. We can add a way to calculate Beta using bars (trading days only) to see if that’s the cause of the differences.
Stocks which Live bought, but Sim did not:
IBEX (solved)
PSTX
NOVN
GIC
EBF
Stocks which Sim bought, but Live did not:
HHS
RYI
SXC
USLM
EIGR
VYGR
TTSH
AVD
FMTX
I guess it could be Beta but those differences would be so small I doubt it could cause such a big difference in final output… the Beta3Y feature is relatively low weight and the beta calculation over 3 years would barely change if its just a few holidays difference.
We’ve been over this in a few other threads, but essentially P123’s data is not completely stationary. We do our best to make it point-in-time, but tiny changes do creep in. The two primary reasons are: a) we’re always fixing bugs; b) FactSet makes some data corrections/changes, both to analyst and fundamental data. Other sources of differences could be user-caused: a change in universe rules, a change in a custom formula, a change in a ranking system. That’s clearly not the case here, but I did want to point it out. When you’re ranking a few stocks out of several thousand, these changes can have a “butterfly effect,” which tends to be rather random in nature. The fact that the performance differences are worse for the live strategy may be coincidental. Again, this was the subject of an earlier thread, and there the performance differences were positive in some cases and negative in others.
If you really want to get at the heart of what’s causing these differences, I suggest the following. On the rebalance date, run the “ranks” function of your ranking system on your universe. It should match the simulation’s recommendations, adjusting for buy and sell rules. Make sure to include composite and factor ranks. Download the CSV file and save it.To be really careful, also run your universe and download and save that. A month later, run the simulation and note down differences. Then run the ranking system and universe again with the as-of-date that matches the rebalancing in the simulation. Download the CSV files and see how they differ from the ones that you downloaded a month ago. That will pinpoint exactly what changed for each stock in question.
Phil , I copied your ranking system and used a slightly different Beta3Y “BetaFunc(5, 152, 70)” and there are these kind of differences everywhere. It could last for a couple of rows but sometimes more like this example. Differences happen every 10 rows or so.
So I think Beta changes introduced with the Europe launch combined with your sector limits could have easily caused the 5% differences you saw
That wouldn’t have caused the changes since I buy the ~top 10 percentile. It has to be something to kick the stock out of the top 10 percentiles and into 80-90… a change from 98.49 to 98.33 won’t do that.
I did not examine all the way down to around the 90 percentile. There may be bigger differences.
The way a small change like 98.49 to 98.33 will cause a difference is because you have SecCount < 25. When you are buying the 25th position is where a slight difference matters. Only Healthcare seems to be fully allocated to 25 names on start , so FMTX, VYGR & EIGR would be affected.
To fully understand what happened will be impossible now because the ranks in the transaction list is missing in the Live strategy . Also the transaction order seem to be different than ranks. You must not be accepting the recommendations and just downloading them instead? Are you then importing transactions from your broker? Or adding them manually !? This would explain why you do not accept recommendations: it would cause duplicates (we could get around this with a reconcile function).
In any event, the only way to know for sure is to do something similar to what Yuval suggest. Start a new live strategy; preferably on Monday, so today. Just accept recommendations, or set it to fully automatic. Download complete rank data (only necessary for the first rebalance). A month from now verify with a sim.
Yes I am downloading them, inputting trades and then importing the transactions from the broker back into P123. I didn’t think this would cause a problem? What is the correct way of dealing with this?
Also yes I will start with Yuval’s strategy to reconcile, must start on a Monday also.
It’s not a problem. You are just losing the Rank score for the transaction, which is a nice thing to keep around.
Another thing we can do is add the Rank column to the rebalance download (which is missing right now) and append a date to the downloaded file name, like “MyStrategy_Rec_2022_27_09.csv” .
Ahhh ok I see what you’re saying. Ok that’d be great.
I’ll start a new strategy next monday and save the recommendations and then a month later I’ll re-run a sim and compare the saved reco’s from live with the opening transactions of the sim.
Should not be currency related. Definitely a ranking difference. What time was the live system launched vs what time you run the simulation ? There’s one last Monday AM update that can change values that happens around 4 to 5AM, right around when you posted.
In any case just to be sure I will download the ranks and the raw values used to calculate them today. I will then compare the download with values next week for the same observation date. I’ll use your ranking system & universe. I’ll send you the files too.
Also, the universe indicates that it was changed on 10/17. Was that before or after the live strategy rebalance on 10/17? That could explain the difference.
No that would have been done before I started this live strategy. Not sure the exact time I did it… What kind of changes occur at 4am or 5am that are so consequential if you don’t mind me asking?
Every vendor adds data multiple times a day. By running one last update around 4am we grab as much as possible for Mondays rebalances. However this could be a problem for Europe since the market is already open. In other words a rebalance in the morning Europe could be different than a rebalance in the afternoon.
It’s one of the areas we need to clean up. Maybe change our procedures for Europe (like adding an update around 11PM on Sunday) and also make it clear what’s going on
