I have begun to dissect and rebuild my models one input at a time.
Although it probably goes without saying, I am focusing all of my attention right now on understanding differences between the old CompuStat using preliminary data vs. the new Beta CompuStat w/preliminary data option.
To make things easy, I thought, I would start with something simple: COUNTRY(“XXX”)=FALSE
I don’t believe the COUNTRY function is working on the Beta site.
The good and bad news about this discovery, if confirmed, is that I started by investigating just one p123 function and it does not appear to be working properly on the Beta site.
COUNTRY is not that important a function in and of itself but perhaps it will point to other malfunctions that cumulatively narrow the differences between existing and the Beta site results?
Hugh, In my previous post, I was talking exclusively about changes to the beta server, not the production server.
The code for preliminary fallbacks has been refactored, but the main difference should be the segregation of annual preliminaries. If you find a specific factor or function that’s noticeably different between production and beta (Compustat & Use Prelim), we can further discuss that in this thread.
Regarding yesterday’s fixes, if you’re still seeing drastic differences, we’ll need to look further into what’s causing the change.
Intentional changes have not been formally tracked, but they have been mentioned in this thread.
None of the changes here have been put into production yet; it’s going to remain in beta until we’re satisfied.
Having said all that, rest assured that we’re not aware of anything being catastrophically broken in production.
I will create an issue for Country and will look into it shortly. Thanks for finding it.
Georg, this issue has little to do with the beta server. I have already answered regarding the ‘undefined’ behavior of systems defined like this in your Different results for same Buy rule thread. Consider the scenario where 1451475 is holding SPY. In that case, your rule would evaluate to true for both SPY and IEF, so it would end up choosing the best ranked ETF of the two, which in your case is undefined. Due to the change of an internal structure, what you’re actually seeing is a more stable ‘undefined’ rank. You should be using FSum or FCount to ensure that your market-timing condition is the same for all ETFs evaluated.
Could you please choose several companies and affected variables, and then present before/after numbers which have changed because of the corrected method by which p123 is handling preliminary data on the Beta site?
Also, is p123 planning to make this change to the production site? If so, when?
If this is the only change p123 intends to make to the production site it may be easier to determine if this change involving preliminary data is causing the differences I’m seeing in Beta site results. If p123 is not planning to make changes to the production site, or if the changes include more items than just the preliminary data – not sure this is feasible but… - if p123 could create a third site which would essentially be the production site with only changes to the preliminary data it would be easy to determine the extent to which this (vs. this plus additional issues that may exist like COUNTRY) are the source of the differences.
I looked at UNIVERSE after looking at COUNTRY. I believe this function is working correctly on the Beta site but it’s hard from the outside to be 100% sure because it’s impossible to feel certain about my “control” on the Beta site given other possible issues.
Finally, I think it should be mandatory for p123 to track (and even publish) its changes. P123 has a dedicated group of highly interested and sympathetic users. Many, as demonstrated in the Forum, are very experienced and expert across a wide range of disciplines. Everybody understands it’s easy for small hidden logic errors to create nightmares when programming. Why not provide the detail so that p123’s subscribers can get more involved? Testing one input at a time, as I’ve started to do… It’s really hard to track down these issues from the outside.
For Ticker(“DHT”) if I look at the calculated ROI% on the Production Server and on the Beta Server using Prelim I see fairly big differences in the calculated values.
Production Server
ROI%5YAvg = 3.88
ROI%TTM = 6.07
ROI%PTM = -0.66
Is this due to the way preliminary data is being handled? Seems odd that it would have a larger effect on the 5YAvg then on the TTM and PTM calculations.
Aaron,
Eliminating ticker IEF and using VTI instead of SPY, and using FSum, then buy rule becomes:
Eval(FSum(“$port1”,#All)>0,ticker(“VTI”),ticker(“0”)) // $port1=Ticker(“SPY”)&Portfolio(1451475)
This would only evaluate to true when port 1451475 is holding SPY. So it can only select VTI when true and no ranking is involved.
The backtest from 2000 produces AR= 12.33% and Overall Winners= (36/41) 87.80% on existing P123 platform.
On the new platform with Compustat with prelim:
1st run: AR= 11.46% and Overall Winners= (24/27) 88.89%
2nd run: AR= 5.62% and Overall Winners= (1/1) 100.00%
On the new platform with FactSet with prelim:
1st run: AR= 10.31% and Overall Winners= (10/13) 76.92%
2nd run: AR= 5.65% and Overall Winners= (2/2) 100.00%
So new platform is not stable for rules with “Portfolio” function. That’s all one can conclude.
Here is another buy rule which is not affected by ranking (VTI-IEF):
FSum(“1*$port1”,#All)>0 & ticker(“vti”) | FSum(“1*$port1”,#All)<1 & ticker(“ief”) // $port1=Ticker(“SPY”)&Portfolio(1451475)
The backtest from 2000 produces AR= 17.76% and Overall Winners= (64/82) 78.05% on existing P123 platform.
On the new platform with Compustat with prelim:
1st run: AR= 5.38% and Overall Winners= (6/9) 66.67%
2nd run: different to above, 3rd run is different again, and so on.
On the new platform with FactSet with prelim:
1st run: AR= 5.62% and Overall Winners= (1/1) 100.00%
2nd run: AR= 6.26% and Overall Winners= (3/3) 100.00%
We’ve found some issues with the way a number of annual items were being handled and have put in a fix for those. Specifically, annuals were being pulled from interim statements. This impacted, among others, NoIncP4YN2Y, EPSEst, EPSActual, EPSSurprise, SalesEst, SalesActual, SalesSurprise, QtrComplete, DivPS5YAvg, and PeriodDateA. I believe this is now fixed.
See the Trello board for the status of the other bugs and fixes, and thanks for all your reporting.
The Trello Bug log suggests that issues with WeeksIntoQ have been resolved.
But when I remove WeeksIntoQ as a variable, my results in the Beta CompuStat w/preliminary data are improved and much closer - not the same but also not showing the heart stopping differences I have otherwise been seeing - to what I get on the existing website with or without WeeksIntoQ.
There is no reason I can think of that removing WeeksIntoQ should improve my results. It’s not a major contributor when I include it but excluding it should only make results worse.
Therefore I would guess that WeeksIntoQ OR some aspect of the Beta w/prelim database upon which WeeksIntoQ is acting must not be right.
Separately, could there be platform issues going back and forth between the existing site and the Beta site? I notice when I run something on one or the other that settings are retained on the other. In other words, if I remove a rule using WeeksIntoQ on the Beta site the rule remains removed on the existing site. I suppose this is as things should be but can we be sure the cache is cleared? Maybe it would be better to make copies and run each copy exclusively on the existing or Beta site?
By the way, is it possible to run simulations simultaneously on the existing and Beta site? That would sure speed things up.
Finally, at the risk of repeating myself, if you could provide some concrete “before and after” examples related to several companies and variables affected by the annual preliminary data-related fix, it would sure be appreciated. This has been discussed in a number of ways. Speaking for myself, I’m really not clear about how the change plays out, what it means, and/or what I might do to adjust, if that’s even a possibility.
I tested several portfolios, and performance / drawdown / Sharpe Ratio are either same or better on FactSet. Only a few portfolios had a similar performance with slightly lower Share Ratio, which is still ok. But I have 2 sims (and respective Live Portfolios) with significant worse performance with FactSet. Could you please take a look to determine what is driving the worse performance in FactSet?
Both simulations exclude prelim. Both 10 year simulation.
I’m afraid not. What we’re trying to do is to isolate data points that are different and try to figure out why they’re different. It would take us many hours to look into simulations with lots of different rules and ranking systems to try to figure out what’s causing differences when so many data points are different to begin with.
What you should do is analyze specific instances. Try to figure out why your results are so different by looking at the particular stocks your systems are picking. Dig deep to find out what the differences are. If they’re NOT related to fundamentals, then you may have found a bug, and you should let us know immediately. If they ARE related to fundamentals, is there a way you can make the two different numbers make sense? Or are the two different numbers so far apart that one of them HAS to be wrong? Can you compare the numbers with the actual financial statements to see what FactSet and Compustat are doing differently? Or do you think we’re using the wrong datapoint? These are the kinds of investigations that can really help us.
The Trello site noticed issues with WeeksToQ and WeeksToY. Issues with WeeksIntoQ have not yet been resolved. WeeksIntoQ is basically completely broken on the beta site right now. Thank you for calling this to our attention.
Just to update you–a few more bugs have been fixed, though there are still more. You can see most of them on the Trello board. In addition, KEEPNA on the beta site was sometimes returning 0 instead of NA. You shouldn’t see that happening any more.
I have looked into this a little deeper to try and understand the differences. I don’t know the exact details of the ROI calculation, but assume it is similar to the following
ROI%TTM = IncAftTaxTTM / InvCapTTM
where
InvCapTTM = Avg(EqTotQ+DbtLTQ,EqTotPYQ+DbtLTPYQ)
If I compare this estimated ROI calculation to the actual ROI calculation on the Production Server I get fairly similar results. For example, for ticker “DHT” on the date 10/15/2013 I get the following
So something is being handled quite a bit differently in the ROI%TTM calculation underneath the surface on the beta test server. Curious if you guys have an explanation. Here is the link to the screen I am using to test the differences (https://beta.portfolio123.com/app/screen/summary/240746?st=1&mt=1), not sure if that link will work or not.
These ports are private, I posted the link case it helped p123 staff. Yuval posted the approach that I should be using about this, so I’ll try to do that, first to find out if it’s fundamental related or not, and then why the discrepancy on the data for the stock.