In this thread you stated: [quote]
We do not snapshot any data any more, so any late updates can change prior results
[/quote].
I know that in the past with Reuters’ data the updates for some of the small and micro caps would not be analyzed by Reuters for up to 2 or 3 weeks after the announcements were made by the companies. That made the weekend snapshots of the data fix in time this 2 to 3 weeks delay.
Since you are no longer saving snapshots of the data, is Compustat also analyzing the data from small and micro caps with a significant delay, but their updates to P123 adjust the data date to the date of the public statements and change the existing P123 data from 2 or 3 weeks back?
Yes, compustat has similar delays since they both enter by hand. I was told that Compustat turn-around should improve since some of CapIQ resources were starting to be used.
Compustat marks the effective date of every data item when they insert it into their database. If data is updated daily , which we do, then new data should not have an effective date older than 1 day (except for older fixes which are very uncommon). I confirmed this with them.
I’m still tracking down the problem where some weekend data changes when re-run a week later. I have some specific samples that I’m working on. I should get to the cause soon.
Compustat’s clarifications I got re. their processes would invalidate the statement I made. I want to make sure it isn’t a problem in our end before I send them the sample cases.
Good to know that you want to get to the bottom of this issue.
Since you are not making snapshots anymore, would it be somehow possible to mark certain data read only just to make 100% sure nothing changes that shouldn’t change?
Another way could be to detect effective dates older > 1 day which only can be propagated to your database when accepted manually by one of the P123 team-members.
My 2 cents… I hope you will keep us posted on this.
There are line items that show update times of Saturday around 4PM. This is past our Saturday update of around 10AM, but they should be picked up by the Sunday update (which is why we did it), but they did not get picked up.
So when you run a screen all the way up to Monday night you are doing so without the line items from Saturday 4PM. When they get loaded on Modnay night (the next update) the results can change.
Because these items have a special flag from Compustat, and after checking our logic, it may not be our fault. I have started an investigation with them
I have been bothered by the changes of historical transactions. Really appreciated that you keep this issue on mind and find the potential causes. It means a lot to P123 members.
What this means is that we need to do another update early Monday morning, and the best time to rebalance your portfolios for the weekend is sometime Monday morning before market opens.
PLease note that for live, running ports, it is in essence a bit of a lag in the data. In the big scheme it probably isn’t a big deal. However, for P123 , it creates this more noticeable effect when you re-run something in the past, especially since ranking is all relative to each other. What this means for historical backtests is that results should “settle” if they are a week old.
Also, Compustat transactional updates are done throughout the day: 8AM, 4PM, midnight Central . Since we run our updates around 10PM and 3AM we always miss the 8AM update. However it would be very hard to use anything from the 8AM update since it’s only 2.5 hours before market opens. By the time we download, refresh our staging server, then the sim servers, 2 hours have gone by.
Marco - does this pertain to certain data fields? (Before there was some discussion about certain fields not being available until Sunday.) Or does it pertain to microcap stocks or any stock any field?
Another thought - any chance of shifting the rebalance to Monday night?
Steve, you are correct. There are a lot of moving parts , different companies providing feeds, etc (CapitalIQ, Compustat, S&P) each at different times. Ex: there is data being update on Sunday around noon: the estimate up/dn revisions. We decided on the schedules based on times that were given to us, but I think those times were correct for the data that we get, but not for the product that we use to download it.
In any case we’ll get to the bottom of it soon. And I think you are right. The auto-rebalances should most likely run on Monday early morning. Not sure about R2G since rebalance emails would arrive basically a few hours before market open.And, honestly, I don’t think that a few missing updates from the weekend will be material.
You can always do a manual rebalance on Saturday as usual. We will make sure to put a note saying that some weekend data may be missing.
I misread. you mentioned Monday night. But that’s not going to change anything. All the data comes in around 3AM, so the rebalances would still have to be done a few hours before market open.
In other words, the best time to rebalance with the freshest data will always be before market opens.
There are advantages to changing to Tuesday morning, or wednesday, etc. We’d be different than everybody else