Live strategy different than simulated strategy of the same date

[yuvaltaylor]

@philjoe

Second problem: not quite following, is it because I use “Price” sometimes and then I use “Close(0)” in other formulas (I assumed they were interchangeable)? Or are you saying that the simulation run on Tuesday uses the Monday close as Close(0) instead of last Friday like it should?

I’m really not sure what happened, but it does look like one simulation was ranking based on one set of prices and the other was ranking based on a different set of prices. It would take some more investigation to figure out which simulation was ranking based on which set of prices. I do feel reasonably certain that if the starting date of the simulations had been earlier than 12/2 there would have been no difference between them. This particular issue should not be a source of any underperformance of a live strategy. It seems to be a problem with using a very recent starting date for a simulation.

[philjoe]

Hmmm ok so I have uncovered some weird bugs but nothing that is related to my issue, other than maybe the industry thing.

[yuvaltaylor]

That node has a substantial weight (1.7%) and industry momentum is a very important factor. If it doesn’t work for live strategies but does work for simulations, that’s going to explain a great deal of the difference. I would suggest that until we fix the bug you replace that node with a normal formula node and use your formula inside the Aggregate command: Aggregate("[formula]",#industry).

[yuvaltaylor]

We have now fixed the first of the three issues (see quote from my earlier post below). We should no longer have fleeting NAs for cash-flow statement items immediately after filing.

@yuvaltaylor

First, we have been getting some “phantom” NAs in cash-flow-statement data immediately after filings, and our fallback mechanisms failed because of the way the data was being reported. We wrote to FactSet to see if they could fix this, and they couldn’t, so we are going to be implementing a better fallback for these instances. These NAs would show up and affect live strategies but would not be in simulations since the NA data is generally overwritten. This explains why simulations would perform a little better than live strategies. In general, there’s not much we can do about the fact that FactSet silently corrects NAs and mistakes after the fact.

[philjoe]

Ok cool, and the aggregate formula seems to work

[philjoe]

Better this week, although it still needs work.

Live strategy: experiment - live micro US - Copy
Sim strategy: experiment - sim

39 / 50 holdings were the same, which is an improvement on 28 / 50. So the new industry momentum formula seems to help, and the cash flow changes. But there is still something very wrong…

[yuvaltaylor]

I don’t see the simulation in your account, so I can’t compare them.

[philjoe]

sorry should be there now, called “experiment - sim”

[yuvaltaylor]

See the simulation experiment - sim - copy. It’s the same as experiment - sim except the starting date is 12/10 instead of 12/9. The transactions match the live strategy very closely now. The simulation you ran was ranking based on a different date than the live strategy.