Marco and others,
When using the simulator over the weekend I have been getting erratic results with sharpe ratios changing from .2 to 1.3 on the same system run the same day. The problems have been occurring when I use the betafunc and/or the maxcorrel functions on ETF. Please correct this and/or explain why this is happening as soon as possible.
Thank you for the help.
The sharpe is calculated after a sim is run so it is independent of the rules you use. Do the equity curves look different ? Do you have sims we cal look at? Can you reproduce it at will?
Thanks
Marco,
Thank you for your quick response.
Please look at momentum low beta 2B and momentum low beta 2B test in the low beta simulation category. Both were run with identical start and stop dates between 1130 am and 1 pm today as well as identical parameters. The equity curves are markedly different.
I can’t reproduce. Re-running the ‘test’ is the same as the other.
I am continuing to get erratic results from the simulator.
After your last message I ran the initial sim (momentum low beta 2B) and the equity curve was restored back to its initial state. However I ran the momentum low beta 2B sim again tonight and obtained a third equity curve, which is different that either of the previous two equity curves. All of these simulations were performed on the same date range using identical rules.
I saw this with several additional sims over the weekend. These also used identical rules and dates but had markedly different equity curves. I saved another example which is LB3 and LB3 test.
Please explain why I continue to get markedly different equity curves/ results on the same sim run the same day with the same date range?
Thank you for the help.
I reran both of these sims and labeled them second test for reference. Each of the two sims above now have three very different equity curves/ performance for the same rules and same dates.
Please explain why this variable performance is occurring and correct it as soon as possible.
Thank you for the help.
Anyone else experiencing this? We have not heard of similar issues.
Do you have multiple open browsers? Simulation settings are global, so accessing other systems while setting up a sim could over-write each other. Please let me know if you can reproduce and how.
I did experience something unusual once recently. I went away from my computer for awhile. When I came back I decided to rerun a simulation I was already on the summary page for the sim (or in the buy rules, I’m not sure).
When I paged through the simulation settings I noticed that the sell rules had been deleted and replaced by Rank < 60. I panicked and got out of the sim and then displayed the trading rules. They were all correct.
Another problem I noticed a long time ago that may be related has to do with a timeout setting. There is some timeout that makes it appear as if you are logged out but if you go back to the home page it shows that you are still logged in.
So my theory for the first problem is that there is a timeout causing parameters to be reset - in this case the sell rules reset to the new simulation default.
Hope this helps.
Steve
I also noticed that there were a lot of strange things going on on Sunday with simulations… bad portal errors, taking a long time to store the simulation run to the database and it appears that someone changed the number of transactions that can be saved.
I have been accessing this site via Chrome as a browser with the Apple IOS 6 open in the background for months without incident. This has only occurred with the simulations using the betafunc and/or maxcorrel functions which could be a coincidence rather than causative if these functions are working for others. My other simulations have been giving me consistent results. I will continue to run these simulations with erratic results at random times to determine whether the performance stabilizes on one equity curve /results or the erratic performance continues.
I just ran one of the previously mentioned sims and labelled them as third and fourth test. Both of these sims were run with the same dates and rules several minutes apart and have completely different equity curves/ results. These equity curves and the trades are very different than the initial sim as well as the sims labelled second and third test which were run over the weekend when I experienced the same problem.
Why do I continue to get such erratic results from the simulator?
Might be caused by BetaFunc used inside a ranking system. I’ll investigate. Thanks
I’m rerunning your sims but I always get the same results. Do you have multiple windows open ?
Early this am when I had the inconsistent results I had 4 google chrome windows open as well as 5 apple iOS 6 windows open. I have run sims for months with this number of windows open without incident. Why would this suddenly begin to affect the consistency of the sims performance? And why is it only affecting certain sims while the others have consistent performance? Have you seen multiple open windows have this affect?
Thank you for your help.
I found the problem.
I’m pretty sure is caused by using the new function BetaFunc inside a ranking system in a backtest. The date for the SP500 used to calculate Beta was not being advanced if a server was handling multiple dates. Only the first rank was correct. We have several servers handling requests, so this could cause the erratic behavior.
Very sorry about this. BetaFunc is a relatively new function and we had to do some optimizations to make run fast.
New sim servers will be released tonight.
Thank You
Marco,
Thank you very much for resolving this problem. I look forward to again having consistent results from the simulator.
Looks like the release of the fix did not quite get through to all the servers. There were two still running the old version, which creates a strange situation since all servers help out in a simulation. I’ll keep a close eye tonight to make sure all get upgraded and report any other problems. Sorry about that.