I took a port that I have used since 8/13. I changed the slippage to zero and made the ranking system Random. I ran this sim 100 times (starting 8/13). My real port beat the random sim 93 times.
Pretty good huh?
Maybe, but not statistically significant. To be significant (p <= 0.05) it should have beat the random sim 95 times or more
[/quote]93 out of 100 ain’t too shabby.
95% significance is an arbitrary number. If medicine worked 93/100 times it would be great!
Robert,
Good point. I don’t use a lot of buy or sell rules. I think your solution is a good one: replace it with a random buy rule. BUT I WOULD MAKE THE RANKING SYSTEM RANDOM TOO! Remember your null hypothesis is that none of what you are doing in your port gives a result that is different from random. That is to say I would make everything in the sims random. I would not change anything in the port.
You are just trying to find out if there is something you are doing in your port that really works. You are not trying to sort out—exactly—what it is that is working.
Good point. Thanks!
Remember your random sample is not supposed to be significant. It is your port that is supposed to have such stellar returns that nothing else can explain it except the value of the rules (and rank) in your port.
If your point is that you should technically run more than 100 sample sims that may be right too. The only thing that made me hesitant to post this is that could add up to a lot of resources for P123. 100 put me in the ball-park on my port. I did not need an exact number.
Lets save the larger trials for those ports we are about to put real money into.
I think I am addressing your point: but I would not be surprised if I am missing it.
@Chaim. Yea. If I were cherry-picking my ports and testing the best one I would actually be worried. I am not too worried but something to keep in mind, I think. Also the sim results (over Max period) give some weight one way or another. If I were trying to sell this as a SmartAlpha (or whatever we call it now) I would absolutely add no qualifiers or excuses. There really are some ports out there that look good just by chance and that probably includes a few of mine.
If I were a real statistician (I haven’t even played one on TV) I would have looked at the “power” of my little study to show significance. I have done this and always wondered why I bothered. But it has a place, I think. There is a difference between not showing that something is significant and proving it has no value.