Marc,
Just want to say I agree with you.
Not that I have succeeded but using bootstrapping, smaller p-values etc are all attempts to avoid some of the pitfalls at least—e.g., normality assumption and excessive data mining leading to “alpha inflation.”
Still I may not have adequately addressed other issues—as I have said—like i.i.d. I do not try to hide this.
I do not want to give details. But I have found things with Excel spreadsheets that have proved statistically significant. Things that cannot be tested using P123. Things that can be shown to have made me money using simple accounting—so for anyway. Things that can be found in the literature but that I found only after becoming aware of them by looking at my trades in a spreadsheet and doing a search on the findings in the literature.
Furthermore, even bad statistics can tell you things. Maybe a regression cannot be done due to outliers and lack of linearity for example. But it can still be good to ask: “What is that extreme outlier doing there?”
The so called “French Paradox” that found a low incidence of heart disease in French people is an example. It is speculated that the French people have less heart disease—and are extreme outliers despite other risk factors—because they drink wine. I am not so sure that regression was really legitimate and maybe should have been thrown out of that paper. But is has the undeniable benefit of giving me an excuse to drink some wine now and again;-)
We are programmed to see patterns that just are not there–as Aronson points out. Any statistics that identifies these false patterns is good.
But we also miss pretty extreme patterns if we do not look for them. It can be shown that we humans regularly miss things with up to .70 correlation unless we look for those correlations.
Don’t even get me started on the fact that much of what is published and is supposed to work does not work in some ports while it works just fine in others. Or sometimes it is so well understood and published that following the herd is actually harmful. Not trying to sort this out is just laziness. Without a doubt an advanced degree in finance is the best way to get the best answers. But will that answer all of my questions on short-term and long-term momentum after I am done?
So I will not give up on the statistics—I know you are not recommending that I do.
-Jim