robust linear regression

Research
21

David,
What is, other the “Block” Bootstrapping?

And isn’t there still an assumption that the distribution is symmetrical which seems to be an unfortunate assumption of most nonparametric tests?

But I am seeing the advantages of this test and am learning.

I’m not trying to be negative. I have used some nonparametric tests on things that are not symmetrical: any criticism would have to be directed towards myself first. And almost certainly better than using a parametric test on something that is neither normally distributed nor symmetrical.

-Jim

22

I would say so. But it depends on what you’re looking for. If you’re looking for the slope that will best predict future data, then I think Theil Sen is far more robust than OLS, and that it does indeed increase the sample size. If, however, you’re looking for the intercept, which is what I’m after, Theil Sen’s only advantage over OLS is its relative insensitivity to outliers. LAD may be better. The trouble with Theil Sen estimation for intercepts is as follows. Theil Sen approximates the line closest to the points in a diagonal direction (i.e. the distance perpendicular to the line). Intercept is a purely vertical measure. So you want the line that’s closest to the points in a vertical direction.

23

Jim,

I just meant to differentiate statistical bootstrapping from its other meanings in finance (e.g., such as starting a business by one’s “boot straps”; interpolating points on a curve; etc).

Perhaps… but in order to assume non-symmetry, one must incorporate additional parameters. More parameters leads to calibration. Calibration leads to increasing chances of over-fitting. Over-fitting leads to the dark side.

In my opinion: A thing which is descriptive and which is also not highly calibrated has better likelihoods of being both predictive and prescriptive than a thing which is more highly descriptive but also more highly calibrated.

Leading to a good next question, I think: has anyone had success in using non-parametric regressions? How would one implement such a regression in P123? As an aside, all of the P123 ranking systems are by default based on non-parametric ranks.

Yuval,

Thank for that. Makes sense. I am interested in learning if you ever do find a “good” Theil-Sen intercept.

//dpa

24

Thanks David and Yuval.

I really do not know much about this test and this is making me want to learn more!!!

Honestly, I did not even know it was a nonparametric test at the onset of this thread. So any comments I made specific to this test should be taken with a grain of salt (at best). This assumption of symmetry has frustrated me with other nonparametric tests in the past, however.

Most of the times that I want to use a nonparametric test it is because it is a normal–but skewed—distribution (e.g., outliers giving a large tail on one side). What have I gained in using a nonparametric test that has symmetry as an assumption? Maybe I have actually gained a lot. I just have to wonder until I get a clear answer on this.

Using non stationary samples has been a weakness of mine (and probably remains so). So my question of what is not affected by this assumption remains a question of mine: perhaps out of frustration.

I am all for any methods we can use to go beyond looking at the pretty graphs: which I must admit is a pretty good start.

I remain committed to finding the mean of my slippage with no concern, whatsoever, about the median (or the mode).

Again, thank you.

-Jim

25

With the intent of making no points this is great question. Really great question!

For a moment I thought I was doing better when I stopped making the assumption that the rank performance test was linear and also using a test for nominal (or ordinal) independent variables for the percentiles. Specifically the ANOVA test gets around these assumptions. But I had problems with the assumption of equal variances.

I do think that the central limit theorem does, often, take care of the normality assumption. Do you remember my post on bootstrapping the daily returns: Bootstrapping over 4,000 daily returns gave a histogram that looked pretty normal to me. The purpose was to just get a picture of what the central limit theorem can do with stock market return data. Of course, they do this in every statistics class with other types of data. Stock market return data does not seem special to me with regards to the central limit theorem—as long as it is stationary.

I would be interested in any thought on this. But for now I think it is possible to do some statistics that strictly fit all the necessary assumptions. I just use those when I can and often use statistics that do not fit those assumptions. Or I just look at the regularly spaced upwardly progressing stair-steps of the rank performance test and know it has to mean something–with no proof. As I said, I need look no further than what I have done to find bad assumptions and outright mistakes.

-Jim

26

I believe that mapping the monthly returns of a simulation against the monthly returns of a benchmark is non-parametric. The same can be said, as David pointed out, of ranking systems. And perhaps in both cases non-linear, non-parametric regression would be appropriate.

However, I’d rather keep it simple. My aim is high alpha (in a recessionary environment, high alpha coupled with low beta), and while playing with non-parametric regression might be fun (and appropriate to my data set), I don’t think I’m going to go there soon.

Below is the typical data I’m working with.

OLS vs LAD big.png
OLS vs LAD big.png797×586 40.5 KB

27

xxx

28

Yuval,

I’m not sure I have any suggestions on how to get a line to fit that data.

It looks like the skew, outliers and lack of linearity are driving much of your excess returns. I would not know how to remove that (or find a statistic that minimizes their impact) and have data that represents the same sim.

Probably not what you are looking for but I think a paired t-test would be totally appropriate: the central limit theorem is in full effect here with that many data points. Pairing the data will remove any concerns about your data being stationary (David’s point). This is effectively detrending. There is no assumption of linearity or of equal variances and I am sure you will find high statistical significance. And you just make 2 columns on the Excel spreadsheet, click Data Analysis and find “t-Test: Paired Two Sample for Means.” Googling our Youtube will fill in any details. There are better tests when they can be used. But I believe this one test that that satisfies all of the usual assumptions in financial statistics for your data. If anyone does criticized this tell them that you don’t want to see them use even a Sharpe Ratio as your test is similar but more likely to be stationary (which could even be tested).

Bootstrap it if you want. It is not necessary but it looks smart.

There are so many things I have tried to use a linear regressions on: like rank performance tests as you and David both mention. I keep trying. I think it just does not work for that. Which is actually David’s point isn’t it? P123 is not using any regressions on their rank performance tests (or any nonparametric statistics that I have seen). They are using the best descriptive statistics that they can: the buckets. They do not even call it a histogram (because it isn’t).

Hats off to Marc, Marco and the entire P123 staff for not taking others down the same path that I have tried to take on statistical tests for the rank performance. And for anyone thinking I am critical of any member, it will not be hard to find quotes of me suggesting that P123 should go down that path. There is a limit to how much I can fool myself on this. Bring on the quotes: it will be humorous (if not sad). And if you don’t want to limit the quotes to statistics, per se, then all that stuff about replacing ranks with z-scores and implementing multivariate linear regressions should be a real hoot!!! Even though it does kind of work. Sort of. If everything would just be linear like it should be………

But I do not think we have to abandon all of our statistics. And those who want to laugh at me should just keep laughing while I head to the bank today. I just hope I can keep you entertained! Yuval, you are ahead of me on extending the use of linear regressions. I will go back and see if I can use Theil Sen for some of my data. Linear regression is a great tool—in all of its forms—when it can be used.

I like what you are doing and I believe it will work for you even if you happen to find that you have to revise a few things along the way.

Best regards,

-Jim

29

[quote]
I did a correlation study to see which better correlates with OOS results, alpha calculated by OLS methods or calculated by Theil Sen estimation, and OLS won.
[/quote]What were the correlations?

30

I’ve now done even more correlation studies, and I’ve come to the conclusion that OLS alpha probably still provides the best correlation between an 8-year 100-stock simulation and the subsequent 3-year performance of a 20-stock portfolio, but only by a whisker. LAD alpha is good, CAGR is good, median and mean excess monthly returns are both good, Theil Sen estimation is good. While OLS alpha usually leads the pack in my correlation tests, the others are off by less then 0.03 (the difference between the correlation coefficients).

In my latest tests, I first tested 40 radically different strategies–different rankings, different universes, different holding times. The results (the average of plain correlation and rank correlation) were OLS alpha 0.781, CAGR 0.775, LAD alpha 0.767, mean excess returns 0.763, median excess returns 0.758, and alpha-sigma ratio (OLS alpha divided by s.d.) 0.725. I then tried testing 30 very similar strategies, the kind of strategies I use in my everyday trading, to see if one method was better at catching the little differences between them. Now the results were almost the reverse: median excess returns 0.580, LAD alpha 0.573, mean excess returns 0.570, CAGR 0.565, OLS alpha 0.557, and alpha to sigma 0.419. But with the exception of alpha-sigma, the correlations are all so close that I don’t think one can make a definitive judgment of which is the best. As for Theil-Sen estimation, it’s very close to median excess returns, and it’s extremely cumbersome, so I’m just guessing it’s in the same range. I’d rather not use it because it’s so difficult to manage.

This brings me to Jim’s suggestion of a paired t-test. I may be wrong, but from what I’ve read, the number that I’d be looking for with a paired t-test is Cohen’s d, which is basically the mean excess return divided by the standard deviation of the mean excess return, because that’s the measure of how powerful the effect is. Now whenever I divide anything by the standard deviation, I end up in trouble. My correlation coefficient drops by at least 0.05, and sometimes as much as 0.2. For a while I was really taken with the alpha-sigma ratio (partially because I invented it myself), but correlation just isn’t as high as plain old alpha. And you’ve probably already heard about my gripes with the Sharpe ratio.

Now if I’m wrong about the t-test, I’d be happy to give it another try.

31

I should add that one of the reasons my correlations are so close is that the numbers themselves are so close. In my 40-very-different-systems correlation study there was a 0.99 correlation between mean monthly returns and CAGR and between OLS alpha and CAGR, and a 0.98 correlation between OLS alpha and mean monthly returns; in my 30-similar-systems study, the same is true but there’s also a 0.98 correlation between median excess returns and LAD alpha. The least correlative of the measures are LAD alpha and mean excess returns, FWIW, with a correlation of 0.93.

I should also add that all the strategies I tested were primarily fundamentals-based, long only, with no hedging or market timing. In other words, they were all relatively high-beta strategies, all well correlated with the market. No doubt the results would be quite different for low or negative beta (long-short, hedged, and/or market-timing) strategies.