Renaissance Technologies

[Jrinne]

So here comes the proof that backtesting does not provide a good indication for OOS performance:

Georg,

Thank you.

Shouldn’t P123 hope this is overfitting? I think they should.

Twice you mentioned AIC which will stop overfitting in its tracks. I leave it to you to try to explain how you would implement this to the staff at P123 and the members—if you think anyone (other than me) will listen to you the third time. I think they should.

AIC is good. I also like LASSO regression. That would mean using that m x n array I keep mentioning to do a LASSO regression. But once the features are selected you could move back to a rank performance optimazation with the selected features. BTW, LASSO or AIC would not use much computer resources.

Result: no more overfitting. Gone, nada, none. Zero, zip overfitting. The big goose egg….

Ultimately I would use something related, myself, but I doubt we will get past AIC or LASSO regression in any discussion.

Anyway, I think P123 should hope the problem is the easily addressed problem of overfitting and give us the tools to end the problem (if that is what the problem is). P123 should hire a consultant if Georg is not understood by P123 staff this third time.

It is NOT rocket science. But you cannot read a post about this somewhere and become an expert either. Georg has obviously studied this to know this as well as he does. But I am going to stick my neck out and say that despite his obvious training and credentials he might not have been a “Rocket Scientist.”

Okay, it helps to be a Rocket Scientist or Engineer like Georg. People should really listen to his ideas about AIC and maybe some of the other methods of addressing overfitting. If overfitting is the cause of the poor Designer Model performance if would probably pay to hire a consultant to make sure Georg is understood.

-Jim

[o806]

I would hazard the following guesses:

1. The old, tried-and-true factors like ROE and P/E have largely been arbitraged away, and stocks are priced in such a way that there are very few values to be found by using those factors. Investors are now so used to pricing stocks by using these metrics that it’s hard to find any advantage in using them.

2. We may be in a regime that resembles the late 1990s, when overpriced high-growth tech stocks ruled the market and S&P slaughtered all other strategies in sight. As we all know, that was not meant to last.

3. While there certainly have been some egregiously over-optimized designer models, given the “base rate” I have established, we cannot blame over-optimization entirely for their overall failure to perform. Some of the “failed” models may actually be very good ones that are just going through a blue streak. After all, every single tried-and-true strategy and every single great investor has had some long periods of underperformance.

Yuval:

In my opinion your first two points are very plausible, and your final one is an absolute certainly.

From a logical and behavioral perspective point 3 has to be true — if a model or group of related models did not have “a blue streak … some long periods of underperformance” then everybody along with all their siblings and cousins would eventually be using the model and it would then stop working for — and this is key — stop working for long enough to get the majority of people to stop using it. At that point it would have a good probability of starting to work again.

Put in other words, if a method is going to work over the long term it has to inflict enough pain, from time to time, along the way to get most people to quit. It is a variation of “no pain, no gain”.

Our data only goes back to 1999, but I’ve heard some who have access to data from the 1990s say that value and small caps did not work very well compared to the growth and larger caps during that period. From that perspective it is not surprising that the typical P123 portfolio (mine included) back tested very well from 2000-2002 (general bear market for the large indexes and range bound for the R2000) and continued to do outstanding well for 2003-2007. Many of our models did well for 2009-2017. Our methods are currently in a 1.5 year period of painful underperformance.

Will this period of underperformance be over in 3 months or 3 more years. I have no idea of the duration but I believer it will need to be long enough and painful enough to get the majority of money to stop using the types of value and small cap models that worked will in the past.

Well that’s my 2 cents.

Brian

[Chipper6]

You’ve probably heard that a picture is worth a thousand words. Here is one of those pictures. In order to make the right decisions, you need to know the facts.

As you can see, this older ranking system did its job even over the most recent twelve tough months. Yet, it did not beat SPY!

Think: How could SPY beat all the buckets? What does this tell us about what happened? What does this tell us about the power of the ranking systems?

Suggestion: Use the correct benchmark. Russell 3000 equal weight is close enough, but an equal weight of the sims universe is even better, especially for targeted universes.

A DM might be working. Or it might not. But what do you learn by comparing it to the SPY?

---

[Jrinne]

I would hazard the following guesses:

1. The old, tried-and-true factors like ROE and P/E have largely been arbitraged away, and stocks are priced in such a way that there are very few values to be found by using those factors. Investors are now so used to pricing stocks by using these metrics that it’s hard to find any advantage in using them.

2. We may be in a regime that resembles the late 1990s, when overpriced high-growth tech stocks ruled the market and S&P slaughtered all other strategies in sight. As we all know, that was not meant to last.

3. While there certainly have been some egregiously over-optimized designer models, given the “base rate” I have established, we cannot blame over-optimization entirely for their overall failure to perform. Some of the “failed” models may actually be very good ones that are just going through a blue streak. After all, every single tried-and-true strategy and every single great investor has had some long periods of underperformance.

Yuval:

In my opinion your first two points are very plausible, and your final one is an absolute certainly.

From a logical and behavioral perspective point 3 has to be true — if a model or group of related models did not have “a blue streak … some long periods of underperformance” then everybody along with all their siblings and cousins would eventually be using the model and it would then stop working for — and this is key — stop working for long enough to get the majority of people to stop using it. At that point it would have a good probability of starting to work again.

Put in other words, if a method is going to work over the long term it has to inflict enough pain, from time to time, along the way to get most people to quit. It is a variation of “no pain, no gain”.

Our data only goes back to 1999, but I’ve heard some who have access to data from the 1990s say that value and small caps did not work very well compared to the growth and larger caps during that period. From that perspective it is not surprising that the typical P123 portfolio (mine included) back tested very well from 2000-2002 (general bear market for the large indexes and range bound for the R2000) and continued to do outstanding well for 2003-2007. Many of our models did well for 2009-2017. Our methods are currently in a 1.5 year period of painful underperformance.

Will this period of underperformance be over in 3 months or 3 more years. I have no idea of the duration but I believer it will need to be long enough and painful enough to get the majority of money to stop using the types of value and small cap models that worked will in the past.

Well that’s my 2 cents.

Brian

Good thoughts. And ultimately it may take a lot of ideas to express our present situation.

The idea that there are a series of events for us to have only 8% of Designer Models beating their benchmark over a 5 year period is a good one. And as Georg said:

the excess 5-year return over SPY= -43.8%

Not wrong to think there may be multiple problems including the above ideas, I would guess. I am not sure it is even possible for just one thing to do this.

I think posts implying there is just one problem–so no worries–are pure speculation.

BTW, Does equal weighting remove the problem of overfitting? I think not. Otherwise, feature selection to remove noise factors would not be the standard of for statistical learning. Look to Georg’s suggestion of AIC for an informed idea about removing noise factor that cause overfitting. You may need to search it for now.

Generally, factors are removed and not given equal weight to prevent overfitting.

Which is not to say overfitting is the only problem. But I see no evidence that it is not one of the factors—not in this thread for sure. I do think P123 should hire a consultant to give us the tools to end overfitting when it is a problem for a Designer.

-Jim

[yuvaltaylor]

So here comes the proof that backtesting does not provide a good indication for OOS performance:

Over the last 5 years the annualized return= 5.0% with a max D/D= -21%.

Over the last 5 years the annualized return of Morningstar’s “Small Value” category is 5.0%. O’Shaughnessy Asset Management’s Small Cap Value Fund’s performance isn’t 5 years old yet, but it has lagged the “Small Value” category by a compounded average of 61 basis points per year over the last three years, so I suspect that if it had been 5 years old, it would have done worse than 5%. I have read a lot about this fund, and it is not over-optimized based on backtests. It is a very well-run and well-designed fund that uses many of the same principles that Portfolio123 has always espoused.

I think we should all accept that the S&P 500 has outperformed everything in sight and that the markets over the last few years have been extremely kind to investors. Considering that capital inflows into public markets have been high recently while inflation has been low, it has been very hard for small-cap value-based strategies (and that’s what most of the designer models are) to outperform a large-cap index.

I also think that if one were to create a strategy that WOULD have outperformed SPY in the last five years, it would have no better chance of excellent out-of-sample returns than a strategy created by the same methods five years ago, which would have likely failed to beat SPY in the last five years. There is no single method to create a strategy with excellent out-of-sample returns. For example, people have been using machine learning to try to game the stock market for just as long as people have been using Portfolio123, and it certainly has potential, as RenTech’s Medallion Fund showed. But that fund held stocks for a day or two and used massive leverage (if you were to deleverage their results and not include their fee-based income, their performance may not have been so impressive, though that’s just a wild guess on my part).

The designer models placed a lot of faith in automation. I know stock pickers who don’t automate their strategy at all who have done very well over the last five years. Does that mean that automation is worse than manual stock picking? Absolutely not. They are two different approaches out of many, and that many includes machine learning and long-short hedging and market timing. In the end, what works? Hard work, experience, constant learning, and sticking to your guns when things go bad. And what else works? Large-cap index funds. At least for now.

[Chipper6]

SPY is dominated by mega caps. (See the attached chart). Mega caps have been outperforming. That might not explain everything, but it does explain some of it.

Why are megacaps now dominating? Here is what I wrote in 2012:

[quote]
…in the long run (over the next 7 years) large (as a class) will almost certainly do better than small.
[/quote]Source
Here we are 7.5 years later and my prediction has come true.

I see the market going in cycles. Mega caps outperformed in the U.S. during the 1960’s, the 1990’s, and the 2010’s. The common denominator is that all three periods were preceded by a long bull market.

Stage 1.
Bear market: Stocks are dumped. Stocks; especially small caps, take huge price hits, sometimes for reasons that have nothing to do with earnings; such as margin calls.

Stage 2.
1.5-3 years. Fundamental investors get to choose between buying fair businesses at great prices or great businesses at fair prices. The long-term investor would prefer great businesses, but in the short term prices for fair companies rebound sharply. That causes small value to outperform.

Stage 3.
People looking at past history stick with small value because of past performance.
Fundamental investors get to choose between fair businesses at fair to good to great prices and great business at good prices.

Stage 4.
The small value juice has been squeezed.
Fundamental investors get to choose between fair businesses at fair prices and great businesses at great prices. This starts a rotation into large growth.

Stage 5.
Momentum is king.
People start jumping off the small value bandwagon. Why own small caps? Past performance shows better returns and lower volatility for mega caps!

Stage 6.
Bubble trouble. As investors become less valuation conscious and more momentum conscious, valuation gets ignored. Nifty-Fifty, Dot-Com, FANG, stocks look like great deals at any price! Until they aren’t.

Stage 7.
Small value stocks do best in the next bear market because they started from depressed prices, had few momentum investors on the bandwagon (who jump abandon ship when prices fall, creating selling pressure) and have little room to fall. If prices were low enough, people will jump back onto the small value bandwagon during the next bull market.

We don’t always get stage 7. But in 2000-2002 as large growth was collapsing, small value was up!

Where are we now? Probably either stage 5 or 6. We will find out.

---

[Chipper6]

I am no longer predicting mega cap outperformance. In the short-term (over the next year or two), mega caps may continue their outperformance, but it is not sustainable long-term. This means that small value is due to be back in style over the next seven years.

But the real opportunity going forward is not U.S. small value but international small value. That’s why I have been from the few (maybe the only) U.S. based user who really wants international data.

New prediction:
It is likely that international small value will outperform over the next seven years!

[Jrinne]

The designer models placed a lot of faith in automation. I know stock pickers who don’t automate their strategy at all who have done very well……And what else works? Large-cap index funds. At least for now.

Yuval,

Does not seem you are recommending the Designer Models for now. If that is what you are saying I would tend to agree.

-Jim

[Jrinne]

Suggestion: Use the correct benchmark. Russell 3000 equal weight is close enough, but an equal weight of the sims universe is even better, especially for targeted universes.

A DM might be working. Or it might not. But what do you learn by comparing it to the SPY?

Chaim,

Correct.

One can easily download the returns of the Designer Models using the benchmark the Designer’s chose. The problem is still there with regard to the 2-year excess returns for sure.

I would be surprised if many of the small-cap model Designers used the S&P 500 as their benchmark which is not to say that there are not some problems with the benchmarks the Designers chose.

Of course there can be problems with the type of model we chose (e.g., small-cap value) as well as with models themselves (e.g., possible overfitting).

-Jim

[enisbe]

I am no longer predicting mega cap outperformance. In the short-term (over the next year or two), mega caps may continue their outperformance, but it is not sustainable long-term. This means that small value is due to be back in style over the next seven years.

But the real opportunity going forward is not U.S. small value but international small value. That’s why I have been from the few (maybe the only) U.S. based user who really wants international data.

New prediction:
It is likely that international small value will outperform over the next seven years!

Very well thought out. I am going to print this and put it on my wall. I have the same opinion on this but I never processed it and put it on a paper in writing. This will do.

[geov]

Here is a new model which selects 15 stocks from ETF QGRO.

This ETF is only about 1 year old. I am using a similar algorithm for this model as for the iM-Best12(USMV)-Trader which has significantly out-performed both SPY and USMV since inception, on 6/30/2014 over five years ago. The other model with similar algo is the iM-Best10(VDIGX)-Trader.

I think that one can do much better using professional ETF providers know-how and select one’s stocks from their ETFs holdings. We have discussed this long ago in the forum and now one can judge the merits of such a strategy which I have tested for longer than 5 years for two models now. Both have performed out of sample relatively well over the same period where the 75 Designer Models failed to deliver.

I have launched this as a DM and you can find it here. https://www.portfolio123.com/app/r2g/summary?id=1592982. There is no performance to see yet but the description is there.

Below is YTD performance, 75% so far. Thanks to three P123 ranking systems and about 3 buy- and 2-sell rules. No market timing. All large-cap stocks.

---

---

[WalterW]

Below is YTD performance, 75% so far. Thanks to three P123 ranking systems and about 3 buy- and 2-sell rules. No market timing. All large-cap stocks.

Interesting. That looks like a sim result. So it’s a 75% YTD simulated? Right?

[geov]

Yes 75% for this year so far for 15 positions, and you can design such a model easily as well.

Why don’t you try it out. Download the holdings of QGRO, put them in a list, create a new universe with this list, and then update the list/universe every 3 months. Try out some ranking systems and you will soon find superior performance than that of QGRO. Then please post the YTD return here so we can all see it, perhaps you can come up with something better.

In 5 years we can compare the performance of this model versus the average of the DMs.
It should perform better because the universe is continuously adjusted to what the ETF provider believes to be the best quality growth stocks. I don’t have to do it with some convoluted selection rules which are fixed and don’t reflect the change of the stock market environment.

In the meantime you can follow it life. No need to subscribe. I will keep it there.

[Jrinne]

Below is YTD performance, 75% so far. Thanks to three P123 ranking systems and about 3 buy- and 2-sell rules. No market timing. All large-cap stocks.

Interesting. That looks like a sim result. So it’s a 75% YTD simulated? Right?

Thank you Walter.

Sim or cherry-picked for sure.

If we need to do anything at P123 I say leave it to Marco to give us the best automated solution he can provide. I certainly like what he has done so far!!!

But I can document that my models are not overfit. Let me be clear however, I would not wish some of my models on my worst enemies (for the last 2 years). It might take more than addressing overfitting to have good models. But it would be a start for P123. Speaking for myself, I do not think I will ever have the perfect ultimate solution. I will need to continue to make progress.

I can say with certainty that I am not here to learn to do bootstrapping on spreadsheets.

Personally, I am not here to do a lot of fundamental analysis either. But P123 probably should not ignore that market. I do believe P123 can walk and chew gum at the same time.

P123 could provide simple solutions to stop overfitting to start with. That means removing factors that will not help the sim or rank performance out-of-sample. IT DOES NOT MEAN GIVING THOSE FACTORS EQUAL WEIGHT AS YUVAL DOES ABOVE. This is just common sense. One way or another—hiring a consultant if necessary–P123 should work to understand some of problems with the sims and giving the members the means to address those problems.

Whatever path he choses to take, Marco has the ability to provide an automated solution that does not entail doing bootstrapping (or anything for that matter) with a spreadsheet.

I would like to see Marco get whatever consulting help he may need (if members cannot just do it themselves with Python). I think the Designer Models can be a guide in deciding whether we can just stick with what we are doing now.

Cherry-picked models, not so much (of a guide).

Georg’s average 2-year excess returns (many models using ETFs with an S&P 500 universe, e.g., Sector SPDR and SPY-SH) is -6.65%. Posters should present their financial interest and how their Designer Models–without survivorship bias–are doing on average. P123 needs to implement this, it seems, if we are to have a rational discussion.

Here is my, not so cherry-picked, example BTW. It is only fair that I receive some of the same scrutiny I have given others even if I am not selling anything. Exactly even with the benchmark (not SPY obviously) over the last 2 years with too much volatility in my opinion. For sure my wife does not like it (the volatility). I am reducing this volatility with uncorrelated ETFs (actually low correlation) and I have a lot more to learn.

-Jim

---

[WERNER]

Georg:

“…Download the holdings of QGRO…”

I found no way to download all the constituents for an ETF like QGRO.
On Yahoo Finance the first 10 are listed but not more.
Is there a website providing all of them?

Thanks.

Werner

[geov]

Werner, you have to go to the ETF provider. They have to give you daily the holdings of the ETF.
https://ipro.americancentury.com/content/ipro/en/products/etfs/fund-list/american-century-stoxx-us-quality-growth-etf.html

[WERNER]

Georg,

thanks for the quick reply.
Will do that.

[judgetrade]

@Chipper6 very good Framework!
well, then international stocks come just in time!!!

[quote]
SPY is dominated by mega caps. (See the attached chart). Mega caps have been outperforming. That might not explain everything, but it does explain some of it.

Why are megacaps now dominating? Here is what I wrote in 2012:

…in the long run (over the next 7 years) large (as a class) will almost certainly do better than small.
[/quote]Source
Here we are 7.5 years later and my prediction has come true.

I see the market going in cycles. Mega caps outperformed in the U.S. during the 1960’s, the 1990’s, and the 2010’s. The common denominator is that all three periods were preceded by a long bull market.

Stage 1.
Bear market: Stocks are dumped. Stocks; especially small caps, take huge price hits, sometimes for reasons that have nothing to do with earnings; such as margin calls.

Stage 2.
1.5-3 years. Fundamental investors get to choose between buying fair businesses at great prices or great businesses at fair prices. The long-term investor would prefer great businesses, but in the short term prices for fair companies rebound sharply. That causes small value to outperform.

Stage 3.
People looking at past history stick with small value because of past performance.
Fundamental investors get to choose between fair businesses at fair to good to great prices and great business at good prices.

Stage 4.
The small value juice has been squeezed.
Fundamental investors get to choose between fair businesses at fair prices and great businesses at great prices. This starts a rotation into large growth.

Stage 5.
Momentum is king.
People start jumping off the small value bandwagon. Why own small caps? Past performance shows better returns and lower volatility for mega caps!

Stage 6.
Bubble trouble. As investors become less valuation conscious and more momentum conscious, valuation gets ignored. Nifty-Fifty, Dot-Com, FANG, stocks look like great deals at any price! Until they aren’t.

Stage 7.
Small value stocks do best in the next bear market because they started from depressed prices, had few momentum investors on the bandwagon (who jump abandon ship when prices fall, creating selling pressure) and have little room to fall. If prices were low enough, people will jump back onto the small value bandwagon during the next bull market.

We don’t always get stage 7. But in 2000-2002 as large growth was collapsing, small value was up!

Where are we now? Probably either stage 5 or 6. We will find out.

[yuvaltaylor]

I was very impressed by Chaim’s seven-stage model and found myself convinced.

I was also convinced by some observations from Verdad Research’s weekly e-mail, which I attach.

If you look at the trend in this manner, a correction is bound to come soon.

On the other hand, I don’t see it coming soon because the CAUSES for this crazy outperformance of large-cap expensive over small-cap cheap don’t seem to be going away. I don’t see any economic changes at hand.

It’s a fool’s errand to assign causes to market movements, so here goes my fool’s errand: if you have lots of money to play around with, you don’t care if something’s cheap or expensive. You just want to buy whatever’s shiniest. That is precisely how the market has been working over the last few years, and Trump’s tax cuts really exacerbated the problem. There are tons of people with too much money to spend. If, on the other hand, money is tight, you’re going to look for cheap stuff to buy. Does anyone think that money will become tight in the near future? That’s the only way I can see this trend reversing and small-cap value beating large-cap expensive.

I REALLY hope I’m wrong. And I’m of two minds. I have no idea if the trend will reverse or continue . . . In the meantime I’m trying to thread the needle by assigning less weight to value factors than I used to in some of my ranking systems.

---

Verdad Research email.doc (603 KB)

[regallow]

[quote]
On the other hand, I don’t see it coming soon because the CAUSES for this crazy outperformance of large-cap expensive over small-cap cheap don’t seem to be going away. I don’t see any economic changes at hand.
[/quote] While staying within the S&P500, I find some evidence that large is cheaper than small, and that might go a VERY long way to providing a cause for the large outperformance. I have not looked at small-cap outside of the S&P500.

Comparing the median PE, PEG, PEGY, and PERel factors for the largest 100 and the smallest 100 market-cap stocks in the S&P500 over time, the largest market-cap stocks appear to be less expensive for all of those factors and have been less expensive for some time. Below are just those factors over time. For me, it doesn’t get much easier to find factors that could favor one portion of the market over another for investors wanting a simple method to choose what stocks to purchase. The better PE measures for large-caps as a group should have underlying causes that are easy to guess at or assume, but I want to see proof. I haven’t worked on that yet.

---

---

---

---