Piggybacking

Research
1

Hi all,
I started a new thread so as not to take away from the “Machine Learning Failure thread”. I may be missing something here but Yuval’s response to George’s post about piggybacking best hedge fund managers seemed a little harsh, unless he has information that I do not see. I understand the concept of look ahead bias and survivorship bias but without an explanation as to how and when George selects "BEST " hedge funds or etf’s I would reserve judgement. If George is picking only from hedge funds that have done well or survived then of course you would be justified in your response but if he is picking from ALL point in time(as he says) funds(without bias) then I would like to dig a little more into his process before passing judgement.
Of course a backtest is just that, a backtest, but if his process is solid then how is that different from any other factor we backtest.
I for one would like to understand a little more on how George chooses the “BEST” if he is willing to share.
I for one do not want to squash individuals from sharing their ideas nor do I wish to dissuade constructive criticism but maybe we should have all the facts first.
George are you willing to share your thoughts?

2

I want to respond to Yuval first:

Yuval, there is a huge difference in my opinion. Fund managers are not like index funds or ETFs or individual stocks. They actually pick stocks which in their opinion will show a good return. For example, David Tepper hedge fund Appaloosa Management has averaged 30% annualized net returns over the last 23 years, according to Forbes.

Some funds charge 2% management and 20% performance fees and the typical minimum hedge fund investment is between $500,000 and $1 million. Most of the big successful funds have a much higher minimum initial investment. No sane person would pay this if they didn’t think they would get a good return from experienced fund managers.

So my strategy is not so stupid. If people are willing to pay high fees and entrust a large amount of money for investment to various hedge fund managers, then why would I not also want to be in the same investments? (Granted, I would be 45 days late getting the information - but that’s o.k., things don’t change that fast)

I selected five funds which have performed well since 2001.
BRANDYWINE MANAGERS, LLC, RA CAPITAL MANAGEMENT, L.P., SHANNON RIVER FUND MANAGEMENT LLC, TCI FUND MANAGEMENT LTD, and WHALE ROCK CAPITAL MANAGEMENT LLC.

If I had the money I would invest in those hedge funds, but unfortunately I am currently not that flush to be able to part with $5-million. So I can possibly get similar returns by piggybacking on the fund managers’s expertise, without paying exorbitant fees. The key is using the fund managers’s expertise, not cherry-picking.

I then decided that I wanted a 20 stock portfolio of the 10 top quarterly holdings from each fund. So I am selecting 20 stocks from about 50.

I then found the point-in-time stock holdings for every quarterly 13F filing of this portfolio from 2005 onward. (I chose 2005 because my backtest ability in P123 is limited to 15 years.)

After I had all the historic holdings I put them into a csv file and uploaded this information into a stock factor file. This took a lot of work to get all the ticker symbols to be recognized by P123, but I did eventually manage to eliminate all error messages.

I was delighted that my returns (screen and simulation) were just as high, or even better, than what the 5-hedge fund group produced. I have no doubt that this strategy will continue to match the performance of the hedge fund group going forward - why should it not when I am using their best stock picks.

One only has to update the stock factor every 3 months to stay current. So until middle of May there is nothing much to do.

3

Hi Georg,

Where are you getting the transaction data? I need a source for myself.

Walter

4

Walter,

I am not sure if Georg will share his source with you, (he is guarding his secret and not sharing his csv with me despite our history of sharing info together.)

You can get all the hedge fund data from https://whalewisdom.com with a subscription.

For the historical holdings of ETFs/Mutual Funds, I am getting the data from Reuters Eikon.

Regards
James

5

Hi James,

I checked out Whalewisdom but it looks like it would require quite a bit of consistent work to get it into a format that is compatible with P123, unless I am missing something?

Scott

6

Thanks James, I’ll take a look.

7

Scott,

Once you have subscribed to whalewisdom, you can download the past 13F filings (historical holdings) of almost any hedge funds in xls/csv format that you can then put in a stock factor csv. Here is the link to the download page for TCI fund maanagement and you can choose from all the historical holdings under “quarter to view”. Quarterly 13/F filings for the latest 2 years are free of charge.

https://whalewisdom.com/filer/childrens-investment-fund-management-uk-llp#tabholdings_tab_link

Regards
James

8

How is this not look-ahead bias? What’s the difference between choosing five funds which have performed well, five ETFs which have performed well, five mutual funds which have performed well, and five companies which have performed well? In each case, there’s a brilliant manager at the helm. But the question remains: how are you going to choose which brilliant manager to follow going forward? What makes you think that funds that have done well in one period will do well in another? That certainly hasn’t been the historic case with mutual funds. Why should hedge funds be different? The only way this experiment makes any sense is if you or someone else had identified these five funds in 2001.

But that’s not my main point. My main point is DO NOT DO BACKTESTS THAT REFLECT FUTURE RESULTS. It is a complete misuse of backtesting to take funds that you KNOW performed well and backtest their holdings. That’s the kind of thing that not only gives backtesting a bad name, it is contrary to EVERY principle of data science and contrary to the entire spirit of Portfolio123.

When we created the custom factor tools, we were quite aware that they could be misused by people importing look-ahead data. I just didn’t think it would happen this quickly.

9

I tend to agree with what Yuval is saying. It’s easy to say in hindsight what funds you should have cherry picked stocks from. So my challenge is how do you choose which funds every year to cherry-pick from? Are you going to pick from the funds that performed the best the previous year, two years, five years? I question whether it will actually work without the benefit of hindsight that you are inevitably embedding into this strategy.

I’m not saying that it doesn’t have merit but one needs to be very careful to define not only how to screen stocks out of the funds without look back bias but to also formulate a strategy for what funds to use without look back bias as well - this is the key. You can’t backtest a fund you know performed well the last few years unless your rules would have selected the fund in that point in time without knowing it would well in the future years. Otherwise you are just executing another form of look back bias. It’s easy to backtest out of a sector or subset that we know did well the last few years, but would have you known that at that point in time? I know I am repeating myself, but this is very important.

I don’t want to dismiss your idea out of hand which is why I am suggesting the following strategy…

Since you selected the best performers since 2001 why not start by selecting the hedge funds to cherry pick each year that performed the best the past 15 years. That means you start out picking the best hedge funds from 1985-2000 for the 2001 investment year, then from 1986-2001 for 2002 and so on. This would create a backtest which removes hindsight bias going forward and provides more evidence to the viability of your approach. You could use the best of past 5 or 10 years as well. You can’t backtest a cherry picked fund over a period of time you know it will outperform. You need to blind your backtest to its future performance.

Jeff

10

Think about it in another way. If one person has good performance in one year, then it might be just luck. However, if 10 yr, 20 yr, it’s more than just luck. There is edge in his investment approach. So a group of hedge funds out-perform consistently over 10 yr. I think the odd of those hedge funds outperform continuously is probably higher. I don’t have any stats to support it. It should be fairly easy to test using hedge fund track record.

gs3

11

Nobody is disputing that possibility, but when you backtest hedge founds because they performed well over that same backtest period you are using look ahead bias. Would you have selected those same funds in 2001 or would you have selected totally different funds? This is no different then selecting the FAANG stocks over the last 10 years and backtesting those because they did well. You knew they did well so of course backtesting them over that period with that knowledge used to select them for the backtest would yield good results.

12

Jeff,

Obviously a good way to do it.

Enough people have recognized this and have used it that it has a name: walk-forward validation. De Prado at AQR Capital Management, for example, has written much about this.

Right now it would be difficult to optimize a ranking system for 2000 and test it on 2001 then optimized for 2000 and 2001 and test it on 2002………optimize for 2000 to 2018 and test it on 2019.

Nonetheless, that is a good idea.

But here is what you can do now. You can divide the stocks in the best hedge funds in to even and odd using stock IDs. Then Optimize for the even ones and see how the odd ones do.

Hmmmm. For some reason that does not seem as good. Probably just me.

Best,

Jim

13

Jim,

You could do that but you still aren’t eliminating the look ahead bias. It might be more implementable to shorten the backtest time to 10 years and then select the hedge funds which did the best the prior 10 years to that backtest point in time. You could also do the last 15 years with the previous 5 years every year in the backtest.

Jeff

14

Yuval, My strategy is based on something that has worked well for almost 6 years now.

If you care to search the forum you will find discussion on this, namely piggybacking on good funds. At the time I chose ETF USMV and Vanguard mutual fund VDIGX and I picked the best 12 and 10 stocks, respectively, from these funds with a ranking system.

https://imarketsignals.com/2014/trading-the-dividend-growth-stocks-of-the-vanguard-dividend-growth-fund-simulated-performance-of-ims-best10vdigx/

OOS annualized return since July 2014:
SPY = 9.88%
Best12 USMV = 14.71%
Best10 VDIGX = 15.19%

So what am I doing here. I follow two fund managers which I believed should do well and I picked their best stocks using a P123 standard ranking system. How does this differ from following the combined know-how of 5 hedge fund managers who have consistently performed well for many years? In a few years there will be proof that this approach works OOS.

The piggyback strategy is based on a successful model proven over the last 6 years. This is not something I just dreamed up because P123 introduced the stock factor tool. And I do take exception to the insinuation that I am misusing this tool. Instead of criticizing my approach, P123 should follow Marc’s and my recommendation to get historic stock holdings of various funds into their database so people can use them for backtests and strategies. That would be a real advance.

15

Jeff,

That was my point.

I have recommended walk-forward validation in the past but people thought even/odd universes were better.

This is a clear example of where even/odd is no good. I think generally it is no good and this just shows the problem clearly.

But this does present a clear example where even/odd universes completely fails. And would be no surprise to someone who has read de Prado’s books.

No surprise to you either, I might add.

Sorry that I presented my point in an unclear manner.

Bottom line: I 100% agree with you and disagree in exactly zero areas on this.

I think you make great points.

Best,

Jim

16

Jim,

I did gather that was your point. I guess I was just reinforcing your argument. What you have stated is the only way you can generate a viable backtest on this type of a strategy.

Best,

Jeff

17

Having read through all the comments, IMHO, it all comes down to POINT IN TIME. If we are selecting stocks from managers that reflect future results then I believe the process to be flawed. If, however, we are selecting stocks from some criteria- Point in time as Jeff has suggested then there may be something there.

18

Taleb calles it the Lincy (from the book skin in the game) rule, if something (pretty complex, e.g. a great bakery at a famous spot) worked for 20 Years, it will work 20 years in the future :wink:

I would say it depends. One could simply buy Berkshire or the smith funds (check it out, buffet style investing!) based on the Lincy rule.

Same with system design.

There was huge alpha in an intermarket system between QQQ and TLT (switching) with a very good capital curve.

Basis of the analysis: Tech does well in 3 of 4 market (4 market regimes = rate of change of GDP and Inflation e.g. 4 combination) regimes, in
the 4th Market regime TLT does great.

So trading this system is kind of the same as thinking that a good fund manager (buffet and smith) will be great in the future.

My 10 cent: It depends, if I believe the success variables of the fund manager (buffet or smith) or the intermarket system
between TLT and QQQ is stayble for a tradable foreseeable future, then I think its find.

For example: I would not invest in Buffet anymore, bc. I think his succession is not clar.
But I would definitly invest in smith funds, bc. the fund manager is not to old and perfected what buffet
is doing.

So simply combining the best funds, without analysing why they might work in the future, hmmn, no rather not (I would have to know
the managers, their style and their age and their ability not to get fired (I know only smith and buffet beeing in that position).

But, as Marc pointed out: Ask youself if that system works in the future and why. If you got good reasons, then bet your money.

19

Here are, in my considered opinion, right and wrong examples of a “piggyback” backtest.

Right:

  1. Piggyback on a quant-based index fund whose rules can be replicated. USMV is a good example. But VDIGX is definitely not quant-based. Find a quant-based dividend growth index fund to piggyback on.

  2. Develop piggybacking strategies that will improve the performance of ANY fund, not just VDIGX or high-performing hedge funds that are cherry-picked after the fact using look-ahead bias.

  3. Develop piggybacking strategies that are based on choosing funds according to criteria that can be applied before the fact. If you can come up with criteria that would have chosen these five funds in 2001, then use that as a basis for a backtest, and also backtest the same criteria for choosing funds in 2005, 2009, 2013, etc. If high performance is your criteria, choose the funds that performed best from 2005 to 2012 and backtest those for 2012 through 2020. Use mutual funds or hedge funds or ETFs–take your pick.

Wrong:

Choose a fund based on its performance up to a certain point and then GO BACK IN TIME and piggyback on it and pretend that the results are meaningful.

We are definitely planning to get the historical holdings of ETFs into our database so that people can do this far more easily. It’s one of many, many improvements in the pipeline.

20

Different that Jeff’s criterion.

IMHO, Jeff got it exactly right and there is not substitute. And I think Brett got it exactly right also.

Whether it is a quant based strategy or not you are not picking the fund PIT and there is look-ahead bias.

If I find a fund that has done well, pick stocks from the fund and find that they have done well I have a hard time convincing myself that I anything more than engage in a little confirmation bias.

Best,

Jim