All that glitters is not gold: Comparing backtest and out-ofsample performance on a large cohort of trading algorithms

This is complicated.

First, there are 10,000 mutual funds out there. They list their current holdings but do not have to disclose them to the SEC. The only way to get historical holdings, as far as I know, is from the fund houses themselves.

On the other hand, all institutional investors who manage over $100 million in assets have to file a 13F listing their holdings with the SEC every quarter. This includes hedge funds, closed-end funds (I think), and investors like Warren Buffett. We get summary data of those holdings, which you can access with the Institutional commands (search institution in the Factor Reference and you’ll see them). We do not, however, get the holdings of individual institutions or funds.

If you want that, I recommend a subscription to whalewisdom.com. They keep excellent track of 13Fs. If there are certain funds/institutions/investors you really want to follow, they’ll give you Excel files with their holdings and API access too. They offer backtesting too, FWIW.

I may be mistaken about some of the above–if so, I’d appreciate being corrected.

Yuval,

The quarterly historical holdings for mutual funds/ETFs are available from Bloomberg and Reuters Eikon. (I have been sharing the data with Georg/Jim to build the stock factor csv). The data is available publicly and out there, Portfolio123 just need to find a data provider (just like the ETF data) to use the data.

Regards
James

The only thing that concerns me is that if the data is retroactive then we will be a quarter behind. Real time ETF holdings shouldn’t have this problem but seeing what a fund held and duplicating that might not work as well. I think we have to be careful about point in time information. Maybe you guys have already figured out how to deal with that and maybe I’m just off base.

Jeff

Jeff,

The cloning/pigging back on mutal funds/hedge funds has been around for some time and the 13F filings is always 1 quarter behind.

In order to make the cloning works, you will have to find a fund manager that doesn’tn hold thousands of positions at the same time and with a low turnover rate (exluding funds such as Citadel, Millennium …etc)

Regards
James

James,

Not disagreeing with approach. In f mo act I think it’s a great approach. Just wondering if we are backdating the the information so our simulations don’t act information until it is known is all.

Since I don’t have time to try it myself at this moment I am just observing at this point.

Jeff

Jeff,

The sims (let’s say on Tiger Global) will always be based on positions that the hedge fund is holding last quarter as reported in the latest 13F filings.

By the way, pls check your email.

I have send you a message about the GLD rotation.

Regards
James

James,

I read your email. I am trying to simulate the strategy with the rules you outlined as buy rules with GLD ticker and sell rules the opposite but it doesn’t seem to be giving good results. Do all rules need to be satisfied to buy GLD and sell GLD if any rules broken? I have it set to daily rebalance.

Jeff

Jeff,

Those are buy rules for your equities portfolio. You can invest in GLD when the buy rules are broken (risk off).

Setting it to daily rebalance. should yield better results.

Regards
James

James,

I think I got it to simulate. I thought it was odd to buy gold on those rules. Makes more sense the other way.

Thanks,

Jeff

Jeff,

For you and others who are interested in GLD

https://www.cnbc.com/video/2020/01/31/chartmaster-gives-golden-protection-play.html

Regards
James

Jim,

Check this out.

Since you like machine learning, I think you will like these two articles.

Regards
James

Quantum Computing Inc. Releases Its First Quantum Ready Software Product QAA – The Quantum Asset Allocator to Optimize Portfolio Returns

Quantum Computing Inc. (QUBT) (the “Company” or “QCI”), an advanced technology company developing quantum ready applications for quantum computers, announced today that the Company has released its first “Quantum Ready” software application – The Quantum Asset Allocator (QAA).

The target market (estimated at over $1 billion) for QAA is financial institutions who are currently addressing asset allocation problems but are looking for better tools with which to optimize portfolio performance. QAA is available both as a cloud based software service and as an on premises software + hardware system. Both implementations are designed to quickly return optimal or near-optimal interactive solutions and analyses of financial asset allocation problems. QAA leverages a financial institution’s strategy for calculating risk and expected return, based on analytical values for the various index sectors and subsectors in its investable universe. QAA has been proven to enhance fund strategy by calculating the optimal portfolio mix to maximize returns in beta tests against portfolios using traditional portfolio management techniques. “This is a major breakthrough for QCI”, stated Robert Liscouski, CEO of Quantum Computing Inc. “We are excited to be releasing QAA which will provide small and medium sized funds the ability to do asset allocation that previously was the province of large brokerage firms, mutual fund and the largest quant funds. Beta tests have demonstrated superior portfolio performance using quantum inspired techniques on both classical and existing quantum computing hardware,” he added. Liscouski stated that QCI is already working with beta clients to implement QAA in their environment.

Quantum Computing Inc. Releases Mukai, the Quantum Application Development Platform

Quantum Computing Inc. (QUBT) (“QCI”), an advanced technology company developing quantum-ready applications and tools, announced today that the company has released its Mukai quantum application development platform. Mukai can be used to solve extremely complex optimization problems, which are at the heart of some of the most difficult computing challenges in industry and government. Its software stack enables developers to create and execute quantum-ready applications on classic computers, while being ready to run on quantum computers when those systems can achieve performance advantages. QCI has already demonstrated superior performance today for some applications built on Mukai and running on classic computers.

Mukai uses highly-optimized parallel code, and is currently centered on the quadratic unconstrained binary optimization (QUBO) formulation well known to quantum annealing users. QUBO is a pattern matching technique, also commonly used for machine learning applications.

The Mukai software stack includes two primary user/developer interfaces:

QCI NetworkX graph-analysis package that exploits that capability for graph problems
QCI qbsolv package that implements a state-of-the-art, high-performance optimization capability for QUBO problems
“Mukai enables developers to create and deploy practical applications that solve very hard problems today,” said Mike Booth, QCI’s Chief Technology Officer. “We believe this will bring immediate value to potential clients, while also providing a clear path to emerging quantum hardware. As Mukai provides an abstraction layer to the actual computing hardware, a client’s investment in application development today should pay off in two ways, without having to take the hardware platform into consideration.”

“In addition to developing the quantum application development platform, QCI is leveraging its expertise in finance, computing, and security to build applications on top of Mukai that we believe will deliver immediate value to potential clients,” said Steve Reinhardt, VP of Product Development at QCI. “The first of those, Quantum Asset Allocator (QAA), was recently announced. It provides small and medium-sized financial institutions the ability to do asset allocation in a way that was previously the province of large brokerage firms, mutual funds, and the largest quant funds. Using Mukai means it can run on either classic or quantum resources, depending on which delivers superior performance,” he added.

[quote]
Quantum Computing Inc. Releases Its First Quantum Ready Software Product QAA – The Quantum Asset Allocator to Optimize Portfolio Returns

QAA has been proven to enhance fund strategy by calculating the optimal portfolio mix to maximize returns in beta tests against portfolios using traditional portfolio management techniques.

Is this not backtesting?

Georg,

I bet they do more than just backtesting. Ideas like your AIC.

Quantum computing–by its very nature —is like bootstrapping. It automatically looks a multitude of possibilities all at once. That is the very nature of what it means to be “quantum.”

That is ultimately why people will move to quantum computing. Regular computers do regular backtesting just fine.

Anyway I agree there is a whole world of ideas out there that can address overfitting without focusing on backtesting. Ideas that we continue not to pursue here at P123.

This is definitely one of them.

In fact there is a whole world of modern ideas that some at P123 fight against. People who use these ideas privately but refuse to even entertain the idea of bringing them to the platform.

The focus on backtesting is just a distraction that prevents us from discussing things that P123 could be doing, IMHO.

-Jim

Georg,

I am only posting the articles from Lexis Nexis for Jim’s reference.

You are probably right that they calculate the optimal portfolio mix through backtesting from different asset classes.

Regards
James

Jim,

I know that you prefer investing in ETFs than stocks, perhaps you are interested to check out my ETF sector rotation model?

I have made this visible to public under my username “ustonapc”.

This model always invests in 3 Blackrock ETFs simultaneouly and changes through sector rotation (except when risk-off). Both the 15/10/5/3/1 years backtest and out-of-sample performance (6 months now) beats S&P500.

Although I am not allocating any capital in this model, it is the one of the best ETF models I have (except for two other risk-on/risk-off ETF models that rotates between XLK/GLD and QLD/TLT using market timing).

Regards
James

James,

Thank you! I will check that out.

-Jim

Jim

What do you think about the ETF rotation model?

It focus on using Blackrock ETFs (i-shares) rather than SPDRs that you are probably working on.

Regards
James

James,

For some reason I can only find your public BPTIX stock strategy.

I have not searched a for a public ETF strategy before and I suspect I am doing something wrong. But I do have “All” selected under model type and trading system type.

-Jim

Jim

It is under the screen section.

Regards
James

Nice!

I will take some time. I will use your ideas and many others to develop the best possible system over the max period up until 2015.

When I am done I will take what I think is the single best system and see how that system does from 2015 until now. I will then train or optimized the system with all of the data (to the present date) and regularly retrain or re-optimize the system in the future if I decide to use the system. Or I might discard the whole idea and look elsewhere for profits.

The best way to not fool myself, I think. And it can be done with a minimum of data snooping with technical factors in my case.

James, great ideas that I will use in my search!

Thank you.

BTW, others have suggested testing models in foreign markets. Very long test periods etc. ALL GREAT IDEAS!!! And, of course, this is not a complete list of all of the great ideas.

Note: lots of backtests—up until 2015 with my method. Not an argument for stopping backtests (over any date range)—just for a rational use of backtests. Whatever rational might mean to the user. That should be up to the member.

-Jim