Market timing will sometimes lower returns, but a basket of different timing (and other risk management) strategies will give you a decent shot at missing big DD’s. No guarantee but a shot. I’ve done a lot of testing on this and talked to a lot of people running hedging programs for family offices and the like. Nearly all of them rely mostly on technicals - including volatility in the market and trend patterns - with other basics repeating (like DD portfolio level exposure ‘stops’ and volatility targeting).
You should also re-read all of Mebane Faber’s stuff, he’s written and done a ton of testing on this. The best defense against over-optimizing a single value is picking a basket of settings and risk management methods. There are many studies across markets and time frames showing generally robust results for wide range of values in terms of DD minimization.
See (from Mebane’s paper listed below):
“Our recent research piece entitled “Where the Black Swans Hide” demonstrated the impact of
market outliers as well as the tendency of big price moves to come in batches. This “volatility
clustering” occurs after markets have already been declining and is due largely to the behavioral
properties of the human condition. Namely, we hate to lose money but love making it and we use
different parts of our brains when doing both”
Can also look at the history of the best original and second generation trend followers. I know many firms that had 2 decades plus success (sharp ratio’s well over 1), trading the same set of systems on 50-100 markets. I’ve talked to a lot of these people. It got a lot harder for them after 2008, and most have added many other types of systems (like index arbitrage and mean reversion and shorter term intraday trading of patterns). But, the original systems are still being run with some portion of the money in many cases. Often with up to 40% of the money or so. And they still tend to work (for example running up big gains this year in betting on oil’s falling prices after the trend was in place).
There is no perfect system across all market cycles - there are only baskets of systems that are complimentary and expected to do well in very different types of markets.
Market technicals are the best indicators available for dynamic index or option hedging programs that are ‘short term in nature.’
Chipper,
I disagree. There are a lot of successful traders using ‘trends’ and ‘mean reversion’ across 50-100 markets for decades. From commodities to asian equity markets - same formula’s often. They are not using any ‘money supply’ or (often) any fundamentals at all - at least for most of their money.
What will predict whether ‘market timers’ fall on their face in terms of reducing big DD’s is the degree to which a) future markets have big DD’s, b) the big DD’s have rising volatility and smaller DD’s before them (they usually have in the past) or whether they come completely out of the blue. There’s nothing magical about it - if a) and b) happen, various market timing hedge rules as part of a balanced portfolio will help in aggregate limit DD’s.
I think that we actually both agree on all of that. I agree that market timing using trends reduces the large drawdowns. I also agree that some of those strategies that you mentioned have been used successfully to reduce volatility of overall portfolios, but they have always been niche strategies. The market timing that is rarely used for funds is the type that exits the stock market when a market timing rule is triggered; similar to the timed R2G strategies. When such a strategy is combined with a typical fund that is really lucky to be getting an alpha of 1%-2% then it will almost certainly look foolish during strong bull markets when buy and hold beats tactical allocation. Even in sideways markets it can look foolish for years before getting vindicated. Therefore the vast majority of equity funds don’t time the market because it can mean the end of a career for the portfolio manager. It only makes sense in R2G’s (when use correctly) because the R2G alpha is so high.
My main point was not to enter the debate of whether or not this type of market timing reduces drawdowns. I am taking that as a given. My main point was why it works.
I didn’t mean to imply that herd behavior explains all market actions, merely that it is a driving force behind the type of trends that moving averages are sometimes able to capitalize on. In a herd you will have leaders and followers. So in 2009, for whatever reason, buying started by those that could be considered the leaders of the herd. Certainly there were those that continued selling in 2009 but as Ben Graham said: "In the short run the market is a voting machine … " and after early March the buyers had more votes.
The main reasons why timing at a market level works are likely pretty simple -
a) The big flow of event specific bad news occurs unevenly over time, but they tend to ‘cluster’ and unfold over days, weeks - months / as the market absorbs and slowly comes to understand them. That means we usually learn about a ‘canary in the coal mine first’ - then some more news that builds on this i.e. there might be some issues with some collateralized debt and these bonds aren’t worth what we thought, then we tend to start hearing more of the bigger picture news items - this bank has huge exposure and some short sellers are shorting it. Then - something bigger happens - a big bank is on the ropes. Wait, the whole economy might go down. Or, for example with Enron - first there was a Wall Street Journal article talking about how a lot of their earnings - nearly all - were future projected earnings, then short sellers got on board and started writing articles and talking on news outlets, more reporters dug in, etc).
b) Market cycles are real for a variety of economic and non-behavioral reasons - i.e. new technologies appear, slowly gain market share - then explode into public awareness. It’s very hard to value them or analyze their real market impact at this point. Most fail. Some kill off the big companies and redefine their sector (and sometimes the whole economy). People can make a lot by being right and being first - and people rush in. Or… for another example, companies make huge up front investments (i.e. in fracking) - and it’s hard to value how exactly this will disrupt the energy market, let alone who the huge winners and losers will be, then we start to get a handle on the level and scale and cost and revenues of everything, and then market reaches maturity, but the old wells are failing and the prime land has been gone, and a new innovation is needed, etc. So long as real market cycles exist, the markets will trend.
b2) Economies and interdependent. The failure of one major sector impacts other sectors. If interest rates rise a lot, this can hurt all borrowers. They can then slow investment and hiring. Consumers then slow spending. Retailers fall. Etc. Or government policies change (or demographics) and these shifts (i.e. birth cohorts) impact markets (consumer, labor, etc).
c) Behavioral. Investors react differently to the same news based on recent events and where we are in the trend - they tend to get much more emotional and ‘therefore market more volatile’, on extreme pullbacks. If my portfolio is up 20% YTD and I hear that one the big banks missed their earnings, then I will react very differently then if I am down 25% and starting to worry if I have enough to retire or I’m gonna be fired as fund manager. Losses hurt more than gains. Almost no pure equity investor will be fired for a 5% pull down, but many will for 20% market underperformance. That leads to -
d) Automatic hedging or ‘risk reduction protocols’ are part of the systems of nearly all big traders. Many traders will start hedging (i.e. shorting a major index often through futures), if 1. their portfolio’s hit some DD level (i.e. hedge 10% if I have a 10% DD, or if VIX hits X, etc). Many have very specific triggers, nearly all of which include some technical market events occurring. So, as larger players will start hedging in the futures or options markets, they are exacerbating the dips by effectively selling the broad indexes. And big trend players will jump on these trends (with mean reversion people on the other side). So, big trends will have major ‘reinforcing trading events’ building them. The same thing happens with all volatility targeting funds - which are some huge volume funds.
EDIT: Chipper, 99.9% of the major technical futures traders I have spoken with use market technicals (that’s a tautology). The derivates markets trade trade huge volumes - much more than all the retail investors out there. And most of this money is now being systematically traded. Only a small percentage of those I have spoken with are using (or admit to using) money supply (although fundamental timing traders are). They react to the price and volume and order book dynamics of the market to make (often) near term predictions of price movements. These then create and exacerbate cycles. But, they are competing with fundamental traders.
e) The ratio of fundamental traders and technical traders (and dollars in each) in the market - and the frequency and rules used for switching among them - i.e. dollar flows into investing styles. If everyone in the world were a trend trader, and everyone only bought stocks, markets would never go down. If everyone in the world were fundamental traders, only event specific and market cycle based ‘fundamental trends’ would exist. So, the ratio of these players and the dominant investment styles (which ebb and flow over time) - impact the types of market cycles we see.
[quote]
…after early March the buyers had more votes.
[/quote]Don, that is true of course. But who were these voters, and where did they get all that money to put into the market? They were not technical traders. They were not fundamental traders. They were not market timers. So where did this money come from and why did it find its way into the stock market?
The money supply theory says that this money originated from financials and that the market bounced up–not so much by traders who decided to buy based on technical rules or whatever–but because financials suddenly got more liquid money sitting around available to invest. With the amount of money available to invest going up, allocation plans that were already in place pushed that money into the market. This, combined with the petering out of forced selling such as margin calls (as indicated by the petering out of the abnormally high volume during the crisis) swung the voting pendulum over to the buyers’ side.
Conventional market understanding (which is reflected in you post) presupposes that bear markets are caused by money moving into and out of the market in a discretionary way; meaning that the traders are choosing to do so due to “fear”. I would like to argue that while some of the market’s movements are caused by traders who choose to buy or sell, most of the bear market movements are caused by the supply and demand of investable money by investors who did not set out to time the market purposefully but are being pushed to trade by the amount of available money that they have to invest.
For example, in March 2007 investment banks started seeing losses on their loans. This caused their portfolio overall to shrink. This caused their allocation to stocks to grow relative to their bond portfolio. This imbalance in turn forced them to sell their stocks. Meanwhile as work got out that these investment banks needed money they had a harder time borrowing cash. This amplified their need to sell stocks in exchange for cash. This forced selling was not a discretionary decision made by these banks to get out of the market because they saw a bear market coming but because they needed to raise cash.
Conversely when the Fed prints money by buying bonds, the sellers of those bonds end up with extra cash in their portfolios. Some of that extra cash will find its way into the stock market. That creates additional demand for stocks and pushes prices up. Hence, QE has been highly correlated with the stock market. What is interesting with QE is that it seems that it was not so much the QE announcements that pushed up the market in a meaningful way but the actual cash from the QE programs that did it.
I remember that link. I stopped reading after these:
“Banks don’t generally hold stocks” …really?
“The two cancel each other out” (talking about foreign ownership of US securities vs US ownership of foreign securities)…that’s a good one
I agree that governments can have huge impacts on markets. If long-term taxes fell to zero and s-t cap gain taxes rose to 100%, we would see a large market reaction. Likewise monetary policy matters a lot. I would view all government and money supply and legislative effects as ‘news events’ that are part of ‘a’, but they could be broken out. They clearly matter A LOT - but even if they didn’t exist, markets would tend to cycle and these cycles would likely be more violent to the downside (the same can be said of any single factor).
Even if the Fed had a constant monetary policy (like Friedman’s proposed X% fixed money supply annual growth), we would still have stock market cycles. And volatility. And the ‘downside events’ would be more violent than the upside events. In fact, without aggressive Fed responses I think ‘down market cycles’ would be much more common. I think the Gov’t intervention in 1987 and 2008 helped overall at times of peak investor stress.
The proposed forced rebalancing by ‘investment banks’ you propose is just one method of ‘risk management’ that occurs when stock markets decline. Margin calls and liquidity restrictions also matter and can added as a factor.
‘Rebalancing’ is no longer the dominant form of trading. What you describe (rebalancing) is more likely to be endowments and mutual funds with longer holding periods. Most trading now systematic and much shorter term. I-banks are more likely to have baskets of individual traders and trading groups using derivatives and trading long-short models, short-term holding systems, etc - and dynamically hedging. All of these would fit into ‘d’ in the factors I suggested.
Since 2009, HFT represents over 73% of all US stock market trading. The ‘games’ that these traders play with each other and the reactions of market makers are the dominant short-term movers of the market. They are all reacting to whatever their models show (things like order book imbalances, price and volume - are in almost all systems).
Below from ‘wikipedia on algo trading’ (see:http://en.wikipedia.org/wiki/Algorithmic_trading):
"A third of all European Union and United States stock trades in 2006 were driven by automatic programs, or algorithms, according to Boston-based financial services industry research and consulting firm Aite Group.[8] As of 2009, studies suggested HFT firms accounted for 60-73% of all US equity trading volume, with that number falling to approximately 50% in 2012.[9][10] In 2006, at the London Stock Exchange, over 40% of all orders were entered by algorithmic traders, with 60% predicted for 2007. American markets and European markets generally have a higher proportion of algorithmic trades than other markets, and estimates for 2008 range as high as an 80% proportion in some markets. Foreign exchange markets also have active algorithmic trading (about 25% of orders in 2006).[11] Futures markets are considered fairly easy to integrate into algorithmic trading,[12] with about 20% of options volume expected to be computer-generated by 2010.[dated info][13] Bond markets are moving toward more access to algorithmic traders.[14]
Algorithmic trading and HFT have been the subject of much public debate since the U.S. Securities and Exchange Commission and the Commodity Futures Trading Commission said in reports that an algorithmic trade entered by a mutual fund company triggered a wave of selling that led to the 2010 Flash Crash.[15][16][17][18][19][20][21][22] The same reports found HFT strategies may have contributed to subsequent volatility. As a result of these events, the Dow Jones Industrial Average suffered its second largest intraday point swing ever to that date, though prices quickly recovered. (See List of largest daily changes in the Dow Jones Industrial Average.) A July, 2011 report by the International Organization of Securities Commissions (IOSCO), an international body of securities regulators, concluded that while “algorithms and HFT technology have been used by market participants to manage their trading and risk, their usage was also clearly a contributing factor in the flash crash event of May 6, 2010.”[23][24] However, other researchers have reached a different conclusion."
Whoever is trading the ‘most’, clearly has the largest impact. So…it’s clear that the settings on these models can have huge impacts on the overall market. Not sure why you would not agree with that?
Automatic ‘portfolio insurance’ has also been ‘blamed’ by government commission for the 1987 stock market crash. As well as Margin calls, government action, lack of clear information, market function failure and a variety of factors. It’s unclear of portfolio insurances (auto trading’s) final contribution, but it likely played some role. See the Brady Report. See: http://www.federalreserve.gov/pubs/feds/2007/200713/200713pap.pdf
In the commission’s view, markets were overvalued and trending down. Then, a government policy change (proposed tax change on removing beneficial tax treatment of mergers) led to further declines and was one of the precipitating major ‘inciting event’ as well as fundamentals (high valuations). Heavy selling and a move to bonds sped things up. But, automatic trading and margin calls were also found to be a major ‘likely cause’ or contributor to a 20% one day decline. And the market makers were and exchanges were nearly failing. It’s worth reading the whole thing.
Some forms of trend following would have gotten you out in time. Some would have made things worse.
Market makers (and their responses) also matter a lot. This paper points to the actions of market makers and total liquidity (and illiquidity in times of market stress) as an additional factor that likely leads to violent short-term down movements. That is something you are touching on in describing a more ‘cyclical liquidity’ for the market. It’s possible that fewer sellers on the ‘way up’ - leads to lower possible liquidity and makes it much easier for prices to rise. This is also what can happen at the ‘market exhaustion’ in peak sell-off’s. Few buyers and nearly ‘everyone’ has sold. So the buyers begin to drive prices up. Volume and liquidity are not even. See:
Paradoxically, very low liquidity stocks with longer term holding, constantly hedged with broad indexes - may be much safer in this form or ‘market regime.’
P.S. I never proposed that most trading in down markets is because of fear. It was just one of the factors I listed. I didn’t assign any weights to them.
The paper cites studies that document (among other things) the huge boom in securitization (mostly of debt), the growing interconnectedness of the portfolios of large players before the crash (that helped greatly exacerbate the spread of contagion), a systematic ‘mispricing’ of ‘true’ risk (due to errors in probability distribution modeling among other things), all leading to the creation of more and more complex products in which risks were not clearly understood (among others) and huge portfolio postions were taken. Once the ‘event began’, people couldn’t sell the securitized loans and their market liquidity collapsed and they were nearly impossible to value in the near term. There was also a breakdown in the ability of hedge funds to invest and the ability of ‘arbitrage practitioners’ to trade, due to tightening (vanishing) credit. And the fact that this crisis occured in the financial sector greatly impacted the financial markets ability to function - which raised uncertainty.
So… there are a lot of entangled factors.
And here’s one on 15 US ‘stock market’ crashes and US monetary policy analysis (credit examination). Can have a ‘stock market’ crash without huge credit distortions: www.nber.org/papers/w8992.pdf
Tom I wrote a long post but it got erased. So here goes again (some highlights only):
We seem to agree that supply and demand drives the market. The measure is net money coming in or going out of the market. If we can predict that, then we can know what the market will do.
We also agree that in the short term such as flash crashes and 1987 type rapid crashes the supply will outpace demand very quickly even while the amount of investable money remains constant.
What I am proposing is that news itself doesn’t drive the market except on an intraday basis and sometimes for a few days by nearly as much as people seem to think.
The amount of money that someone is trading daily doesn’t move the market unless that entity is changing his net exposure or unless his portfolio size changes. This means that even if HFTs were 90% of the volume but keep a net 0% exposure then if they totally withdraw from the market the market will not go up or down, although spreads will widen.
I am also proposing a new factor that seems to drive bear markets much more than the other factors. It seems that the supply and demand is driven a lot by the amount of investable money available that doesn’t need to be liquid. This supply of money is different than the money supply that is measured in standard economic textbooks also it can be affected by the official money supply.
We don’t really agree, but it’s been fun talking. Big news can clearly drive the market. If we live in Germany or Japan during WW2 for example. Or if a huge source of oil is discovered in the US and it costs pennies to bring to market.
The evidence suggests both that increasing stock market volume (and automated trading) is very likely driving volatility up significantly and b) potentially exacerbating down events - i.e. that HFT’s are not providing equal liquidity across all types of market conditions - they are making huge amount of money based on the studies I’ve seen - at the expense of other market players. This was discussed in one of the above papers I referenced - predatory HFT activities, and market makers responses to them, as a probable leading cause of the 2010 flash crash.
Below is a quote from one such study (“The Dark Side of Trading. Ilia Dichev, et. al”). These are real studies - with empirical data:
"This study investigates the effect of high trading volume on observed stock volatility controlling for fundamental information. The motivation is that volumes of U.S. trading have increased more than 30-fold over the last 50 years, truly transforming the marketplace, and it is important to map out the effects of such a momentous change. First, we employ a series of three natural experiments to examine the existence and direction of this relation, while controlling for fundamental information that endogenously drives both volume and volatility. We use exchange switches, S&P 500 index changes, and dual-class stocks as settings with substantial variation in trading but good natural controls for underlying fundamentals. Our main finding is that in all three settings volume of trading is reliably positively correlated with stock volatility, and this relation seems economically substantial. Second, we examine the aggregate time-series of U.S. stocks since 1926 and the cross-section of stocks during the last 20 years to better calibrate the economic parameters of the identified relation. Using annual measures, volume and volatility are correlated on the magnitude of 50 percent in the aggregate time-series, suggesting that much of the historical variation in volatility is driven by the prevailing volumes of trading. Tests in the cross-section confirm the positive volume/volatility relation but also reveal a pronounced convexity, where the relation is weak to non-existent for low levels of trading and becomes much clearer and stronger for high levels of trading. Efforts quantifying the volume effect reveal that trading-induced volatility accounts for about a quarter of total observed stock volatility today. The combined impression from these results is that stock trading injects an economically substantial layer of volatility above and beyond that based on fundamentals, especially at high levels of trading. "
For average retail investors - I think most of all these discussions are likely a big waste of time. They are very unlikely to lead to high quality predictive models (in my opinion). There are a variety of reasons for this - but I don’t think understanding this will add much to most retail investor’s risk-adjusted returns.
That study is probably 100% correct but they are clearly using a different definition of volatility than the VIX if they think it is up recently. There has been a clear decline in the VIX since the year 2000.
It is possible that the expected volatility (VIX) has has diverged from the real volatility is suppose. They are probably right, as I say, but I wonder what their definition is.
I agree with Jim. The volatility can be totally different depending on the time period studied (daily, monthly, quarterly, annually or whatever). My primary focus is not on daily or even monthly volatility (which is the focus of many of these studies) but to perfect reliable timing systems for long drawn out bear markets such as 1929, 1974, 2002 and 2008.
BTW, I don’t consider war on home territory a good example because I am not trying to create a timing system based on that. One of my favorite quotes about the market is: “As I like to say, stocks do well, absent war on your home soil, out-of-control socialism, and severe recession/depression.” – David Merkel. I would not invest in a country that is fighting a war on it’s home soil that threatens it’s businesses such as Germany and Japan in WWII (9/11 does not count as war in this sense because it did not threaten to wipe out the economy). I would also not invest in a country that has the threat of out of control socialism where the government is shutting down all industries. (This does not mean Obamacare but a full Communist takeover which is somewhat predictable). However I don’t see a need to develop a market timing system for these events because I don’t see it as an imminent threat to the US, and it should be somewhat predictable. Therefore my market timing focus is on avoiding the recessions/depressions. My S&P 500 Gems market timing system uses a very good gauge of recessions; simple, sensible and the market correlation is good. But I am looking to supplement it with something else mostly to try to catch the market bottoms (my market timer stayed out of the market until late in 2009 when earnings picked up) but also to try to avoid the QE market tantrums which has been signaled by money flows.
Thanks for the quote. I like that one. (John Henry’s left the money management business altogether after having a very rough go of it since 2010 or so, but he made his billions, so I believe he knows what he’s talking about).
@Jim. You can read the paper - the author’s are comparing same stock volatility when, for example a stock is listed at the same time on two indexes, or a stock moves from not in the SP500 to in the SP500. This is not broad market vol, it’s stock specific and isolating the effects of just volume on vol.
@Chipper. You are now asking a different question. The first question in the thread I responded to was by Aurelien and was does trend following help in managing risk. Then I jumped into your more philosophical question of why do trends exist? You stated that news has little effect and almost everything is liquidity driven. I disagree. I think market recessions and depressions are driven by a confluence of events and market fundamentals and news events matter. But I think all of that matters very little in creating tradable timing systems.
You then started trying to talk about isolating ‘money flow’ as a timing signal. Many pro’s trade money flows as one factor in a multi-factor model, but they have detailed tick-by-tick data on order book info (bids vs. asks). It’s very valuable in enhancing system returns I’m told. Systems like the ‘Chaikin indicator’ attempt to do this on stocks, and you can build custom series with averages and moving averages and breadth and volatility of them with P123. But, there’s no likely way small investors will play competitively in this without very strong a) data source and b) programming skills. Most of us lack the quality data and intraday trading and real-time monitoring abilities. Order book is one factor I see hedge funds and CTA’s with short holding periods using.
And now you are asking ‘how do we call a market bottom.’
I’ve tested a lot of these (and others) and use some. Can get the basic economic data on-line for free to backtest for longer time frames.
But… I would never put a ton of my portfolio into calling a bottom unless I had enough set aside to live with a huge DD for a long time. It may take decades to recoup a large bet at this time, if the timing ends up off or the market doesn’t cooperate.
Many systems will be wrong in picking bottoms - it’s a time of very high vol.
PS. HFT’s have an effect on markets and this effect can be sizable. Your assumption here doesn’t feel correct to me. The paper I quoted above concludes that HFT’s response to intraday moves was likely a prime cause for the size of the 1000 point 1 day decline in 2010. It says, for example this:
“Providing liquidity in a high-frequency environment
introduces new risks for market makers (HFT’s). When
order flows are essentially balanced, high-frequency
market makers have the potential to earn razor-thin
margins on massive numbers of trades. When order flows
become unbalanced, however, market makers face the
prospect of losses due to adverse selection. The market
makers’ estimate of the toxicity—the expected loss
from trading with better-informed counterparties—of
the flow directed to them by position takers becomes a
crucial factor in determining their participation. If they
believe that this toxicity is too high, they will liquidate
their positions and leave the market.
In summary, we see three forces at play in the current
market structure:
• Concentration of liquidity provision into a small number
of highly specialized firms.
• Reduced participation of retail investors resulting in
increased toxicity of the flow received by market
makers.
• High sensitivity of liquidity providers to intraday losses as
a result of the liquidity providers’ low capitalization,
high turnover, increased competition, and
small profit target.”
So… HFT’s adapt positions to their expected daily losses / gains, and they withhold liquidity when the market starts going against them. Retail traders are likely to have a very hard time playing this very short-term to short-term intraday market timing game, but it can have a huge impact on total market moves. However, none of this likely matters for us either - I’m just thinking out loud here.
Tom, I apologize for the late reply. Thank you for your thoughtful post, let me flesh-out what I meant in my own antecedent post.
In the OP Denny talks about “ranking” R2G ports. I think he implies, and others implicitly agree with him, that ranking will help protect R2G subscribers from subscribing to, or help them drop, lemon R2G ports.
Definition of Lemon R2G port: a port which consistently under-performs its benchmark index “excess” or benchmark ETF “excess” OOS.
This is a subjective definition, no end point for judgement is specified, but such a port will identify itself as it loses subscribers, it’s ignored and possibly removed by its designer and sent to the R2G graveyard.
Like porn, I think lemon ports can’t be defined but we’ll all know them when we see them.
I introduce a new concept to help with the sisyphean task of managing a portfolio of R2G ports: Personal exclusion of some ports from consideration for ranking, at least for the time-being.
I certainly agree, and did not intend to imply otherwise, but it’s presence is bad.
As R2Gs mature with more OOS, often now at a year for many models or nearly two for some models, I am using a crude down-and-dirty tool that looks at 2014 as a block of excess OOS that can be compared to excess performance of previous years, which are generally in-sample. For some ports 2013 is mostly OOS too. If 2014 is the worst (or best) year ever for excess performance, I consider the risk of overfitting too high, and reject it from further consideration for subscription at this time.
It is just one tool, not sufficient of itself, that acts as a sensitive but not specific test to detect the possibility of undue risk of over-optimization/over-fitting. Notice my emphasis on the word “risk” I’m not saying I know with certainty over-fitting is present in a port or not.
I leave aside the issue of whether or not there is something else causing worst-case-ever-yearly-excess-performance, for 2014, for many R2G ports, of many different types.
I have yet to encounter a R2G port having its best-year-ever excess performance in 2014.
For me, I simply reject the R2G port in question at this time for inclusion in my portfolio of R2G ports. In the fullness of time, with more OOS, this down-and-dirty approach will become unnecessary as the port declares itself a lemon, or not.
Overfitting: Define it, Identify it
When I did my Master of Science in Computer Engineering I worked with backpropagation neural networks (BPNN). The nodes in the network, like those of P123, are the “rules” for learning to predict an outcome. Too few nodes, the system can’t learn to predict. Too many, and the network memorizes what worked in the past but the trade-off is the BPNN can’t generalize enough to predict what will likely happen in the future, i.e; it STILL can’t learn. It has overfit the data, it’s useless. Finding an optimal number of nodes (rules) can be done with the tools of artificial intelligence but a discussion of this would be long, and such tools currently cannot be employed on P123.
So, like artificial intelligence, do some R2G ports have so many rules that they memorize what worked in the past but simply can’t generalize enough to predict what will likely work in the future?
Looking at something like Filip’s SuperValue ranking system, with 20+ ranking rules, I worry. I never saw a good BPNN with that many rules. It’s not unreasonable to assume some R2Gs may have too many rules and overfit.
Let’s look at Hemmerling Value Rockets, launch date April 3, 2013, to show my thinking as a prospective client of Hemmerling.
The Sortino, Sharpe, performance and so on are all great. Worthy of consideration so far.
Sorry these do not line up properly. But last number from each row is last column.
The last column is year 2014. It is the worst year ever for excess performance for the port, a negative in my view. But in this case, year 2013 is mostly OOS, and it is a good year, outperforming four previous years. The way I apply it, the tool cannot say there is a risk of overfitting.
Hemmerling High Yield Russell 1000. 2014 is not the worst year. 2013 is mostly OOS and outperformed six previous years. It shows good year-to-year variability. No discernible risk of over-fitting.
Alpha Max - 10 Large Cap Stocks w/ Improved Metrics-V4 - No Hedge . Launched May 2013. Same situation as Value Rockets. Most of 2013 is OOS, in my view, and compares well to previous years, no risk of over-optimization identified this way. 2014 outperformed 2010. V5: too little OOS data, really, but as of this writing 2014 matches 2010. Pass on it [/i]for now[/i] just because of too little OOS compared to peers.
Tom SX20 launched Oct 11, 2013. Most of 2013 is in-sample. 2014 looks poised to outperform 2012, but not by much. From my point-of-view, weak risk of over-optimization.
Tom’s SX10 launched Sept 2013, so 2013 is mostly in-sample. 2014 excess performance: worst year ever. As a prospective client of this R2G port, I would let 12 months go by before studying it again. A lot of R2G ports currently fit this same yearly performance profile.
As the prospective client I can’t take the risk that it is, if other ports show better variability and have other performance metrics that are comparable to or even better than this one, I would spend my time examining other ports more closely.
Let me choose one. I will keep it anonymous. Launch date April 14, 2014
Sortino is 8.42! Yeah, sure, this port looks like the answer to my prayers. Last column is year 2014. But my down-and-dirty tool is screaming “high-risk” at me.
Sure, I’ll look at it again in a year, but I doubt I will ever subscribe to it.
Let’s look at a couple of individual designers. I appreciate Marc Gerstein more. Only one of his seven R2G ports is flagged by my method as at risk for over-fitting.
DennyHalwes. All four ports show the same profile. 2013 and 2014 are both OOS, for my practical purposes. 2014 is always the worst year ever for excess performance. 2013 beats out just one other year in all four cases. From my point-of-view, weak risk of over-fitting in all cases. I would look at them again in a year.
I appreciate your candour. I suspect there is something unique about 2014, not just over-optimization, that is degrading performance of many ports despite a variety of investment themes. Marc Gerstein seems unaffected by this phenomenon, whatever it is. But that topic is outside the scope of this post.
Your candour leads me to the nature of the challenge confronting the prospective R2G client: it’s the Used Car Problem from Game Theory. The vendor has more knowledge than the buyer, the buyer is at a disadvantage, so he must use indirect means to obtain information to support a decision. He doesn’t want to buy a lemon.
I do not disagree with any of this.
I employ several different tools to evaluate ports and support my subscribing decisions, most already mentioned in this thread. As time passes, I learn more and circumstances change, so I continuously adjust my portfolio of R2Gs. There is more material to work with now than there was when R2G started almost two years ago, more opportunity, but it is still buyer beware.
In conclusion, Denny started us off looking at ranking R2G ports. I look at the issue as one of what decision support tools do I have, and how do I use them, to design and maintain a portfolio of R2G ports in real-time.
I have some experience with neural nets (NNs) but probably not as much as you. I have a couple of comments…
While choosing too few or too many nodes will lead to poor results, choosing the “ideal” number will not give OOS results that are as good as backtest.
OOS results will degrade with time. A “rule of thumb” is one year OOS for every 4 years of in-sample data optimization. Beyond that, one is pushing one’s luck. So if a model was specifically optimized over the last 5 years then one could expect it would continue to “work” for about 1.25 years assuming no regime change (such as fast dropping oil prices).
Nodes are internal to the NN. The “ideal” number of nodes is directly related to the number of inputs. We don’t have an equivalent to “nodes” in our ports.
Ranking factors are “inputs” not “nodes” and the quantity of factors should not be judged by the same criteria as for nodes.
NNs are only as good as the inputs. i.e. garbage in, garbage out. This is why I gave up on NNs. If you can identify good inputs then why do you need the NN?
I think that one of the problems you and others have is that you believe that there should be a strong relationship between in-sample and out-of-sample results. Some model providers may design with this assumption in mind. But to make this general conclusion is wrong. If you truly like Marc Gerstein’s results then listen to what he has to say about backtest It is not outside the scope of this thread.
As for how you choose R2G ports, it looks to me as if you are choosing models that are performing the best. This can very easily lead to buy high/sell low. Just as an example, many of the smallcap models that performed exceptionally well the first six months after R2G started, subsequently flopped.
Further to what is being said, it seems that the Piotroski model has failed in 2014 when it did well earlier. I know my two are doing poorly, the R2Gs are not performing and neither is AAIIs. Can overfitting be also a general model design theory? Like factors based upon management effectiveness? Or concentration on a sector like health care or focusing on high dividend stocks in a low interest environment? Since we cannot see what is inside of R2Gs, it may be selecting stocks on macro trends that have worked well over the last 5 years but now the general market circumstances have changed. I personally think that Piotroski failed because it led to an over reliance on energy stocks. Does that mean that his concept was bad or just the types of stocks selected moved against it. Could be the same of over reliance of selecting small cap versus large cap stocks. Small cap were hurt this last year too. I guess I am saying that the fundamental ideas in the model could be fundamentally effective over the last 15 years, on average, but failed in 2014, screwing up the OOS results. I think just looking at 2014 could be misleading. I think developers do need to talk about more what are the drivers for the market.
