Over-Optimization

Models
61

David,

de Prado’s book has a chapter about position sizing (Advances in Financial Machine Learning).

I do not think his method could replace or refute the Kelly Criterion in any way. I see your points and much (all) of what you say is provable. To be valid de Prado’s method would have to be consistent with yours—which I cannot confirm.

De Prado’s method has the advantage of updating the probabilities (e.g., with a logistic regression). For example, you may want to update the probability of excess return as the rank changes.

The chapter is very difficult and I could not use this even if I were strongly motivated.

I think you probably could (if you considered it useful). With your interest in the subject, you might check it out. But it is difficult by any standard.

The formulas will not copy and there is more than one formula/algorithm/graph so images are not practical. You will have to get the book if you want to pursue it.

-Jim

62

I absolutely find that logistic regressions provide good fits against my ranking systems.

And, yes, non-stationarity is absolutely an issue. Actually, I don’t know of a really good way to deal with it because I don’t have a great way to reconcile factor recency, reversion, and decay.

Using Bayesian weights to update beliefs over time probably has merit, but I haven’t seen any convincing empirical research on factor momentum/rotation. This may be due to the fact that I spending my time on other things lately. But if anything, my best guess is that factor styles experience both cyclical momentum and decay.

I don’t think we’ll see a big upswing in our models’ performances due to momentum and reversal factors at any point in our lives. Value has a shot, maybe, but that depends on how well it is defined and measured.

63

Yes, and this cannot be assumed when comparing ETFs, models or portfolio managers based on historical data.

64

Yea. Because the assumption of being ergodic (and adequately) mixing may not be right.

BTW, mixing implies that it is ergodic so mixing in the most important (necessary) assumption. From Wikipedia: “…strong mixing implies that for any two possible states of the system (realizations of the random variable), when given a sufficient amount of time between the two states, the occurrence of the states is independent.” Or asymptotically independent.

Perhaps, you could if that assumption (of adequate mixing) were rock-solid.

Uh, that is if the tails are not too fat. But they PROBABLY are not too fat, so I do not think we can throw the idea out based on this (without further evidence about the tails).

I trust David to handle some of these details with conservative assumptions of returns and variance where they are needed. Not that I use Kelly criterion: I don’t.

-Jim

65

Gotta be honest, I’m a little lost on the mixing. Part of it may be because my sigma algebra is a bit rusty. However, the numerical examples given on Wiki are nothing more than logic problems, so I think I grasp the core intuitions.

Another aspect is that mixing implies ergodicity. My first principle on this is that the relationships between returns and independent (predictive) variables are non-stationary over time. Any assumption of non-stationarity implies that ergodicity is a non-starter.

But if we’re just talking about returns themselves, then I understand why mixing a bunch of off-normal distributions would result in an approximately normal distribution. But that’s beside my point.

Jargon aside, I base my investment decisions and models on the best information I have available to me. And sadly, yes, my best models do implicitly assume ergodicity. I’ll have to go back to the drawing board in a major way in order to build good models of time varying relationships.

But even then, what else would these better models tell me other than that the data is losing predictive power? Hopefully, something more useful, but I haven’t seen any research that compels me to believe otherwise. Yes, we all know that cycles/rotations are important market drivers. However, economic and market cycles tend to metamorphicize soon after they are publicly revealed. Heroic efforts or supreme intellects are needed to stay ahead of the game on this stuff. I’m afraid I can’t offer either of these at the time.

I know I may seem like the Debbie Downer of these forums, but I see myself as a realist. For me, looking at the glass being half empty versus half full never made me feel any differently.

66

David,

If your port does not act like the sim then you can say (with certainty) that there was not adequate “mixing” (somewhere). You probably will not need to know the formal definition to say this.

Of course, we like to assume the port will behave like the sim but that is an assumption. Usually an assumption of “adequate mixing” in statistical parlance (more so than the assumption of ergodicity). Even if you are using the STATISTICS OF THE BUCKETS you are probably making this assumption, implicitly. Again, no need to know the formal definition.

I only posted about this to defend your view that the math behind the Kelly Criterion is valid in theory. Frederic brought into the discussion the fact that the future may not reflect the historical data which relates to ergodicity and mixing I think.

I leave it to you to defend the Kelly Criterion from here. Although, it would be hard to make a consistent argument for it at this point. I am coming around to Frederic’s view after reading your last post. Your post was more a convincing attack than a defense of the Kelly Criterion–at least for any practical purpose.

-Jim

67

Have not read the whole thread, but here are my 2 Cent:

1: Keep it simple, especially on the buy and sell filters. Ranking stuff can Have a lot of criteria, hard to overfit here, but I also like
Systems better that Have less then 10 ranking criterias (Just in case!)

2: Trade factors that are backtestet at least back to 1870 by academics. There conclusions are usually off, but their backtests are
usually longer then back to 1999 (which is the Limit of p123): momentum, value, size effect, Quality factors, liqudity etc. The Gold
is to be found in those papers and they are free!

3: Use factors that are bound to emotions, e.g. that are hard to implement, those factors will work in the future, because humans do not Change

4: The capital curve of your System should not react radically (but smoothly) to changes in

  • number of stocks (1 / 5 / 20 / 50 / 100 / 200 / 400)
  • time period backtested
  • across industries
  • across small caps, mid caps, big caps, nano caps

5: Use Even ID (evenID = 0 and even ID = 1 it cuts your universe in half)

6: Real Time Test of at least 1 Year

7: Scale into your System (e.g. take 12 Months and invest 1/12th at a time), you will make sure that
the first DD does not kick you out of the System and after that (if your are lucky) you Play with the “Banks Money”

8: Use Restricted Buy List (exclude the stocks from runs before): also here your capital curve should only
get worse smothly not radically

9: Be carefull with restricting your universe, best case use all 10.000 stocks first and then you might work you
down on smaller universes (e.g. industries etc.) Also here your capital curve should react smothly to your
changes

  1. Use only small parts of leverage (max 10%)

  2. Time the market: your System should do fine in 90% of all market Regimes, but go out if the earnings
    of SP500 trend down (e.g. lower then 20MA etc.)

  3. Volatility is not risk, robust Systems Have regular DDs, 20% is totally normal impossible to avoid, if you do not like it work with e.g.
    only invest for example 50% of your capital. A capital curve without DDs is a pipe dream, that everone wants
    to reach, but it is not reachable, if your backtest Shows a capital curve without DDs
    your System is overoptimized. Work with psychological Coach or with your self to understand that
    regular DDs are normal and recocnise your emotions but manage them!

And:

Do not use statisticall Tests like P-Test, Correlation, Regression etc. IT DOES NOT WORK because the assumption of those Tools are
that your numbers / Returns are normally distributed. THEY ARE NOT! You want to catch the fat tail of the Distribution, if you
use statistical Tools that do not understand thos fat tails, they will mislead you.

“If somebody talks about beta, zip your pocketbook” (Warren Buffet!)

Best Regards

Andreas

68

NOT PIT. Testing over same period leads to same results: should not bother.

Would take just a little research to find the right way to do this. de Prado writes extensively but maybe would have to buy a $15 book.

-Jim

69

Andreas, could you give an exemple of the factors that are “bound to emotions” that you prefer?

70

yes, all factors I am using

1, small caps or Micro caps: usually avoided bc they are not Story stocks you can talk About with your neibour

2, volatility is interpreted as Risk, but that is what our emotions do with it, we say, I do not like a volatile stock (which is not
equal to a volatile Portfolio) or a volatile Portfolio. If you do not leverage Risk does not equal volatility. What you
should fear is a low return in 10 Years, not a 20% DD today (Important: if your Interpretation of volatility is not Risk, then
do not leverage, bc, then its going to be Risk!).

  1. Momentum (Price or fundamental Momentum): nobody likes to buy at the all time high (of Price or earnings), but it works.

  2. Komplexity: People love to reduce complexity to complicated: “I am a value Investor”, “I am a technical Investor”, I am a macro Investor".
    Well what About to embrace all complexity and come up with a trading System that combines factors (for example value with Momentum etc.):
    a lot of People do not do this bc they fear complexity and put them self in a box and figth with People out of the other boxes (“Technicals do not
    work” “funymentals”)

  3. value: hard to buy stocks that are beaten down in Price but have high value (just do not use Price to book, because
    there are to much intagibales in todays Software / IP World, I use FCF / total assets).

  4. low volume trading stocks: almost everbody likes to trade liquid stocks, put in an order and get your fill in 1 Second.
    Well, low volume stocks print very nice Alpha (though volatile!) if you are Patient entough to
    work with GTC orders and buy at the bid and sell at the ask.

  5. following a System: I know a lot of good System Designers, but they simply can not follow their System.
    “This time its broken”…

Whenever I Show my strat to (even experienced) Traders or Quants they come up with:

  • Small caps are dangerous, how can you trade stocks that you even not know their names of and Forget
    the Symbol after you have put on the trade?
  • Volatiltiy is Risk
  • low volumne stocks are dangerous, bc, I want to be out of the market or in the market fast, you can not liquidate
    low vollumne stocks fast
  • this System is way to complex, “I am a Momentum Trader” (put yourself and others in boxes)
  • I am discreationary better then a Software based System, it is impossible that Software is better then me
  • the System will break at on time in the future (thats my favorite, especially bc, every factor I use
    is backtested by Fama and French back to 1870)
  • it is impossible to trade a stock where you got 10% of the daily volume

Those emotional reactions Show me: this works and it will work in the future, because
it hacks human emotions. Every robust System is an emotions hacker, emotional behaiviour
does not Change in the future, People do not Change, so it will work in the future.

It might not work in the future, if they Change the structure of the market (e.g.
restrict small cap trading for private Investors or some other cracy ideas etc.), but
in todays rules, it will work.

Will it produce Profits every year. No. But it will produce stellar Profits every three years or even better
every 10 Years. → goes against human emotions, hard to implement
Will it be not volatile. No. → goes against human emotions, hard to implement
Will it be easy to put in the next trade within a 20% DD. No. → goes against human emotions, hard to implement
Will I be famous, bc I can my neibour I just bougt Facebook. No. → goes against human emotions, hard to implement
Will I be the nerd who trades low liquid value Momentum Quality nano caps. Yes. → goes against human emotions to be the nerd
Will I buy a 911 Turbo S with 20% of my Winnings after tax in 5 Years. Yes :wink: → goes against human emotions, bc then
they do not understand what your are doing and you are making a ton of Money. There is and will be a lot of social
pressure to get “back into the line” (espcially in Germany, the Country of People who hate complexity and love to
reduce it)

90% of your sucess is Managing your emotions and understanding that you hack other
People emotions. 10% is the ability to create a trading Systems.

Be prepared to swim against the river….

Regards
Andreas

71

Thanks a lot Andreas for share this, your ideas are pure gold.

72

thank you!