Testing robustness with the Random function

There’s been a lot of talk about robustness test, Monte Carlo sims, etc. Has anybody tried using the Random function in a buy Rule ?

For example in a 10 stock simulation simply add this Buy Rule:

Random < 0.9

This rule will throw out 1 rec out of 10 on average and produce different results every time the sim is run.

See the image below. It’s an Optimizer run that has exactly the same parameters that is basically generating a Montecarlo-ish output. Annual returns range from 25-33%, DDowns from -33 to -45

For more “noise” use Random < 0.8


Wow! That’s pretty cool!

Thanks Marco,
Denny

Here’s another thought. When you launch an R2G , we automatically run 10 sims by adding a random function in the buy rule. We will then have 10 values for each statistic. In the R2G pages we could show

Ann Return X % stddev Y%
DDown A% stddev B%
Sharpe C% stddev D%

Marco,

This is a very good idea to play with. Thanks. I will begin adding this to my ‘robustness’ checks. I may also play around adding a ‘random’ node to the ranking system at various weights (5%, 10% and 20%) and seeing what happens.

Play with various portfolio sizes and see the effects of introducing a ‘random element.’ My bet (after testing many ‘random rules’) is that it can have a much larger impact on small number of stock systems. I can frequently boost a 5 stock sim by 10-20% if I play for an hour or two with rules on the buy and sell side.

Should be an interesting learning experiment.

There is not one ‘magic bullet’ here, but this is a good idea to begin working with - and something I never knew existed / thought of.

Best,
Tom

Marco,

I don’t know the particulars yet. Haven’t thought enough about it.

But…I do think automating some form of random element ‘robustness’ checks in R2G’s makes sense. However, without enough trials, ‘random’ noise can just be random noise. So, don’t know if ten trials is enough. Something like running 20, 5 year periods and even odd tests and then this 10 trial ‘random’ run. Or just one of them. And reporting the aggregated stats. If it can be automated, so it’s a standard process that’s the same for all systems - and that happens when we convert to ‘R2G’, that would be great. That’s one goal. Aggregated stat’s of multiple runs are better predictors for investors. And give a better range of outcomes for people to know when the system’s not working anymore.

Thanks for thinking about this stuff with us! :slight_smile:

Best,
Tom

I think an R2G page needs an unbiased “Robustness Check” that is completely out of the control of the designer. I like the Random one because it’s relatively easy to do:

When designer launches (Opens?) a model WE run 10 sims with one extra rule, collect stats, calculate std deviations and we’re done.

But the other elephant in the room that the above does not address is market timing.

All,

Thanks!

Like that a lot. Wondering what the optimal number is. Random < 0.9? Seems that one would still want to have good returns at Random < 0.75 or 0.8 for any traded sim.

Edit: fun to see probability at work. Sometimes the returns are improved.

Jim

Great idea Marco.
An observation - To consistently filter out 10% of the available stocks in a given sim, the Random rule should be the last buy rule since (I assume) the buys rules are executed in order. For examples:

  1. If you have the a liquidity rule as the first rule and the random rule as the 2nd rule, then the random rule will work as expected and filter out 10% of the universe that is left after the liquidity filter. You should get the same number of stocks in each rebalance (my tests show the # of stocks is actually off by up to .5% ie 90.5% or 89.5% remain after the filter).
  2. If you have the random <.9 rule as the first rule, it would filter out 10% of the universe. Then the liquidity rule would filter more stocks but the % of stocks that were already filtered out by Random would change in each rebalance period for the sim. So the number of stocks available in each rebalance would differ. An extreme example is if the random rule happened to pick all microcap stocks. If I had a liquidity filter set high, I could end up with very few stocks for that rebalance if the random rule was placed before the liquidity rule.

I did a quick test for this using the screener and it does work this way. Not a big deal, but I would want to put the Random rule last so I get the same # of stocks each time. Others might prefer to do it the other way to add another level of randomness to the test.

Interesting thought Dan. I just tried it but didn’t make too much of a difference from what I could see. It might depend on the Universe and liquidity rule though.
Steve

Marco:

Great idea to add the std to the stats.

However, 10 runs is not enough statistically speaking. 30 is usually considered the minimum. If it could be automated so as not to take much staff time, 100 runs would be ideal. But if 30 is not possible due to staff time, 10 would be much better than nothing.

Since the minimum number of stocks for a R2G portfolio is 5, I’d like to see the random rule set so that on average 1 stock is excluded. For my own portfolios I often find 1 stock (sometimes 2 stocks) that can’t be bought due to insufficient liquidity on the buy day even though I have a minimum liquidity filter in place.

Brian

Your correct Steve - it wont make much difference unless you have other buy rules that filter out a lot of stocks from the universe you have selected. Even then, its not a big difference. I guess the best thing to take away from this would be that you should be consistent and place the Random rule either first or last in all the sims you test if you want the results to be comparable. If you run a large number of iterations, it probably would not matter where you put the rule.

Marco - all:

I do not know what a “random” buy rule does. If it throws out 1 recommendation and replaces it with 1 randomly (out of the universe?) selected position, then it is useless in my opinion: If I deactivate my model rules partially, then it will not work as well as before. I don’t need this test to know that.

A good robustness testing instrument are “even” and “odd” universes, in contrast.

If you run 30 sims like this you can just return the median values for CAGR, sharpe etc… no need for the detailed stats of each run.
That would be a big improvement in terms of the reliability of the stats shown.
Also if you can use the median rather than the mean, it’ll protect the stats from being affected too much by an abnormal/improbable sim.

I’d like to see the CAGR for the portfolio as optimized by the designer as well as the “median” CAGR from 30 sim runs. I think these two numbers will be much easier to mentally compare than have a single CAGR number with the standard deviation of that number.

As aurelaurel notes, it would be important to use the “median” rather than the “mean” or average.

Since this is designed as to test robustness, I think the randomized factor should at two levels.

random < 0.9 as originally proposed (for 30 sim runs)
random < 0.5 to create more “noise” and thus produce a greater stress test (for a 2nd set of 30 sim runs).

If two sets of sim runs are consider too much, then I’d strongly urge the robust factor be something like

random < 0.75

Brian

The random function won’t work for all simulations, in particular those sims that use Rank<x as a sell rule.

Random < 0.5 is too stringent.

Steve

I was assuming the random < x was only going to be a buy rule.

Using it as a sell rule makes me wonder about unintended consequencies such as reducing the average trade gain and average holding period and increasing trading costs, etc.

Brian

Brian - I was refering to Random < x as a buy rule. This concept is great if all other rules, buy and sell are relative. For example, if you have a sell rule Rank < PrevRank(1)-2, it will work. Rank < x as a sell rule won’t work so well because it is using an absolute limit instead of relative. I found this out when using exclude from previous runs for degradation testing.

Steve

All:

I did a study of the Random < .X buy rule for one of my sims testing X from .99 to .5. It looks to me like the random rule is just lowering the cash invested level, at least for my sim.

The rule prevents stocks from being bought which limits the selection that the sim has from the ranked stocks. Since the 5 stock sim I tested has a buy rule of Rank > 99.5 the major impact of lower values in the random test was that the sim ran out of stocks ranked at > 99.5, This resulted in lower investment levels which lowered the results, which one would expect.

The attached spreadsheet shows the tests and results. A quick glance at the cash invested in the optimizer results shows the lower levels of investment. I also copied some cash invested results below.

I guess I can conclude that my sim is reasonably robust since eliminating various levels of stock picks did not drastically lower returns which would be expected if the sim was curve fitted to specific stocks.

Glenn


Sim 8 Random Tests.xlsx (1.29 MB)


No Random test.PNG


Random less than pt 9.PNG


Random less than pt 8.PNG