I’ve been looking for ways of reducing risk by improving position sizing and also Monte Carlo simulations (those who have read “Fooled by Randomness”) will know why Monte Carlo simulations are crucial to understand the maximum risk of a trading strategy.
i.e., the maximum drowdown we see in our simulations are just one possible outcome. A Monte Carlo simulator would look at millions of other potential outcomes to give a statistically significant confidence on the performance/risk data we see.
I found the following software that seems, at first look, pretty good to look at both areas by busing p123 historical transaction data.
I am afraid that I believe trying to evaluate “maximum risk” is an effort in futility.
Your worst drawdown is yet to come.
The point about “fooled by randomness” is that just because you have never seen a black swan doesn’t mean one doesn’t exist.
I am very suspicious of simulations that somehow re-order the data. This would only be valid if the data were random, yet clearly developing any trading strategy requires that the market is non-random. If you re-order the data you lose all of the subtle serial autocorrelations, what you get out becomes meaningless.
So I side with Stitts on this one, I really don’t see the point of monte carlo simulation, unless I have missed something!
Monte Carlo Analysis has some real drawbacks when trying to use it for analysis of a specific trading system with a fully defined set of rules. Monte Carlo Analysis requires the use of a random range of variables to calculate the different outcomes of a system. However, a P123 trading system has no unknown variables. Everything is defined. Re-run a Sim 1000 times and you get the same answer.
There is currently no Monte Carlo Analysis that can run P123 Sims. So we are forced to come up with a semi-equivalent method that can simulate the results. If we only want to evaluate position size, that is easy to do with Stefano’s Add-In.
So to use a Monte Carlo Analysis we have to define a real range for a set of variables. This is not the same as running different Sims with variations of buy and sell rules. Each different Sim is a fully defined independent set of rules that represent a distinctly different trading system, and Stefano’s Add-In is the perfect tool for that purpose. The closest thing that I can think of to a set of random variables with P123 Sims is the output of running a Sim over different starting dates and/or periods. Stefano’s Add-In is also excellent of that purpose.
To run a Monte Carlo Analysis properly you must define a range of real possible values for various variables in order for the results to be realistic. The results of Stefano’s Add-In run over many different start dates and time frames represent a real variation of a fully defined P123 Sim.
Say you use 100 Sims for this test. Then the output of the 100 Sims can define a set of variables as a range of % gains and losses for each stock bought for all 100 Sims. In addition you need the range of % winners, the range of % gain / trade, and the standard deviation of each variable. This data then can be used to run a Monte Carlo Analysis.
From that analysis you can get the variation in performance and the probabilities for gain and drawdown that is based on a specific set of rules.
Is this worth the effort? Are the results signifficantly different from just running the Stefano’s Add-In? I don’t think so.
Hi, Denny. A very provocative post, especially where you assert that “… a P123 trading system has no unknown variables. Everything is defined. Re-run a Sim 1000 times and you get the same answer.”
I understand your point, but there are other ways of looking at it. In simplest terms, if you generate a series of pseudo-random numbers using the same seed every time, you will indeed get the same result. However, change the seed and keep the population statistics constant, and you get something different. Here are two paradigms:
Paradigm 1:
The sim produces trades that can be characterized statistically (ie mean, sigma, min, max values of such things as gain, gain/day, holding time). The particular sequence of numbers in a sim can be considered to be one draw from a population with those statistics.
Paradigm 2:
The weekly or daily gains for the sim can be characterized similarly. The autocorrelation function is probably very important here (in simplest terms, telling if a profitable week tends to follow a loss). Again we have a set of numbers that can be considered to have been a single random draw from a population whose statistics can be estimated from the stats of the sim itself.
From either of these approaches, a Monte Carlo analysis could easily be performed. One particularly useful result would be an understanding of the statistics of drawdowns and monthly/annual returns. Sims provide only one point estimate of these very important performance measures, and can therefore be devilishly deceptive.
There is a re-run feature in Stefano’s Add-In which allows us run a system with fixed buy-sell rules multiple times by setting the % of symbols used by previous runs to exclude from each rerun.
Can we think this as the monte carlo analysis? I believe each re-run excludes different set of stocks so this actually works as a random variable which Denny mentioned above.
Intersting thread. I am wondering if there may be an issue of getting enough runs for P123 multi-stock trading approach to get statistically significant results from Monte Carlo analysis. In contrast to P123, even if trading just one index, it seems that there are many variations that need to be run to build up statistical result. If one expands universe to P123 type multi-stock universe and throw in wanting to vary both buy and sell rules, number of cases would seem to balloon astronomically.
Maybe I’m missing something (highly likely), but Monte Carlo Simulations are used to approximate the probability of certain outcomes of trials (simulations) using random variables. Here’s the rub as I see it… when we run P123 simulations we are doing so on known data, not random data. Past stock prices are not random. A Monte Carlo Simulation works great if you want to approximate the odds of coming up with 10 consecutive “heads” when flipping a coin. I don’t see how a Monte Carlo Simulation could be constructed to test P123 sims because the main variable - price - isn’t random.
Perhaps this will help. If you take your analogy, you would also say coin flipping isn’t ammenable to Monte Carlo. You can extend your argument to conclude outcome of the coin flipping is fully known since it happened in the past and you can measure which coins were heads and which tails so it is not random. But going forward in time, we don’t know what will be heads or tails for your coin flipping. Similarly we don’'t know future prices of securities. So both coin flipping and P123 are equally amenable to Monte Carlo analysis.
However, I think there is a practical issue of how to run enough simulations in P123 to get a measure of the system response beyond what we get with Stenci’s add-in. I think that was Denny’s point.
In principle, it should be doable to apply Monte Carlo to P123. It just may take a lot more runs.
For analysis of single security trading systems, it seems Monte Carlo is much more easy to implement. I think that is what mostly Aronsin did.
Similarly, I suspect that majority of the retirement planners that are available online, for the most part are just doing 1-dimensional random variable of a very simple system to predict whether one will outlive one’s assets or not.
But P123 systems are immensely more complicated. There is a large interplay between ranking system, buy rule and sell rules. I am not sure how one builds a tractible Monte Carlo analysis for P123.
Someone correct me if I’m wrong but I believe that Monte Carlo runs only work on real trades, not simulations. The reason being that sim’s are highly optimized and generally there will likely be some degeneration when you start trading for real. Thus it doesn’t make sense to try to model sim results. Second point I have to make is that you will need a fairly large number of real trades to make this practical - in all types of market. It doesn’t do much good to run Monte Carlo based on 10 or 20 trades done in a bull market and try to extract something out of it.
I don’t think my coin-flip analogy can be extended as you suggest KAR. Let me attempt to clarify the intent of my post. With the coin-flip experiment the potential outcome of each flip is known (heads or tails) and the probability of each potential outcome is known also (50% each). Also, each flip is an independent event. It’s easy to then conduct thousands of sims with hundreds of thousands of flips each. So while the outcome of each simulation is unknown, the components of the simulation (potential outcome of each flip and its respective probability) are known with precision.
But with stocks the potential outcomes (the stock’s price at a given point in time) and the probability of each potential outcome are not known. I don’t see how one can model a Monte Carlo simulation going forward without the potential outcomes and their probability being known. One would need to know the range of possible prices and the probability of each price.
I agree with you in sense that coin flip is much easier to predict and prediction is much more accurate than it would be for stocks.
For Monte Carlo to be used to analyse (anything), one needs to have a description we believe is adequate of the probability of the future.
One could argue that for stock price, it is impossible to predict the future price. Well that is true. But, that is ok. Because we do have an ability to make a prediction of probability of future price in P123. We are already making a prediction of what is is likely price rise of a stock having cerntain attributes (eg valuation ratios, trading volume changes, etc) in the future. So we have a tool at our disposal in the form of P123 to develop a set of probabilities of what future price will be based on those attributes. We are not using real trades to make that prediction. We are using historical data that covers a certain range of market time to make that prediction. We don’t make a prediction of exact future price, but we make prediction of which stocks are more likely, e.g more probable, to have a certain higher price in the future. Of course, we don’t get it right all the time. Far from it. If we thought there was no predictive ability of future probabilites of price to decide which stocks would have higer future price, we would not use P123 to guide our stock selection.
Since we have already a proven way in P123 of computing improved probability of picking stock which will have price increase one could, in principle, apply Monte Carlo to formlize and extend the analyse of this prediction if we desire. Of course, one is aware that the past historical data does not at all include all the important cases of what will happen in the future.
Here are a couple of other references that touch on aspects of Monte Carlo that may be of interest:
I’m with you for the most part KAR. I agree that we can use P123 to identify stocks that are more likely than not to increase in price, but how can the magnitude of the price change be estimated, or the probability of numerous different magnitudes? Is there a way to come up with a probability distribution along the lines of x% chance of a 0-5% price increase over some time period, y% chance of a 5-10% increase, z% chance of a 10-15% chance, etc. Wouldn’t that be necessary to construct a Monte Carlo, or did you have something different in mind?
25 years ago when Monte Carlo Analysis was first tried to be used in the financial and investment industries there were a flurry of papers touting it and a flurry of papers debunking it. The outcome of all of that was, although there were many elegant simulations tried, it fell out of favor as a useful working tool.
Monte Carlo assumes a degree of randomness or unknown elements in the data set that the analysis inputs are based from. Because of this randomness the outcome of future events is unknown. So the idea is to run an analysis that can put a range on the possible outcomes. The inputs to the analysis must capture the real range of the randomness. Based on that range of randomness, an estimate of possible outcomes of future events can be made. The flip of a coin has 2 outcomes. The flip is assumed that it cannot be identical each time, so the outcome is unknown. So there is randomness in the data set, and you can define a set of statistics that can be used to predict the probability of future outcomes.
Monte Carlo Analysis fell out of favor 25 years ago because the stock market data set was not random! It was the actual results of the outcome of specific events and investor reactions to those events that occurred in the past. Although you could define a “set of statistics” from the data set, you could not leap to the assumption that this represented the “randomness” of the data. It was simply the variation in the actual events of that specific data set. That assumption would require us to assume that under the exact set of conditions that caused investors and traders to react the way they did in the past, that they would react differently in the future, but within the statistical variation of our “set of statistics”. This was an assumption that had no justification to be made. The variation in investor behavior to an event and the variation in the statistics derived from the data set are 2 very different animals.
Fast forward 25 years later. We now have new and faster computers that enable everyone to have the ability to run Monte Carlo analysis. So now it is being offered by many web sites and trading platforms. However, even with 25 more years of data the stock market data set is still not random. Can we assume that under the exact same conditions investors will react significantly differently in the future? Can Monte Carlo Analysis actually calculate anything that is real and can be relied on? The dominate conclusion 25 years ago was that it couldn’t. I think that it still can’t. It is after all, the same analysis.
Our whole basis for following P123 Ports is that we feel that under the same circumstances investors and traders will react essentially the same in the future with some small variation or randomness on their reaction. However, this randomness has nothing to do with the “set of statistics” we might calculate from a Sim. That set of statistics captures the variation of the price changes of the stocks that the Sim bought. There is no correlation between that and the specific events or causes that made investors and traders change the price of a stock. I still think that the best tool for evaluating a Sim’s predictive ability is Stefano’s Add-In.
The real unknown is what will be the circumstances in the future. Monte Carlo can’t help us with that.
I totally agree with Denny here he has made some good points:
The main problem is that while we think we have a “model” the reality is that the stock market contains many patterns that are statistically not completely random. Some of those patterns may be known, but others may remain unknown, undiscovered.
E.g. I have known about trending for a long time, but until recently I didn’t know that very short term weakness/strength would likely be followed by a reversal.
If you chop up the data then you lose all of these subtle patterns. Who knows what other patterns exist. These patterns may have subtle effects on how the portfolio is traded, e.g. there is a limit to how cheap markets can become before value investors provide buying support, if you just take it as a random data series you could easily end up estimating drawdowns far higher than what is really observed.
Also, what statistics do you think you could gain from a Monte Carlo simulation? I would treat any such stats as highly suspect.
As stated in a previous thread, I think the best way to run these optimization sims is not by using random variables, but a fixed set of variables with (range-bounded) random values. As usual though, it was an idea that didn’t go anywhere.
