Are there instructions for how to use the optimizer anywhere on this site?
I could not find any documentation in the tutorials and the help key does not work for this function. All I could find was Marco’s post from June 1, 2012, that discussed the beta release. This post was too brief to be instructive as I am not able to get the optimizer to work properly.
Thank you for the help.
I have a simulation… I click the optimize button… I make changes but usually only to one field at a time. For example, I run my system against all the ranking systems. But this is the only variable I change. Becuase if I change the universe as well (for example) it will start doing combinations.
You will have ranking system A with universe X and Y. Then you have ranking system B with universe X and Y.
Add three variables and the combinations multiply with a lot of wasted sims.
When I get a setting I like, I click it and modify that one by changing one other variable.
To run it, click Generate Permutations and then click run. There are settings where you can see them with or without charts.
Is this sort of what you were asking for or am I off in left field?
Kurtis, I appreciate your response and your excellent articles. I understand the idea of permutations but am looking for a pictorial step by step guide of how and where to input variables to get the optimizer to work. Portfolio123 provides this type of documentation for their other applications.
Hi
Did you ever track down any written instructions for the Optimizer?
David
No. This site is very powerful however the documentation often lags the development of new tools/commands. Often only through trial and error or the kind explanation by another member do I understand how to properly use a new tool/commands.
A few weeks ago I realized that there were two optimizers. One is for simulation optimization and is relatively straightforward. The second optimizer (which I wasn’t aware of) is the ranking system optimizer. I am currently writing a step-by-step tutorial for this optimizer at a very basic level. This is not “blessed” by P123 but I don’t think they will mind given there is no documentation.
I’ve written the first two sections and I should have the whole thing done in a couple of days if I don’t get distracted.
Steve
http://stockmarketstudent.com/stock-market-student-blog/preparing-piotroski-for-optimization
Thank you.
Next post in the tutorial.
http://stockmarketstudent.com/stock-market-student-blog/setting-up-the-ranking-system-optimizer
Great instructions. Thank you.
Scott
Thank you Scott.
Getting into the meat of this tutorial now.
http://stockmarketstudent.com/stock-market-student-blog/ranking-system-optimizer-the-first-iteration
Steve,
Thanks! Would not have thought of randomizing. Would have tried generating permutations manually. Or in practice go up a little (in weight) on one node see what happens, if good go up more, if not go down. Repeat. Kind of like my job as an ophthalmologist: which is better 1 or 2, up or down…This method is akin to a strategy called reducing the entropy. Perhaps not ideal for a large number of nodes and certainly not automated in any way.
Your strategy is similar to (same as?) what is called a genetic algorithm, I think. That is pretty cool that you know how to do that. Let alone figuring out and sharing a P123 implementation. I guess eventually you arrive at the best permutation and the new permutations that you generate are never an improvement. Any experience with how long it takes to get close to this point?
Thanks for the spreadsheet!
Jim
J - it should only take an hour or two before the node weights can’t be further optimized.
Steve
That is quick! Was just thinking with 10 factors somewhere around11^10 permutations (changing weights 10% each time starting at 0%).
Thanks!
Jim
Hi Steve:
As a potential R2G subscriber, I don’t like to hear about such extensive measures to optimize. At best, it virtually guaranttees out of sample results will be less than back tests. At worst, the result will be a ranking system or buy/sell rules so finely atuned to “random noise” that very poor future results can occur.
There may be several ways to detect variations that give improved tests due to curve fitting to market noise of past data rather than to persistent characteristics that will work in the future. Here is one I especially like.
Divide the data up into several time periods (for example, 1999-2002, 2003-2005, 2006-2008, 2009-2011, 2012-2013). Apply each variation to all the time periods. Keep a variation if it improves each the time period (that is a definite keeper). Also keep variations that improve all but 1 period. Variations with less consistent improvements are suspect. If a factor is added with hopes it would reduce drawdowns, then one might be happy if only improves the three periods with significant market drops. I’d be very suspicious of a factor or variation of a factor setting that improves results for just one time period.
For fine tuning that does not involve market timing, Odd and Even universes can be used to double the number of available confirmation points to 10.
Best regards,
Brian
I must say also that the high level of optimisation of the rank nodes surprised me also. But, I assume you are just demonstrating the process,
.
Brian, Gary and all,
I think there is a subtle misunderstanding of optimization. Once you decide to use a factor the optimized value is your best guess as far as the best value going forward, I believe.
If you had 2 values you could choose, would you take the value that gives the worse returns? If there is no other information, you should never choose the value that gives the worse returns. Taking the worse value or not doing the backtest in the first place does not help going forward.
The problems come when you think you will actually get the returns of your optimized backtests (as Gary correctly mentions with regard to R2G ports). This is a big problem and I get it. I have a live port with only 5 factors in the rank and one buy/sell rule: RankPos > xx (sell). For those 5 factors I use the optimized value. I then screw up by adding a market timing buy rule and a hedge rule. The problem is more that I added those rules than that they are optimized.
Great discussion!
Jim
Brian - As the optimizer function exists and appears to interface to a spreadsheet, I suspect that there are a few people using this functionality to create very impressive backtest results. Otherwise why is it there and why is P123 quiet about how to use it? I have provided the tutorial to level the playing field so that anyone who wants to use it can do so.
Now I didn’t optimize any of my R2G models in this fashion and I consider this optimization feature to be “experimental”. If I decide sometime in the future that I can provide reliable and superior results using the optimizer then I may consider introducing an R2G model using its output. However, I don’t believe in fixing something that isn’t broken, nor upsetting subscribers.
There are a couple of deficiencies I see with Ranking System optimization. One, is the limited insight into what is really going on inside buckets. Your solution is to test separate time periods and that is a good solution. The optimizer allows you to do that but I haven’t tried it yet so I don’t know if it works. The second deficiency is that optimization comes at the expense of diversification. Every stock factor that I include in a ranking system works very well on at least 3-6 separate sectors. The optimizer doesn’t care about such diversification. I find that I may be able to creat high performing models but have to relax or remove the SecWeight<xx buy rule.
Jrinne is right on with his view of how optimization should be used. If your stock factors are exhaustively tested before inclusion in the ranking system then I don’t see where there is anything to lose. The problem comes about when bacltests are treated as “performance”.
Steve
Hi Steve:
I’m glad to hear this type of extreme optimization was not used for your current R2G portfolio.
Brian