Recently reading some outstanding books on how to systematically develop and test trading systems:
1. Thomas Stridsman:
(a) Trading Systems that work: Very good book, with a whole chapter devoted to how to measure goodness of a model and then few chapters with example technical trading systems and followed by systematic analysis. Especially liked the idea that predicability of a system is more important than its profit potential. (In other words, strive first for systems with smaller standard deviation of returns, then look for max return). This reinforced the view I once read somewhere: Novices chase returns and risk eats them while professionals carefully look at risk and returns take care of themselves. The book systematically looks at the key parameters that can tell us the goodness of a system.
(b) Tradings systems and Money Management: is a followup book with more detailed analysis but the first one is very good.
(c) Stridsman’s 10 rules from Global Investors rules book capture many of these ideas in concise prose.
10 rules
2. Weissman - Mechanical Trading systems: has an excellent chapter on how to avoid overoptimization and deal with outliers etc. Has very good coverage of pschological issues in why people tend to stray away from their trading systems recommendations.
3. Tushar Chande - Beyond Technical Analysis Second Edition
Outstanding book, I would say the best primer on Trading systems and the most concise and mature and data-driven approach. Surprisingly, the first edition is good too, but does not have good recommendations on Amazon. If you had time to read just one book, this is the one. (I am not in any way related to Tushar 
This is definitely one book that will teach you how to think about testing your system.
4. O’shaughnessey: Invest like the best (is from 1994) but very fascinating introduction to how he went about reverse engineering the portfolio parameters of successful mutual funds. His latest third edition of “What works on wall street” gives some very good insights about market cap and about single factor simulations. I am sharing this because this is closer to the fundamental-ranking and yanking approach and reading it can provoke independent thinking.
OK, when rubber meets the road, what new insights did I get from these books, you ask.
Before I get to that, what are your answers to the following questions?
(a) Given two sims, which is better, one with more turnover, or one with less?
(b) Is a sim with 55% win rate good enough to make money?
Is a sim with 80% win rate prefered to one with 70%?
(c) My sim has a rule Mktcap > 100 and MktCap < 500. Should I use it? Why is this giving better returns than without the rule?
(d) Is a 5 stock sim better than a 10 stock sim? or vice versa?
and
now for the toughest and probably the most important question of all:
(e) How will you know if your model has stopped working? How can you distinguish whether what is happening is another drawdown or whether it is a breakdown because at least some market rules it exploits have changed?
My takeaways:
-
Dont just look at annual return, look also at std deviation of returns. Very unusually high returns can also be caused by curve fitting
(tuning the system ie changing its rules until it manages to pick the past winners). Do not tune rules to get rid of one loser here, one loser there. When you change a rule, it should improve most of the performance most of the time. Often, deleting a rule, accomplishes this. Which leads us to next one.
-
Fewer the rules the better. Every rule, every filter on the market and stocks, reduces the degrees of freedom in your system. For instance if your model works well only for small caps and is below market for large caps, then probably your model is fragile and likely to stop working for small caps, unless you can clearly explain to me why it works for small caps and why it can never stop working.
This also leads to other idea, rules that make sense are better than rules that use numbers like Rank > 99.25. Also, test each of your rule with a ± 10 percent increment, it should still work well, even though it may not be the best. For instance if your ranking systems makes you 120% annual return when Rank > 99, For Rank > 98 and Rank < 99 it should make at least 80% and for Rank > 97 & Rank < 96 it should make at least 60%. Why? You can never guarantee that a real 96-rank stock might be misranked as a 99 or vice-versa.
Buy Rules limit the universe of your choice, your catchment area, so for some reason, if that area dries up, you are left out in the open. It is like a lion deciding, I am going to eat only rabbits. All-weather sims are better than market specific sims, because long-term consistent profits is the key to growing and preserving the gains.
At least understand your restrictive buy rules and make sure you have only very few of them, and with good reason. Sell Rules, I think, can be more and might be the key to consistent long-term profitable systems.
-
More the trades in the sim, the better you can trust it, as far as a mechanical system is concerned. If a sim makes 500 trades in the last five years, it means it has picked up 250 stocks during different market cycles. The % winners, average gain expected, worst lost, drawdown and almost every statistic is more reliable and repeatable with this sim, than
a sim that has rebalanced only once a year. (Turnover 100%).
In general, of course.
This has been mentioned many times in this forum by Brian, Denny and others: The more trades a model makes, more reliable it is. In other words, like casinos, systems need enough trades to get the statistical edge.
-
Good models should work comparably well in all market cycles (every quarter, say). For instance, a system that does 200% return in 2003 and 150% in 2004 but 25% in 2002 is not that great.
-
Predictability is more important than profitability. In other words, first strive for a system that gives good returns consistently rather than great returns once in a while.
-
Same rule leads at the individual trade level. Make sure your returns are not due to few big winners, but due to lots of small profits and the elimination of big losses. Stridsman says it well: Strive for mediocrity. One of the forum members wrote this once very well: You cant count on these rare winners happening again, so remove them from simulation. In fact, thinking along the lines led me to cutting your winners short to make sure your system does not depend on such big winners to become profitable.
-
I am looking a lot more into the trade statistics now. Especially win rate, average holding period, av gain and av loss
and gain/day and loss per day and experimenting with using these to set the sell rules. Actually, I am comparing these with Rank sell rules. I am experimenting with NOT using Rank to sell.
-
Closely look at historical draw downs. Even if your sim had only a 22% drawdown back in 2002, try to walk through that closely week by week. Can you see, for instance, that it took 7 months for the portfolio to reach back to where it was? Do you think you would faithfully trade that sim week after week during those 7 months?
-
The most important thing is to be able to detect that your model is losing its edge. This is also probably the most difficult thing. Though after a lot of thought, it seems possible. The way, it seems to me, is to look at things like, how often do you get 4 consequitive negative weeks? What is the probability that you will get 4 days of 3% dropping in a month? How often can you expect 5% drawdowns? 10%? 20%? In your sim, are the worst drawdowns early in its life or in recent times? If you have meaningful study of these, and assuming that your sim is a tamed and well behaved machine churning out small profits and small losses at regular intervals, then you can learn to ignore ‘expected drawdowns’ and be alert when the behavior of the sim is deviant. (Finally, it is all about mean-variance and standard deviation, My Dear Watson!)
Without such a study, you will not have the reasoned faith to follow your portfolio during its casino-runs, which you interpret to be coaster rides.
-
Bottom line: Mechanical investing is more profitable than discretionary investing, provided you
(a) absolutely convince yourself through testing that the system is very reliable and
(b) follow every signal without exceptions, without trying to improve upon the returns of your model. The most hurting tendency is to selling winners earlier than the model says, hoping to catch them on a decline, but then not doing it and seeing that they keep moving up. Sims and successful discretionary traders make money by having on an average three kinds of trades - big wins, small wins and small losses.
Even though one should not trust sims that make 60% of their money from the big wins, I do think at least say, 30% of the money should come from top winning trades. A few of them are needed to wipe away the losses and give an extra edge. By selling them early, this opportunity is lost, leading to only having small wins and small losses. (The other even more harmful and the most prevalent tendency of letting losers stay for long to become big losses (anything more than 2% of overall starting capital per trade), I assume, most of us have overcome. If not, send me a Selfaddressed envelope (so I can share my experiences).
Consistency in strategy (no matter whether it comes because of you have what Buffett calls temperament or because of following a trading system) and making many many small bets and playing for the long run are the keys to making money. The exact opposite behaviors of chasing new tips, strategies and systems, making too large bets that either way shake up emotions leading to overtrading and oversize bets and dropping in and out of the game when one feels like it are the keys to losing money which all new-comers are blessed with and most traders and investors may not overcome in one life.
My appreciation for portfolio123 has tripled - (you can see how I have started thinking in quantitative terms 
- after reading all the books, since this is one platform that can (a) help you get the statistical edge using FUNDAMENTAL analysis and (b) help you realize that potential with a trading system simulation and (c) test your hearts out with real data.
I apologize for having said so much of my personal reflection in this thread, and realize that for many of you reading this, this must be child’s learning (‘Welcome, Ravi, Who said the markets are easy?’). Still I wanted to share for those who might benefit.
Ravi