All, I think that when my port has an “Average Return” less than the sim’s “Average Return” it is probably less by a constant amount rather than a coefficient. In other words if the sim has an average return of 4% per trade and my sim has an average return of 2% the real world is somehow subtracting 2% rather than multiplying my returns by 1/2.
This would be the case, for example, if I’m not using the correct number for slippage. Say I calculate 1% for slippage but the slippage in the real world is 2%. That error in slippage is being subtracted from the average return.
I’m certain many of you are saying: “well yea, duh.” But for me this is pretty important if basic. I have a sim that has an average return of 2.84% per trade with an annualized return of 82%. The real world port has an average return of 0.92% with an annualized return of 38%. So in round terms the port underperforms by about 1%. Obviously, the days held are different in the out-of-sample data compared to the sim.
So here is the thing. I can modify the sim (longer holding period with totally different sell signal) to have an average return of 5.24% and an annualized return of 70%. Now if the real world port of this sim underperforms by 1% (average return underperforms by 1%) the port can be expected to have an annualized return of 56.6%. This would be true if this new sim and new port have the same average days held–the sell signals are pretty concrete.
What is wrong with my thinking? I would rather be a fool here than in the market (all things considered). I plan on changing my port Monday.
Jim,
From your past posts I understand that you trade on FolioFN using their window trades. I think there are some slippage issues trading small cap stocks that way. I have not had the time to collect and analyze the data but using window trades in high turnover small cap portfolios has resulted in some serious under performance for me. So I have tried to come up with lower turnover higher cap portfolios.
I had posted this in another thread a while ago.
“I think there is a major splippage issues with small cap stocks in window trades especially if the same stock is also traded by one of their “Ready-to-go Folios”. I traded one of their Zacks folio (trading mostly small cap stocks with over 50% turnover each week) for about a year. My return was less than half of their “calculated performance” during the same time period. Also some of these folios have a lot of “unneccessary” turnover, a lot of stocks sold one week were bought next week repeatedly. I think this is due to these folios being generated automatically by screens. I used only window trades and most of the time it took almost the whole trading day to get some of the orders filled. But these folios are free (I think even non-Foliofn people can see them) and who knows how many people are trading them. I have come to the conclusion (although I have not had the time to analyze the data) that to be successful with window trades, one has to trade higher market cap portfolios with medium to low turnover. All of their weekly folios are balanced on Monday so maybe trading on another day might also help. I have been running an experiment with trading their folios on a monthly balance and I have been lot more successful. But this is not one folio, I re-evaluate the folio every month based on certain criteria I have from the previous month’s performance.”
Hope this helps.
I have been seriously thinking about opening an account with IB primarily for Canadian stocks but also try to trade small cap portfolios.
KJ, thanks. I looked at my slippage for a large number of trades. See the fourth post in this link: here.
So for this universe I always use 0.35% as my slippage for “Next Open” in sims which turns out to be pretty close to variable slippage with “Next Average of Hi and Low.”
I don’t know if the helps: your universe may not be the same.
Can you clarify this question? You’re right that it’s likely a ‘slippage setting issue’…
However, in your past post on Foliofn that you reference, you say that you analyzed all your trades to date at FolioFn and…
“I can be 95% confident that the slippage for the population is 0.35% or less (compared to opening price).” You don’t still feel that way then?
Is there a large amount of trade data you haven’t analyzed? If yes…you’re slippage is likely too low. Using window trades with FolioFn on small or micro caps is not something I’ve done for years. But, I was seeing some huge slippage trades (6% plus on some trades).
If not…if you’ve run the numbers on all your trades and know the slippage from the next open…what exactly is it that you ‘don’t know’…?
Are you skipping buying of recommended stocks that ‘gap up?’ overnight or in the very early minutes of trading in the live model, but not the sim? This would be a second potential source of the largest difference. I usually have to skip several buys each week and find replacement stocks. A sim can potentially still buy these stocks at prices I can’t.
Are the two systems buying the same stocks? If you have a small number of holdings…say 5 (or even 10)…a single trade can make that difference in average (even over 1 year). Given that the sim and the port. may be buying slightly different stocks, this could be the difference.
As far as turnover…people seem to make money (or claim to) at all levels. But, I try to trade 500% turn or less on most of my actual dollars invested / live money systems. A few go up to a 1000%. One or two to 1200%. That’s just where I’m comfortable. Occasionally ‘play’ with 1800% to 2000% turn systems. But never more than 3-5% of my money there.
Tom, Could be slippage (some), and could be that the stock is gapping up slightly so that looking only at slippage compared to the open is not the whole story (more likely as you detail). But I was trying to generalize to a situation where the reason for under-performance is not known. I was wondering if, in general, the under-performace can can be mitigated by increasing the average return through a longer holding period say.
Are you talking about returns of the sim vs. port over the same time period, i.e. not confusing the issue with “in” and “out of sample” performance?
It is not mathematically possible to relate average profit per trade with annualized return like this. This would only be true if every trade was exactly the average.
The annualized return is path dependent. To show a concrete example, imagine a 1 stock port with 2 trades, one -5% and one +5%. The average return is 0%, but the total return is negative:
(100*0.95) * 1.05 = 99.75
I.e. less than you started with. It becomes an issue when several stocks are in the portfolio because multiple simultaneous bad trades will cause the portfolio to drawdown significantly from which recovery is harder.
Also be careful about assuming that a higher simulated profit per trade will lead to a similar result in real time - the market is not that perfect!
The path dependancy is a very good point. This is definitely one case where some hi-turn pattern based sim’s (if you can trade them well)…can do much better. It can be possible to achieve very high win rates with some very short-term patterns. But the win rate falls significantly over time (some factors exhibit very hi rate of time decay).
So far…52 higher turn R2G’s with more than 100 days since launch (average of 239 days) are not showing any correlation between prior to launch sim results and out of sample. But…the 25 R2G’s with 17 or more stocks and more than 100 days are showing a reliable correlation (some systems are in both categories, mostly yours). Possible issues ‘corrupting’ this simple analysis…
The second group’s been around for 316 days…so this could be skewing the correlations (I have no idea how P123’s ‘combined’ formula is working). The market returned about 2-3% (depending on bench) over the 2 additional historical months that the larger number of holding sim’s have been around. So…that could be up to 10% of the difference or so. Maybe more.
Can’t say that underperformance will be lessened by improving the average return per trade unless we define and clearly understand the sources of system underperformance.
If the source is slippage, then it likely helps.
If the source is trading different stocks from sim to port and random variation of very small port sizes, then increasing holdings or some other solution may help more.
If the source is ‘gapping up’ rapidly after opening at rates that can’t be achieved by trades…then better backtests and live port. tracking that takes account of true fill prices and/or more holdings may help more. And more holdings.
Etc.
Higher average returns / trade isn’t always better, depending on underlying factors being traded. As the tradeoff may be much lower win rates.
Seems there may be some lack of clarity around what, exactly, slippage is. The Investopedia definition, “the difference between the expected price of a trade, and the price the trade actually executes at,” is probably as good as any.
Bear in mind that p123 sims/ports can be set to pick up actual prices at the open or the close, or a imputed price based on the average of the open and the close. But there are many prices that occur throughout the day. If you execute a trade at 9:42 AM, you can expect the 9:42 AM price. If the 9:42 AM price differs from the 9:30 AM price, that’s life not slippage.
Slippage would occur if you expect a price of say, $6.73 based on the 9:42 AM market but wind up with a different execution, typically because your trade impacts the market. With large cap stocks, the impact of your trade is not likely to be visible. But with micro and sub-micro cap stocks, it can be considerable to the point where no p123 slippage assumption can work for you.
The only answer is common sense. I deal in sub-micros all the time and have often seen the market extract frightening penalties for those who don’t think this way; the worst “slippage” I can recall having seen within the past year was about 65%, yes, that’s 65% --no typo – because one or more idiots shoved grotesquely outsized orders into the market for a decent stock without having thought about what they were doing and that’s what it took for the market makers to attract enough action on the other side to fill the orders the idiot(s) failed or refused to modify. (By end of day, after the idiot’s trade was filled, the market went back to its regular range. Seriously. They really do need to create a Nobel Prize category for Trading Stupidity.) While 65% was an unusual, to say the least, episode, I have seen many more instances of so-called slippage in the 10%-25% range probably because of careless use of quant models (possibly on R2G,but bear in mind R2G is not the only quant game in town), a trend I fear is getting worse, not better. In fact, in the low-priced stocks newsletter I write, I advise subscribers to worry less about Mr. Market and a lot about Mr. Model, and suggest that if it appears Mr. Model is working a stock that interests you, hold off for a day or so until Mr. Market can clean up the mess Mr. Model is making – unless, of course, you want to take the side of the trade that can benefit from Mr. Model’s stupidity, in which case get in there and join in with others who are ripping his guts out.
Thank guys. Very good points. Oliver, I can confirm your point about path dependence. I did not end up changing my port today because the 70% annualized returns were due in large part to very high profits in 1999 and 2000.
Still, I have not completely given up on the value of Average Return/ Average number of days held. In fact, one could argue that the value of this statistic is that is less path dependent. Also, when your average return goes below zero there is a problem, period. It seems one might be in more danger of going below zero if your targeted average return is 1% per trade than if it is 5% per trade. In the latter situation you could make a 3% error (for whatever reason) and still make a reasonable profit in the former situation: not so much.
So maybe this is more of a problem with robustness than decreased returns.
It could be idiots or hackers. I have fallen victim to hacking where they hacked into my account (along with others) and placed buy orders with all my cash (I was mostly in cash at that time in 2008 due to the market conditions) for a thinly traded stock. The hackers were on the sell side and made a hefty profit while I was left with the loss after the stock came back to its “normal” price once the big trade was over.
What broker were you at? Have you changed anything in how you handle your account security since then?
This raises the issue of P123 security once autotrading starts. In order to sign up for autotrading, P123 will have to have user login and password somewhere, right? So…if P123 gets hacked into…then all of this user data can be taken, right? So…for people to autotrade comfortably…would have to be really comfortable with P123’s security…as this might be a target (with hundred to thousands of user data in one place). Does anyone know much about this?
It was FolioFn. I had posted about this in these forums a few years back in a thread regarding FolioFn. Since that time I have stopped using Windows and only log into my account using a Linux computer. FolioFn has not made any changes as far as I know. I think your concerns about auto-trading are legit.
Have you checked the difference between the Port’s previous close transactions price and the Sim’s next open price? Have you added that difference to the 0.35% slippage that you use in the Sim to the slippage you use in the Port? The 0.35% is in the ballpark to what I found for stocks with a liquidity > $1MM. For the smallest caps in your Port the previous close, next open difference can easily add another 0.5% slippage needed in the Port.
The biggest problem I have found over the last 10 years I have traded stocks based on P123 Ports is the difference between the open price and the ask price when I place the trade. That can sometimes be a few % (buy &/or sell) and wipe out the profits of a Sim that averages 2% or 3% /stock. I now don’t trade a Sim unless it has > 5%/stock, and most of mine have > 10%/stock. That way I can lose an extra couple % and still have a winning Port.
Also to consider, are you trading more than 1% of the AvgDailyTot? You can buy more than that with larger stocks, but will move the price on many of the smaller ones especially at the FolioFn window times. They may trade enough shares during the day, but not have enough shares for sale at the time you place the trade since the highest number of shares are traded near the open and the close.
I also follow the greater than 5 % profit per trade guideline (I believe that I got this idea from Denny) , particularly for micro and small caps where the price variability of the day of the entry or exit could be several percent. If it gaps up by too large a number I can move on to another stock (passing on those that gap up can decrease my average profit per trade relative to p123s sim as these may gap up for a fundamental reason) but often have to accept this difference when I sell if it gaps down. This can also decrease my average profit per trade relative to p123s sim. Increasing your average profit per trade will increase your tolerance for error, which will increase the probability of a profitable system, as Denny alludes to above. Note that I allow a smaller average profit per trade when using less volatile securities, such as ETFs. Good luck!
