I'm new to Portfolio123 and have been developing my first ranking system for the past few months. The results look good but being new to P123 and to rank-based investment strategies I'm hesitant to take the plunge and invest real money in the strategy. I've been doing sensitivity analysis within my formulas, weightings, universe, time periods, etc. - using the Rank->Performance function, Backtesting, and Rolling Backtesting. I've been unable to push Alpha below zero in Backtesting, and in Performance the correlation ratios look good (Spearman > 96; Pearson > 92). Rolling backtesting indicates issues if I my holding period exceeds
8 weeks. Note that my universe is sector specific and I've pulled in external data to help analyze specific metrics within the sector.
Can anyone suggest further testing/analysis I should perform before investing real money in the strategy. I've read many posts about running statistics processes outside of P123. Are they necessary, helpful; what benefits do they provide that P123 doesn't, etc.?
I guess my main question is: how do you know when a strategy is ready for real-world implementation?
Hi RJSchierm, I'm by no means an expert on this, but would suggest if you're trading illiquid stuff to actually start trading it with a small amount of money to get a feel for what it's going to take to get fills and how to trade it. (Apologize if you're not trading illiquid, I just saw the min liquidity of 3000 and thought maybe it's an illiquid strategy. If it's easier to trade this is less concern).
fwiw, I'd also expect you may keep iterating on models for years and years rather than get one locked down. Again, I'm not expert, but feel there's benefit to really trading it with live money to see how it feels to you, and probably expect to keep tweaking. Again, I don't know that'll be the case, but I'm miles away from models that I started with. all fwiw, best wishes.
I agree with Shane: you learn so much in the process of actually trading a strategy that a strategy developed without any actual trading can never be fully "ready for real-world implementation," as paradoxical as that seems. Practically every week I notice something about one of my holdings or the process of buying/selling it that makes me tweak something in my portfolio management system or my backtesting procedure or some factor that I'm using. Putting real money into a strategy is more of a "test" than anything else you can do outside of P123.
I would try the rolling backtest on a subset of the universe; Universe rule Mod(StockID, 3) = 0 (or 1 or 2) splits it in 3 "sub universes".
Depending on your intended portfolio size I would increase the liquidity threshold to at least average $50k+ and a higher value for median.
But I agree with the previous replied, even just a tiny bit of skin in the game changes the dynamic, and your focus, substantially
How do you make that decision to tweak, is it an informal process where you notice some things here and there every week or do you look for specific characteristics of your holdings?
In my case, what I try to do is to look at the top holdings and go through my factors with a screen to see if these top holdings indeed 'rightfully' score high on the factors I use. In that way, I spot inconsistencies in my factors in terms of factor construction. This doesn't happen to me every week though, hence my question.
It's a very informal process. For example, yesterday I was talking to a colleague and we got on the subject of arbitrage (in the sense of investors arbitraging away known factors). Because a lot of P123 users use similar factors and choose the same stocks, it occurred to me to make some adjustments in my buy/sell/hold spreadsheet for that fact. Last week I changed a few of my factors to use IsNA(xxx,0). I also recently made a relatively large change to a delay-cost factor I use to size my positions because I realized my calculation of it was flawed. And so on. You can get a good sense of my tweaking process by reading What I Learned About Investing in 2022 - Portfolio123 Blog
Welcome to the community. To add to what has been already written it looks like the heavily cap-weighted S&P 500 is not an appropriate benchmark. With metrics like a too high a drawdown versus the benchmark and a standard deviation that is way high versus the alpha it just looks off. Take a close look at what is being selected by the system and look at the market caps to get a better read on a potential benchmark. The equal-weight S&P 500 may be something to consider.
Also, the universe specified seems to be about 200 stocks (10 Deciles withe 21 stocks per decile). Why such a small universe for an initial Ranking System?
Thanks everyone for the responses. I've read a fair number of posts, papers (Yuval's), documentation, listened to webinars, and had a brief session with Riccardo about sensitity analysis. I just never got a sense for when is the right time to put money down. I tend to overanalyze and it felt like I was going down that path....hence my question. I'll move forward with a small investment in the strategy
to see what happens.
A follow up question: what kind of data do you gather/analyze when first implementing a new strategy. Price, slippage, rank at time of purchase, etc. come to mind. Are there any statistical analyses that would help me determine the accuracy and robustness of my strategy. Also, if I wind up rebalancing every 4 weeks, it would take a long time to build up enough trade/performance data to reach statistical significance.....is it preferable to begin with weekly rebalancing just to amass more data?
I'll toy around with the benchmarks...thanks calling that out, rwbattyaz.
To the question of the small universe and low volume, my target sector is US banks - less than 600 are public and of those 250 are OTC, which I'm avoiding at the moment (but they provide a nice OOS set for validation testing). This is a space I know fairly well and I've industry-specific data that I'm able to import. I wanted to learn the P123 system so creating a ranking system for this sector seemed the
best approach for me. I intend to one day create a more generic ranking system that can be applied to a much broader part of the market.
You certainly have picked a rough road to walk. I just spent the afternoon working with the subsector BANKS and sector FINANCIALs. The subsector work was bouncing around benchmark with the sector work somewhat better.
It does look that your industry knowledge is adding significant value in this domain, especially with 20 stocks as the target portfolio.
FWIW it looks like the S&P 1500 Financials is the least worse of the available benchmarks and 4 weeks a reasonable holding period.
The afternoon did yield a niche, 2 stock, screen against the sector which will be funded during my rebalance this weekend. Thank you for prompting me to investigate this.
The thing I struggled the most with, and still do, is when my modell picks stocks that make me wonder why it's picking this piece of junk company. I still get those, and putting real money on it focuses me. Sometimes at the extremes of a ranking system some strange stuff shows up. I'm guessing with your expertise in banks you'll look at some of the companies it picks and wonder "why is my system picking this? I don't like this at all."
There's some high ranking companies right now in my ranking systems/screens that when I research them all I find are terrible reviews from customers, other business owners advising people not to use them, suggesting it's racket,etc. But they do generate cash. Another company has gotten into putting bitcoin on their balance sheet and has issues with whether medicare will continue to fund their test procedure and they've laid off a lot of people. Some companies I have a history with and they always fake me out and end up badly - like maybe something cyclical about them makes them hit the ranking system at the worst time, but I'll always lose money with them. I don't know, just lots of weird things that when I see the selected companies makes me wonder if that's what I intended w/ the ranking system. Sometimes stuff that doesn't fit the expected profile makes it through. It helped me a lot to try to reduce the amount of those type situation in the ranking system/screens, because I hated losing money on companies I had to hold my nose to buy.
Rich & Spaceman, I appreciate your thoughts and insights into this topic. I suppose it's time to at least do some simulated trading with my strategy and see what happens. I'm hoping to put some $$ into it later this month, which - as just about everyone on this thread says - is the real test.
Cheers
