I’m not sure I quite understand where you’re coming from, but some observations/comments based on general theme of questions and where I think you’re coming from:
- I test on biggest universe that meets my required liquidity, not a subset of stocks. (I exclude financials and real estate). I’ve tried subsets with certain characteristics as test universes, but I’ve found it’s usually better to just test on larger universes and let the ranking system sort things out.
- Probably test variations beyond 20 top holdings. I used to have a detailed and specific testing process, but now normally simplify and equal weight the top 5% of a ranking system. When I refine things to fewer holdings performance gets better, but I feel the top 5% gives me decent signal on quality of ranking system without worrying too much about idiosyncratic stuff going on. Maybe I used to think I could find the “perfect” way to express a factor and combine factors and weights, but over time as I’ve seen the market change and factors wax and wane I’m thinking I was trying to apply too much precision to something that’s more messy.
- I like to test from 2007 forward to capture the financial crisis - but that is getting to be a little bit long in the tooth. The problem is since 2009 the US market has been in an almost persistent secular bull market with lower/lowering rates and only very brief periods of crash/rapid recovery. I worry about that, but not sure how to handle other than worry about it. I feel like going back to 2000-2003 bear mkt is too far back and factors worked very differently back then (many were much more effective then than they are now).
- Ranking systems are the core of my stock selection process, but I do have a few screen rules to set minimum requirements. Ideally I would incorporate all the screen rules into my ranking system, however, I’ve had a few cases where I’ve been unable to make that work and use screen rules rather than contort the ranking system. In general, for the best backtests I think it’s usually best to put as much as I can into the ranking system. (As to specific screen rules variances - if I want stocks to have positive revenue growth or some maximum amount of debt load or positive earnings, I may have to hard code that into the screen rule because the ranking system will let some through no matter what I do. In my experience doing this almost never improves system performance - but since I find it hard in “real life” to buy certain types of companies I’ve found this approach helps me better stay in the game. But to be clear - the tradeoff is that the backtest almost always decreases when I do this. Also be careful of curve fit at this stage. I do this because I have Buffett’s voice in my head reminding me that the market doesn’t have to open tomorrow, and therefore may have a tail risk scenario where I have to hold things for a long time.
- My ranking systems are kitchen sink approaches. I’m pretty sure I use something identical to P123 Sentiment as one component of some of my ranking systems, but it’s just one factor of many.
- Beating the market is hard - and John Bogle points out as a truism - after backing out fees, the average investor return must be below average return for the market. But 25% annualized returns are extraordinary and will make a legendary career if sustained over time. Something about that universe seems suspect imho.
- My ranking systems changed a lot my first 2-3 yrs here as I tried all sorts of stuff, all sorts of factors, and learned all sorts of lessons, but have stabilized over the past year or so. I still try out new ideas, but my key learning is for behavioral reasons I needed to use a system that gelled with my personality and concerns even if it didn’t have the best backtest. Obviously still learning and taking things in, with open mind toward what I see - but strong respect for observation that quant approaches can lead me to companies that don’t (currently) make sense to me at both an intellectual and visceral level, even if the backtest is great.
Hope some of this gets at your question. I’m no pro - and make plenty of mistakes - but wanted to share this in case it’s helpful.
