What now? 12345

All,

(The “12345” in the subject line was to make the mail handling software happy, as it insisted that my title be at least 15 characters long!)

I will have been with P123 for 1 year on Mar 25. I have been getting discouraged in that I don’t think I have progressed all that much over this past year. I had hoped that by now I would be well on my way to developing my own system with better returns and lower drawdowns than what is available via the core ranking systems. I have not succeeded in that. Back in November I decided to start building a ranking system using the spreadsheet named with “list_of_factors” somewhere in the name. I think it was attached to a message from Mr. @marco posted in June 2006. It has a list of stock factors and their performances.

The performance of the ranking after adding 10, 20, 30, 40, 50 and finally factors, while mildly OK, were definitely not even matching the core ranking systems. I was getting ready to just quit the forum, believing it not being intended for someone like me, who has the financial equivalent of pre-algebra, but is for someone more akin to a financial equivalent of differential equations.

Earlier today I looked over the articles I had saved on my computer shortly after joining P123, articles which I had read and had saved for rereading. A good many of the articles were from the blog associated with P123, written by our resident stock market guru/genius Yuval Taylor. (For what it is worth, I know there are other such gurus on the forum.) After seeing those articles I thought “Dummy, you need to reread those articles before throwing in the towel !”

So I am asking for your thoughts as to what I should do? Should I keep plugging at it, maybe getting things to work in a while, or is it best to cut my losses and go use something else like AAII? All opinions are welcome.

Cary

1 Like

Carymac07,
Don’t give up. That’s my two bits. AAII systems are so riddled with problems it isn’t even funny. Stocks with no liquidity to start. Screens that sound good when averaged over 25 years but with 85% drawdowns, nobody is going to trade them. And their earnings revision model…that has lost all its steam long ago. At least any practical trading system.

My two cents is that there needs to be a balance between the logical and the mechanical.

One good trading system can make all the difference and keep you going while you discover others. Sometimes I find it hard when trying to make a ranking system for the Russell 3000. Too broad. I try to narrow down my universe. Let’s start with microcaps for instance. Define that universe. Now before you go down the ranking hole, try to filter out stocks you don’t want to include. No point ranking stuff that you don’t want to trade.

One rule I almost always use with micro and small is to have positive free cash flow. I don’t want to trade story stocks who spin a yarn and dilute for years without being profitable. Been there done that and embarrassed myself. Got sucked in.

From there I will often run various ranking systems across it in the optimizer. I pay particular attention to the core strategies. I try to get a feel for which broad themes or styles work. Then I drill down. I look at individual factors in that style. You can just make a copy of the ranking system and turn on the individual formulas one at a time to experiment. All the while try to discern why various factors might work better than others. Try to come up with a hypothesis and test it at every turn.

Here is just something you can play with. I took the P123 smallcap focused system and limited the model to microcaps. I made a rule for free cash flow. I made it so it looks at stocks every week instead of every 13 weeks. If FCF goes negative you don’t want to wait around. Short interest less than 5%. No slippage and transaction costs are factored in. I would first get the system performing the best you can and then reduce turnover and finally add back in trading costs. The ranking system gave me anxiety so I tried to organize it into styles, I removed the size factor as I was already in tiny stocks, and then equal weighted all factors within styles and equal-weighted all styles. I want to be clear this is not a ready baked system. Just something I fiddled with for 20 minutes for you so that you can play with it further.

https://www.portfolio123.com/port_summary.jsp?portid=1719358

But I would definitely open up their trading system and figure out which factors or styles are driving this. And then see if you can come up with a few ideas of your own. Or add in other styles (e.g. growth, value, sentiment) from other ranking systems.

The bottom line is that it isn’t about one amazing ranking system. There are many pieces to the puzzle. Try to focus on one at a time. Your universe is a good start. Next play with the 100 biggest stocks in the S&P 500.

These are just a few ideas. I am sure the other guys here have more and better ideas.

1 Like

I hope you stay! My results were not any good after my first year, but I felt I’d learned a lot, and the process itself was really rewarding. So I stayed, which has worked out well for me. Do you like spending time studying and building models at P123? Do you find the process itself interesting?

As for ideas to try: Build much simpler models, stick to max 20 nodes for example. Understanding/optimizing a model with many nodes is really difficult. And make sure you understand what the nodes do. Avoid large caps, life is much simpler with small caps. Perhaps try building models for other markets than the US, I’ve found it easier to beat the markets in Canada og Europe.

And read Yuvals blog, it’s a great resource!

Cary,

Here is a a simple idea. Remove factors like this from the Core Ranking system (or any ranking system you are looking at). Just do it (remove it) is my simple recommendation:

I know some will say that factors like this may belong in there because of “favorable interactions.” But I actually agree that some of the explanations on the P123 forum site are overly complex as well as not always being right. Removing factors like this is easy and best policy, IMHO.

Fitting to past interactions can lead to overfitting if you do want a more complex explanation. Past interactions may or may not persist. Find your own factors to remove but this is SalesGr%PYQ.

Even a little more complex to be accurate, this would clearly fit the definition of a noise factor partially responsible for overfitting. But it is even worse in that we fit to factors that are noise as well as noise interactions that may have helped in the backtest but simply do not work going forward.

There may be some advanced users that can articulate the interaction they were looking for in advance, and perhaps they may want to keep a few factors like this. It is great that they can do that. I am not that advanced myself.

TL;DR: remove the garbage that wasn’t even doing that well in the past with uncertain interactions with other factors in a future market. A future market that we know will be different with certainty.

FWIW.

Jim

Cary,
good post. Thank you.
What you are describing is probably very common and true for most subscribers. Certainly for myself.
I struggled for TWO YEARS with P123 to no avail. I was frustrated, angry and disappointed. The site IS a complex beast or rather Rankings are. By their very nature the complexity of interaction of factors cannot be eliminated.
Small changes can have big effects. Big changes can have very small effects. It’s all possible.

Now, what to do?
As for myself I took a break for a long time and almost never tried anything new, just about forgetting P123.
Then I started again and - as Jrinne pointed out - started to remove some factors to reduce complexity. Suddenly my models improved. It is a long and ardous trial and error game and sweat, tears and rage are almost certainly emotional reactions to occur. Don’t despair. After many years of P123 I am still frustrated with P123 from time to time. Want to give up after drawdowns. Forgetting that drawdowns ARE a (big) part of this game.

Take a break. Start again, simplify.
Read the posts of Jrinne, Georg (Geov), Yuval, Marco Kurtis, WalterW, Andreas and others.

Marco and his team have developed some good models as well. Try them and go from there.

Emotions and “stories” tend to sabotage good models. Reduce exposure until you have more confidence.

Good luck !
Werner

My first reactions is to say “stick it out”.

I’d have to agree with most of the comments here. Bottom line there is a steep learning curve to this stuff, but once you get there, you can apply learnings in different ways and the returns (actual and metaphorically) will come.

When I started, I had some early success, but then nothing seemed to work. Took a 1-2 year break, and came back.

The amount of data we have at P123 is quite impressive; at first I found it overwhelming, but you’ll eventually find ways to use it (i.e. I love aggregate series) and wonder how you did things before without it.

Here are some things I found improved performance:

  • As some others have pointed out, small and microcaps are a good fishing pond for P123, factors seem most responsive to them in my experience.
  • It helps to look at the values the system is screening/ranking. You may be getting some high ranking stocks based on some strange formulas, which may not be interpreting the data the way you want.
  • The “core” systems are helpful, but (I believe) they were intended to be more for demonstration purposes. Some of the factors in them are very well known and alpha may be limited.
  • SSRN is a great resource for new research ideas. Some of the papers can get jargon-y, but if you take the main premise and try and implement in P123 you could be pleasantly surprised.
  • Dealing with NA’s is important - how many NAs are your factors producing? (related to point above about seeing what the system is screening)
  • Look at factors from first principles; i.e. what is the P/E ratio? Price to earnings of course, but play with the numerator and denominator while keeping the main premise intact. Some kind of price or market valuation (price, enterprise value, or other) divided by some kind of earnings or output (sales, income, gross profit, cash flow, etc).
  • robustness checks - try different start/end dates, different universes, different # of stocks
  • sims with very few stocks (5-10) can produce amazing backtests, but are not often very robust. My sweet spot is around 20-25 stocks.
  • see how the sim performance varies over time - was most of the outperformance clustered during a 2-3 year period (2002-2007 was phenomenal for quant systems and standard value ratios for example), or fairly consistent? I prefer consistent for my “evergreen” models, but you can try and get fancy by timing for the current regime (not easy I’ll say)
  • or if too tightly fit, there may be 2 or 3 extreme performers that are driving most of the performance - you can check “transactions” and see if there are any of these and remove them from the universe and re-test.

There’s more, but that’s a first pass.

Hope it helps, I’d be happy to provide any other feedback.

Cheers,
Ryan

Thanks to all of you for your comments and encouragement. You have given me much to consider. I will try it for a few more months and see if I can make more progress.

I will definitely study your trading system, Mr. @hemmerling, and see how it is different from what I have been doing. You wrote "One good trading system can make all the difference and keep you going while you discover others. " I certainly would like to find such a system, and perhaps the one you provided would be a good start.

Mr. @test_user, thanks for your inputs. I initially started out trying to build simple systems. Just prior to joining P123 I read the book Quantitative Momentum by Wes Gray and Jack Vogel, the folks at Alpha Architect. I set out to test their ideas. I started with P123’s Core Momentum strategy and adding a “Frog in the Pan” component, I was able to get a slight increase in performance, but nothing to brag about. I got sidetracked next with reading about how to test systems, a necessary but premature step. Next I read a post by Mr. @Whycliffes who mentioned that he had a system that had 80 nodes in it and looking to add more (“What I have been working on has 80 nodes”) and Mr. Yuval mentioning that “I’ve always said, the more nodes the better, as long as they make good financial sense.” (see Extreme number of nodes (300)– does it damage the ranking system? for both quotes.) So I decided to try to add lots of nodes, Later I read the thread that mentioned the file I referenced in my initial post above (see Ranking your ranking systems - #10 by sraby). So I figured that the best way (for me at least) to develop a trading system that simply added one factor after another. I had sorted the spreadsheet based on the column titled “avg top 5 buckets” (column AE). What made me take so long was that I ran each factor individually over 6 stock universes and 3 different “Rank < X” rules, making 18 combinations. I documented each one individually. It thus took me a LONG time. Finally, after doing 60 factors and not having a better performing system than simply using an ordinary P123 core ranking system, I decided that it was time for a change and come here and post my feelings.

Please note that I am not blaming either Mr. @Whycliffes or Mr. @yuvaltaylor for making me build a system with a lot of nodes in it. Instead, I recognized that I, with a pre-algebra financial education, was seeing two smart users with Mr. @yuvaltaylor definitely having and Mr. @Whycliffes possibly having, the “differential equations” level of financial education. By using that spreadsheet, which I thought was based on individual factors rated well for financial performance, I had hoped to produce a ranking system that better performed. As to whether it has many nodes or few, I don’t care so long as it performs better than the market with (hopefully) lower drawdowns. (I prefer fewer node as I with my pre-algebra financial education will have a better chance of understanding it.)

Mr. @Jrinne, what you have suggested makes a lot of sense, even to me with a pre-algebra financial education. I will try that on different systems and see what happens. FWIW, in response to one of your posts you made a few months back, I posted what was meant to be humorous that I wish Sabine Hossenfelder was here to translate your post for me. I quickly deleted the post as I thought the joke got in the way of the discussion. But I wish she (look her up on YouTube) or someone like her would explain to me many of your posts. And I mean that as an immense compliment.

Mr. @WERNER, thank you for your encouraging words. I am glad to know that I am not the only person who took awhile to learn the system.

Thanks again to all.

Cary

Mr. @rtelford, sir!

I posted my previous posting before I read your post, so now I will respond to yours.

Regarding “small and microcaps are a good fishing pond for P123”, that is what I expected to happen. After all, the big investors will invest in stocks with larger share volume and not fool with the small caps / microcaps. But that was not usually what I found for the individual factors I used from the spreadsheet. For each individual factor I ran 18 different tests, with combinations for 6 stock universes (S&P 500, S&P 1500, Prussell 1000, 2000, 3000, and Easy-to-trade) with 3 different rank sell rules (“Rank < 95”, “Rank < 60”, and “Rank < 40”) for a 50 stock portfolio rebalancing every 4 weeks. The stock universes that usually had the best performance (CAGR) was either Prussell 1000, or S&P 1500, not Easy-to-trade. So what you say is probably true, but my (Naive ?) testing found something else. (I have attached a XLS file for reading enjoyment.)

I will try to read more papers at SSRN.

Regarding N?As, I wish there were a way (and perhaps there already is) to set a somewhat global switch to either skip stocks that have N/As, or would fall back to the previous value for the item, or something else. I currently have N/As as negative.

Regarding first principles, here is where my pre-algebra financial education gets in the way. What measure of earnings should I use? There are lots to choose from, most of which I don’t understand.

Regarding robustness, I am using almost all of the data available to me, namely 2002-Jan-01 through 2022-Jul-02. The reason I stop on that date is that was when I started doing massive studies and saving the results and I have kept those dates to be able to compare the different studies I have done. I initially started out using 5 stocks in the portfolio, but was shocked to see how much variation in performance when I used Mod(StockID, 5) = 0, 1, 2, 3, 4, When I posted about the large variations, fortunately you folks helpful showed that a 5 stock portfolio was highly suspect to luck. I then changed to using a 50 stock portfolio for my testing which I have done ever since.

As for checking the performance over time, my plan is to see one that has performed well over the 20 year time frame, then narrow it down to smaller time frames, such as 3 years.

But thank you, Mr. @rtelford, for your help, and thanks again to all. I will reread everyone’s posts and try to go from here.

Cary

sell_rules_versus_universes_for_single_factors.xls (146.5 KB)

I joined p123 11 years ago.

At the beginning it was a pure disaster. I used to create a system with a lot of garbage, 100 factors, without liquidity constraints, without any logic.

It took me 7-8 years to learn and create my system that has beaten a benchmark 3 years in a row by a substantial margin. So 1 year is very short time :slight_smile:

The guys here at forum helped me a lot. People here are nice and helpful. Just learn from them … and I’m pretty sure you will find your system one day!

I don’t want to hijack the current thread, but I use the screener to check for NAs.

Set your rule as factor=NA, pick your universe, set max stocks to zero, and run a backtest.

The number of positions plot will give you a feel for how often and when NAs occur.

Try to stick with it if it’s what you enjoy. It took me several years to gain a sustainable edge. And I’ve only had luck with a longer holding period, ie 1 year, which might put me in the minority here. Anyway you might test different investment durations to see if that helps.

I second what everyone else as said. I’ve been doing this for 4+ years and it’s taken me at least 2 years to actually start executing strategies with an edge. Very frustrating the first couple years, but unfortunately everything that is worth figuring out is difficult and a steep learning curve. My wife gave me a lot of crap about it too, but I kept it up.

One thing that Marc Gerstein said that really stuck with me is there needs to be a logical reason why your strategy should work. It doesn’t mean that all things that should logically work will logically work, but it’s important to avoid data mining and over optimizing strategies. That philosophy has really allowed me to hone in one ideas not waste my time aimlessly trying to match various factors that don’t make sense. The core rankings are an excellent place to start but you can modify to strengthen the focus of your systems. Also, I’ve found defining narrower universes and applying rankings to those universes to be an effective means of creating a system as well.

And also which has also been stated is that you need to make sure your system is actually tradeable. Aim to reduce the turnover of your system as much as possible, pick stocks that are reasonably liquid (if you can), while still achieving outperformance. A system that simulates 60% returns a year but has 800% turnover in highly illiquid stocks (large bid ask price spread) is not going to get you a 60% return and maybe not even 20%. Trading is a cost, not to mention tax implications especially wash sales. Needless to say I now trade much less than I did in the beginning and have much better performance.

Keep at it. Keep reading papers and buy some books if you haven’t. Try new ideas as well and think about what makes sense. You’ll get there. And remember this is a hobby too, but one that can payoff if you are passionate and persistent about it.

Jeff

Some tips after reading over the recommendations above.

  • Reduce turnover
  • Use volume rules so that the stock you are running the simulation can be traded and apply “MedianDailyTot” to volume: MedianDailyTot(120) >( 70* 1000) and StaleStmt = 0
  • Run all tests according to the same principles as DM= variable slippage + Average of Next High, Low, and 2X Close, and avoid too low volume. This will affect returns extremely. Here is the return in “inhibitor” simulation with variable slippage + Average of Next High and volume (MedianDailyTot(120) >( 70* 1000) and StaleStmt = 0)

Especially volume rules and variable slippage + Average of Next High, Low is important because it forces the simulation to be more real.

Otherwise, I agree. This is extremely challenging, and sometimes very frustrating.

Easy to Trade US universe has already a volume filter.
MedianDailyTot(126) > 50000
Increasing it to 70000 does not make much of a difference.

Sorry everyone. I didn’t really mean this to be a ready-baked model and there is some concern over liquidity and costs. So I made a couple more changes to the port.

  1. Modified the universe so that the easy to trade US has a minimum of $2 per share instead of $3.
  2. I added a trading rule so that $100,000 per day of turnover to buy is required
  3. I increased the cap minimum and the maximum. $50mm to $500mm. So no nano and some small caps
  4. A trading rule I like to cut losers and take profits. If the stock return is more than 30% of the bench, run the stock back through the buy rules (trigger a possible sell but it can be held if it still is a highly ranked stock and meets all the buy criteria). Same if stock is 20% below bench from time you bought.
  5. Execution is average of high, low, close etc. etc.
  6. Variable slippage

To be honest, this likely isn’t the ranking system I would use. Some of the formulas seem overly specific ranking in the industry only or will just result in lots of NAs. This was just meant to show how you can take an idea and build on it.

Thanks again to all of your thoughtful comments. I have more questions that I hope you folks can help me answer, but I will post them in separate threads.

Cary

You also might want to start with this: https://www.portfolio123.com/app/ranking-system/403577 and this: https://www.portfolio123.com/port_summary.jsp?portid=1701030. It’s a lot like what I use myself.

1 Like

I had one other thought about modelling. Don’t go too broad but don’t go too specific when selecting your universe of stocks.

Too broad is the problem of the academic world. They want one simple theory to explain asset pricing and returns from OTC to the S&P 500 and all markets around the globe. Beta, 3-factor model and so forth. But the universe of stocks is so broad. Different market participants will react and value stocks differently.

My dad only invests in junior and exploration gold stocks. He looks at inferred oz and so forth. This has almost no relation to the price of Coca Cola. Or a utility stock. I would suggest picking a universe that is somewhat more narrow than just all publicly listed stocks. That’s why microcaps and smallcaps are good because you are tapping into a sub-set of investing mindsets. Not the same as the big institutional investors. Keep looking to categorize universes and build a model for it.

Just be careful that you don’t go too narrow and lose perspective and curve-fit. If you are designing a system for a small industry group of 10 stocks, chances are that won’t be great either. Too much randomness and idiosyncratic risk which cannot be defined by a factor rank.

Need to target sub-groups of stocks but not too specific.