I'm with you on all of this.
I think it's mostly about building trust with a designer, and maybe that's the truest opportunity for P123 to facilitate. Of course, even just spending time in the forums can help on that front.
I also didn't mean to imply that designers are nefariously trying to deceive subscribers; I just think it can be easy to deceive even yourself at times.
I think this actually works against P123. The statistic we know for sure now is that the average designer model underperforms its benchmark. This is true even with the incubation bias and the survivorship bias.
I think it is probably true that some designers have enough skill to overcome the problem of transaction-cost drag. P123 COULD show that it is possible to reliably outperform a benchmark if we had complete track records of those capable designers.
I can't answer these questions, and I'm not defending Portfolio123's designer models, which have all sorts of problems. Perhaps the world would be better off without them. The only point I was trying to make was regarding the term "survivorship bias."
I'm always a little worried when I suggest improvements or ask for things like API access, because I can see this outcome. Which would be super unfortunate.
Deleted by the author.
I am completely without influence in this matter, so please don't be alarmed.
Right. But the thought had crossed my mind before today as well. If suggestions for improvement seem too complicated or like complaining…
Anyway, I hope it doesn’t come to that!
Here is an interactive Claude artifact that begins to give us reliable statistics on P123 Designer Models using the Benjamini–Hochberg (BH) procedure. It is not perfect, and the truth is it will never be perfect without better data. Anyone who has a specific concern can modify it with Claude’s help.
The BH equations are straightforward and can easily be run in a spreadsheet without any programming. Anyone who has read past the abstract of the academic papers regularly cited on P123 has seen this test before. If we cite those papers in the forum and suggest they are a good source of information, we should actually use their statistical tools, though not everyone here agrees, for reasons I still don’t quite understand.
The BH procedure identifies the subset of models that pass a pre-selected False Discovery Rate (FDR). The test here uses each model’s 2-year excess return over the designer’s own benchmark. Setting the FDR at 0.20 (meaning that within the group of passing models, the expected proportion of false positives, the “just lucky” models, is capped at 20%), 3 models pass:
The winners are Marchus Wagenberg, Kumar Kumar, and Andreas Himmelreich. At the stricter FDR of 0.10, only Marchus’s model passes.
However, that baseline assumes zero survivorship bias, since it only counts currently live models. When we account for retired models, the multiple-testing hurdle gets steeper:
Note: This survivorship adjustment is an estimate, as we cannot know the exact number of retired models without seeing the graveyard data or their p-values.
The takeaway is encouraging: P123 is close to having real statistical evidence of the out-of-sample value of its top models. The main obstacle is the survivorship blind spot in the data. The retired models and each model’s weekly returns would settle much of this.
Take a look at the calculator here: Claude Artifact
Note: It would help a lot if we had each model’s excess return relative to the designer’s chosen benchmark since launch. With that data, and with the retired models included, I suspect P123 could show the usefulness of its platform statistically, free of both incubation bias and survivorship bias.
While some might disagree, I think that kind of evidence would not be an entirely bad thing for a platform that offers machine learning as an option.
Good start!
Currently your portfolio is very concentrated in Energy.
Why your benchmark is Russell 2000 ? You have currently +70% exposure to medium and large stocks.
https://www.portfolio123.com/app/r2g/summary?id=1959080
Thank you Piotr.
Right. I had not noticed that until you mentioned it:
That is an anomaly I believe. I do not use any features that select for market-cap so the model will select mid-caps and large-caps. But generally the holdings look more like the Russell 2000, I believe.
I ran the sim ending exactly a year ago and this is what it looked like then:
Arguably the Russell 2000 is closer to fitting the holdings in the second screen-shot than the cap-weighted Russell 3000. I believe this is generally the case. Also, the correlation of my strategy to the Russell 2000 is 0.80 over the entire period of the sim and this is the highest correlation I could find for a benchmark.
If I had to guess, I would say the anomaly in both the heavy energy weighting and the larger than usual holdings of large-caps is due to the unusual situation in energy now.
Here are the sector holdings from a year ago. It was concentrated in healthcare then (no energy stocks):
