Two Pre-Registered Designer Models on OSF

Research
1

Quantitative model marketplaces often suffer from publication bias, survivorship bias, and p-hacking—assuming any statistics with p-values are presented in the first place. Much of this can be mitigated by pre-registering studies.

I recently launched two Designer Models and have never opened any others. They should become publicly visible in about 3 months, ensuring no survivorship or publication bias.

My hope is that these models will perform well and that the pre-registration will validate the statistical results. But this should add to the discussion of backtesting decay and the effects of survivorship bias in any case.

Both models are pre-registered here: OSF

Edit: I was thinking about why pre-registration helps. The literature on this is pretty dry, but really, it is just like calling your shot in billiards.

I can be aiming at the 7 ball trying to get it in the side pocket, miss entirely, bank the cue ball off the cushion, hit the 9 ball, and send it into the corner pocket. Then I post that shot online claiming I'm great at billiards. I post if did not call my shot ahead of time, I should say.

The analogy is looking at 156 models, finding the single best performer, and claiming it was pure skill. If you had called that specific model ahead of time, it actually would be impressive—or at least the odds of true skill shoot up dramatically.

Looking at Designer Models should be exploratory at first. You may like the designer, the backtest, or the out-of-sample results—all legitimate. But then you need to call your shot going forward. Write it down, specify some statistics, or set a clean benchmark—something like "I think this will beat 95% of Designer Models 3 years from now."

That is dramatically different than picking whichever ball drops after a break and calling it skill. For most players, it wasn't.

If you want the dry statistical version: unless you set a strict end date for the study in advance, over half of all random studies will show statistical significance at some point. And actually, given enough time, the probability approaches 100%

2

Looking forward to seeing the results, always been interested in seeing a DM from you. And advancing the discussion on DM backtests vs. reality is also a noble goal.

3

I'm always grateful to see new DM additions — thank you, Jrinne. There are so many of you I would love to see post DMs.

The OSF registration is commendable — more on that below.

Having navigated the DM market for a while, I've found that getting to know the designer and understanding his logic are essential. Most designers rely on the backtest to sell the model, when the things that actually matter — the designer's cred, the model description, basic attributes — are typically all but absent.

For instance: does a model have buy and sell rules beyond rank? What, roughly, does the ranking emphasize? I know designers want to keep their models and ranking systems secret, but even a basic sketch of the approach would help.

As discussed before, the 3-month rule should go away — in favor of full transparency and a launch log. Imagine someone challenged you to show subscribers the best-looking launch possible. What truer gift could you get than 3 months to spawn as many candidates as you like, unbeknownst to potential subscribers, and reveal only the ones that worked? And the window buys nothing in return — 3 months of live returns can't statistically separate skill from luck anyway.

I appreciate Jrinne's approach — pre-registration is the antidote to exactly that scenario, adopted voluntarily. The simplest version, though, wouldn't need OSF at all: remove the 3-month veil and have P123 log every model created, including dead ones–same antidote, built into the platform by default.

The one rationale mentioned for the 3-month rule is reducing "clutter," but clutter is a display problem, with display solutions — filters or default views. The veil solves it by destroying information instead: the launch record that tells you what a surviving model is actually worth. We are removing the denominator–it’s like knowing how many hits, but not how many at-bats. I hope this gets consideration — or that someone can explain what I'm missing about why the current setup serves subscribers better.

Thank you again for the models and for raising the bar, Jrinne.

4

This is not new. It is a common practice for mutual funds. It is called the incubation bias. Here is the abstract of an article about this: Mutual Fund Incubation

"Incubation is a strategy for initiating new funds, where multiple funds are started privately, and, at the end of an evaluation period, some are opened to the public. Consistent with incubation being used by fund families to increase performance and attract flows, funds in incubation outperform nonincubated funds by 3.5% risk-adjusted, and when they are opened to the public they attract higher flows. Postincubation, however, this outperformance disappears. This performance reversal imparts an upward bias to returns that is not removed by a fund size filter. Fund age and ticker creation date filters, however, eliminate the bias."

5

Jim,

It is almost 3 months.

When will we see the 2 new DMs?

Regards

James

6

This week. Wednesday for the 100 Easy to Trade Stocks model and Friday for the Canonical Solutions: 15-stocks model IF I decide to make them public then (which I will do).

But new to me is that I think P123 gives me the option of incubating them longer than 3 months. So Designers Models have a survivorship bias and an incubation bias. Possibly with flexible incubation periods.

7

Jim,

Just curious, the 100 Easy to Trade Stocks model has 100 holdings?

Regarding surviviorship bias, I see that Walter has not deleted 2 of his models and just call them deprecated. (This solution doesn't work if some designers want to launch a lot of models and keep them incubating for a long time).

Regards

James

8

It is 100 stocks from the Easy to Trade US universe.

9

Right. I get why mutual funds are incentivized to trick people, but I don't get why we would be. And I guess that is my biggest complaint about this: it's being presented as protecting the potential subscriber, but in reality it's a 180-degree pivot.

10

Right. P123 could still keep it closed. Just closed and visible.