Why are there so few subscribers to designer models is the question?

General
21

Jim, I really like what you’re doing with bootstrapping. After I finish my SQL class, I’m going to look for something in statistical analysis.

Years ago someone from a large investment bank contacted me about my dividend income model. He wanted the transaction log and I suspect he ran an analysis like you proposed. He never followed-up. :frowning:

I don’t think your proposal conflicts with my desire to see return distribution. I’m just trying to get a (qualitative) feel for how difficult trading the model would be and also how risky. I would consider a model with a small average return but bimodal distribution as risky. On the other hand, if it had a well defined mode, I would consider it difficult to trade.

If I had to prioritize features, I would pick yours first. Quantitative answers are better.

22

Walter,

Thank you. For the designer models using empirically Bayes is not actually that hard.

P123’s AI expert could have it done by noon (I hope but I do not know her).

P123 could market those results in good faith I believe. To be accurate they would have to include discontinued models so they may not be able to go back in time.

And I am actually talking about the out-of-sample port results including the discontinued models,

Good models including (but not limited to) yours and Yuval’s will rise to the top with time. Actually, to be accurate they will probably start at the top and prove their statistical significance with time. So not really any downside.

Bad news: it will take a little time to prove statistical significance. Good news: clear statistical proof that your models are superior (assuming that they are). VERY CLEAR IF TESTED THAT WAY!!!

And not just clear statistical proof but a realistic expectation regarding the predicted returns (the returns will be shrunk in other words).

One word describing your situation if you ever had a model like that with someone who tested it that way: Wow!!! And of course, more than one word: “I think I will raise the price.”

Jim

23

All,

Personally, I would like to see the backtest results. I am not for hiding information from people or censoring information.

I would like a lot of information about the backtest to be downloadable, in fact!!! I would like to bootstrap the results myself—especially if P123 does not end up doing that. I would run a Bayesian analysis knowing the result is not accurate without the discontinued models.

But people who think newbies might not understand and think that anything will perform as well out-of-sample could be misled are right about that I think.

I mention Yuval’s designer models because they are an example of good models. He has commented on the regression-toward-the-mean for his models. I will let Yuval quantify that or correct me if he does not think there is regression-toward-the-mean for his models.

But regression-toward-the-mean is a force of nature that we can do nothing about as humans (only slight hyperbole here). Actually, if you look at information theory (and entropy) it is not hyperbole at all. There are equations for this.

I do not even have to appeal to overfitting here.

Any professional financial advisor will show you a Monte Carlo simulation at least once (maybe so she does not get sued for false promises but I am not sure).

As a somewhat southern hick I want to say: “I do not have a dog in this fight.” I will not debate this much. But I do think bootstrapping is a great tool and is as good or better than Monte Carlo simulations. Better than Monte Carlo because of the assumption of normality for Monte Carlo simulations if nothing else.

I would consider adding a bootstrapped confidence interval if you continue to show backtests. Again, this (or Monte Carlo simulations) are established professional tools.

And a disclaimer along the lines: “Due to regression-toward-the-mean the designer model is expected to perform below historical returns and toward the lower confidence interval.”

This is, actually, a provable mathematical fact. Full stop. And there is no need to imply that the designer is overfitting in any way.

Using my sim/port as an example, I have redone the above a little using log returns (and is a little better but not perfect yet). If it were a designer model the backtest would show about 42% annualized return.

The lower bound of the 99% confidence interval is 22% annualized return.

Put simply, I think it would be wrong to suggest to a buyer that it would perform better than 22% annualized return. If it does do better then sue me.

But any implication that it might perform as well as the backtest would be just short of a lie most of the time. I guess anything is possible but that is not likely.

BTW, as per Walter’s request for a distribution. Fat tails, I think (something favoring bootstrapping over Monte Carlo):

Distribution Plot
Distribution Plot1212×770 17 KB

Jim

24

One alternative would be to go live with the best ranking system performance for a specific universe over a recent time period determined by the developer. Then update on a regular basis. The “best ranking system” could be one of the canned ranking systems offered by P123, or of the developer’s own creation.

This is not optimizing on noise. And (in my opinion) if you are not over-optimizing then you are under-optimizing and leaving money on the table.

My two cents.

25

I might be wrong and would happily be corrected but I do not think that Yuval simply went live with models that were most optimized over the entire universe and full time-period.

If I am correct, that may be one reason for why he has done well out-of-sample.

Generally, the inspector makes a good point that there are other ways and I think there are more than 2 ways to reduce overfitting (more than Yuval’s and the inspector’s).

When I started I just used simple optimization over the entire universe. It is attractive and it is hard to know what else to do. I was extremely over-optimized, I believe.

Over-optimization can be thought of fitting to the noise but can also be looked at as not correcting for a huge number of optimization trials. I.e., if random() had a seed you could have a great sim in-sample by just running random() as a ranking system: try it 100 times and see how well the best one does. The seed would allow you to save the best one and have it as a designer model.

Using random() might not do well out-of-sample, however. And it is accurate to say it will regress-toward-the-mean out-of-sample. Using random() in this makes the outperformance all luck. But there is some luck involved with anything in life. There will always be some regression-toward-the-mean. When you use a large number of optimizations you will get lucky with some of stocks picked or with market conditions and interactions that will not persist EVEN IF YOU HAVE ONLY GOOD FACTORS THAT ARE NOT NOISE. They will be weighted in a way the involves some luck with optimization. Maybe one of the buy or sell rules gives you a lucky pick on some of the stocks.

There will be regression-toward-the-mean when you optimize and pick the best optimization. No way around it. It is literally a law of nature.

You should expect it and perhaps have ways to develop realistic expectations for out-of-sample results. Maybe be able to predict that a model might regress too far to be of value to you before you fund it. Yuval and others have ways to minimize the amount of regression-toward-the-mean in the first place.

26

Just wanted to add my two cents to what has been said here, most of which I agree with.

In my experience, whether or not you optimize a system, and how much you optimize it, is of lesser importance than the quality and thoughtfulness of the factors in the system and how well-balanced the system is. The goal of optimization should be to see how various factors combine well, how to manage your portfolio successfully, how slippage will affect your strategy, and so on. Robustness is extremely important and should not present an insurmountable hurdle. I’ve found that the best ways to make your backtests more robust are to test on multiple mutually exclusive universes and to winsorize or trim your returns to exclude extreme short-term results. (For example, I measure alpha by comparing all my two-week returns with those of my benchmark but first I use elliptical trimming to eliminate outliers. That may be a bit extreme; simply trimming 2.5% to 5% of your most extreme data points should suffice.) Out-of-sample performance may be terrible for a year or two (2019 is a case in point for my strategies), and that should not be cause for despair.

27

All,

I will write as a retail investor (i.e., someone who doesn’t know what he is doing). I think the reason there are so few subscribers to the designer models is that this site’s primary purpose is to help people who do know what they are doing to do it better. (For a retail investor like me, this site is intimidating, and for several months I didn’t even know that P123 had designer models to follow.) In my opinion, P123 and its designer models would succeed better with a second dedicated site that focused on the retail investor. Of course, advertise on this platform and on other platforms, such as Seeking Alpha and AAII, to your new web site, On this new web site, you could mention the P123 web site as well, with a lead like “Want to build your own trading system? We’ll show you how: https://www.portfolio123.com/”.

So I think a second, entirely new web site is the better approach. Of course, just my opinion.

Cary

1 Like
28

Yuval - Is it safe to assume that you are performing the “winsorize or trim” off-platform? If so, any plans for P123 to include this functionality in the future?

Also, are you trimming individual trades, daily/weekly/monthly returns, or via an alternative method?

29

Yes, I do trimming (of 2-week returns) off-platform. There hasn’t been any demand for such functionality, so it’s not in our plans.

30

We’ll promote them on the Dashboard soon. We need to see more subscribers to justify more investment. We’re also introducing discretionary rebalanced Designer Models DMs to attract models from other sources .

31

I have several DMs but I don’t open them for subscription until they are at least one year out of sample. If they can’t perform substantially better than their benchmark over one year then they are no good.

32

Thanks @mv388158 . BTW… Our marketing person left last year and we are ready to restart with someone new. He was very good, a lot of his campaigns are still running, but he was more tuned to market traditional consumer products (he’s with Shopify now). That is products that are easier to use and with wider audience (we’re getting there lots to do still).

Marketing financial services “that can make you loads of money” is very tricky. With the wrong message you are written off as snake oil, and competition is everywhere. So we’re looking for the right balance: a person that can use p123 and/or invests in stocks, and has good marketing ideas that don’t require huge development efforts. A big plus if experienced with B2B. Send us a msg if interested.

33

@Jrinne and others. Lots of interesting discussions about overfitting, robustness, etc. Bootstrapping is interesting but it’s main use-case is when additional sampling is impractical or expensive, no?. It’s not in our case. Sims are cheap and fast. Bootstrapping feels like another data torturing method.

Hard to keep track of all that is being discussed and suggested. But in the end the past is the past. There’s simply no way to know if a model being proposed is just the best one out 1000 sims, or if it’s the product of skill, restraint, and common sense.

So the safest for us, and model subscribers, is just to highlight Designer Models (DM) with longer out of sample data. This also has a nice side effect of creating stickiness for our research subscription service (you have to stick around to be have out of long out of sample data).

Here are more ideas we’re toying around:

  • Highlight top DMs in the dashboard that have at least 5Y out of sample data, and come from “active” or “engaged” designer (ex: those that login frequently)
  • In the DM list by default only show models with 5Y+ of out of sample from active designers . Around 100 models fit that criteria; more than enough. And users can easily see all models by turning off the filters

With the above changes I don’t think we need to make any modifications to the presentation of the models.

34

Marco,

Just quick to say I agree bootstrapping may not be for P123. In part because anyone—including me—can easily implement it with downloads if they find it useful.

That having been said it does a lot of different things.

For example, it is an integral part of the random forest AI P123 will be implementing. It would not be a “RANDOM” Forest without it.

TL;DR: Agreed

Jim

35

I would still like to have a direct link to just my DMs. Something like www.portfolio123.com/walterw. With that, friends and family can have quick access to my work and not have to hunt through the current site. The landing pages could also have links to the other DM offerings and/or to the main p123 site. Consider them feeders into p123. Add referral credits for signups through those pages and now you have a motivated designer.

Currently, in total, DMs earn about 3K/month with around 30% going to p123 clients. It really seems like a forgotten part of p123.

36

@WalterW

You do not need to be logged in to view DM models . Try logging out and go to Products→Models

You do have a profile page which you control Profile - WalterW - Portfolio123 Community

What we need to do is make parts of the model presentation page embeddable (similar to how you can embed tweets). For example give you ways to embed a thumbnail (summary+ chart + stats), or individual parts so you can design your own landing page.

The customers have, not us. We’re still reeling from the massive failure of version 1 of DM (or R2Gs). If there’s any positives from v1 is that there was demand for models, and people came regardless of the confusing presentation (it was generating $50K/mo at some point). A separate, simpler website is not needed to prove the biz model. There will be one at some point, but not required now. So no worries, it’s not forgotten. We’re still developing it and have several ideas.

37

True. But that’s not a direct link. One still needs to click through the DM section and then search for a particular designer. On the web, even small hurtles will stop people in their tracks. And how do I put that on a business/contact card?

Happy to hear p123 wants to build up the DMs.

38

Yes it would be https://community.portfolio123.com/u/WalterW

This is your community profile. Right now it has a two sections: Summary & Activity. You can add links to your models in the Summary, but it’s hard to do and only supports static images and text. The Summary layout is not ideal either. We can add a third section for Models like in the image below that would be purpose built for models that is dynamic, with a good layout, and always have updated stats and charts.

The nice thing about using our existing forum software is that it comes with many things that a client would want to see: a) is the designer engaged? b) what are they writing? c) what’s their reputation? (like Reactions, Solved, Votes in the image). The forum software also has many add-ons for ‘reputation’ like :+1: or whatever from other users. And a lot more. It’s definitely the way to go for a “Designer Landing Page”.

Makes sense?

image
image1800×931 117 KB

39

BTW,

Your profile page is better viewed when you are logged out or incognito. Here’s my summary and my activity

40

Marco,

I am going to vote with Walter on this—although those hearts by EbenezerScrooge are very important and add a professional touch.

Just my 2 cents. Not meaning to be trite. But I have seen Walter’s point through all of this. I believe he has a serious point. That is not like a Goldmann Sacs or Fidelity investment page for people who what to invest serious money, I think.

While I know you, Dan and Yuval and would be happy to have one of your models (at the right price) I would not spend any time on a page like that if I did not know you.

Jim