Just wanted to re-ask what aslanzadeh asked: How can we see complete simulated performance (OOS) from '99 to today?
For now, to see simulated performance you will have to ask the designer.
There will (likely) be a way in an upcoming redesign. We’ll have more info soon.
Marco, You said "
First thing that had to go was the backtest results. This shifts the focus of a designer from stressing out about past results (to compete with other
models) to future results. "
Now the back test only is 2 years! How is a subscriber going to make an educated decision on only 2 years of data I wouldnt. Its tough enough to make one with 16 years of back testing . Also where does it show when the model was launched and how long it has been live ? This is less information for a subscriber to make a decision and easier for a recent new model to look good but only for the last 2 years . Sorry this is good for model designers and bad for subscribers .
The only data you see is since the model was “launched”, i.e. rules frozen, transaction cannot be edited, etc. What you see is what we consider out-of-sample, not backtest.
There’s only one out-of-sample.
Backtests can be generated and regenerated at will by tweaking rules.
Marco and Marc, I am very subjective and diffidently not right in every aspect of this discussion, but
[quote]
It didn’t work as expected. Models that fell apart after launch were not identified by rolling tests.
[/quote]Having in mind only <3 years OOS, this rolling backtest results is clear indication, that models were optimized to every market condition available in the past data, that means was over-optimized so kept consistency in their performance. It looks like results was misunderstood? IS vs. IS consistency is probably over-optimization mark as well as IS vs. OOS inconsistency.
[quote]
They fell apart because they were not robust to changing economic cycles, psychologies, etc, things that the past 16y of data does not have.
[/quote] …because they were optimized to past market conditions and the name for this is over-optimization. I don’t know why designer constantly repeating this mantra think they are clean. What do they expect from investors? To check their kamikaze models with real money? Sorry, this is not for me.
[quote]
So we’re thinking of other ways to help investors choose a set of strategies.
[/quote]This is a sad statement. I don’t want to be this Mr.Market and I have no Father Time Marc proposing to figure out things even their creators not aware about. Probably I will sound harsh, but at least true. I see NO development on P123 for 2 last years I am in and this disappoints me greatly. I will explain: I prefer content over form. I see form development, but quite opposite move on content. And most frustrating is not a step forward to really protect investor. A lot is done and redone with selections. The point is not about investor choice, it is about trust! I don’t need data great looking at presentation. I need data telling the true story I can trust.
[quote]
There’s more to come. Ability to auto-trade, run combinations, easier interface, etc.
[/quote]Can you disclose more information about “run combinations”, suppose new name for books? Will it involve new functionality (content) like model of models or just re-branding (form)?
[quote]
For now, to see simulated performance you will have to ask the designer.
[/quote]Does this mean the model daily performance downloadable data will be discontinued too?
I agree with the lack of content in changing to smart alfa from r2g. Smart alfa has way less decision making content. I can not even get a return figure for 2 years or since launching date on the models I can only get excess return . I can get the 2 year return in top to bottom order but not the exact amount. The new format allows a 2 year out of sample model to look great, even if the last 2 years were the only 2 years in the last 16 that back tested good. This allows for model designers to try to test for what is working well now. Don’t get me wrong I know all faults of backtesting. Yet yesterday we as investors had both to compare . Today we only have in sampled to out of sample. If I was new to p123 I’m not sure I could figure it out.
I don’t think the removing of the back test results is a big deal. You can still see the backtest results when you create a book and add the assets. You can just add one asset and run the simulation and get your back test results since inception.
What would be nice is if you could create the book simulations only with OS and compare the IS with the OS performance. I do this manually all the time to see which R2G work best together. That’s one of the main objectives of creating a book.
Regards,
MV
Yes, IS performance is very important part of any system along with OOS. How do investors evaluate a system based only on two years OOS? They have to wait another 15 years? Right presentation of IS (at least the average of full pack of parameters of one year one month offset rolling test is required) helps to understand what to expect in future. OOS especially short can be lucky or unlucky realization again, because stochastic component is large especially in low holding number systems. Short OOS alone is not representative statistic at all.
We should distinguish IS fitted negative influence to OOS performance and negative performance due to factors/themes that are not correctly chosen for current market conditions (if it is all cyclical, time will pass and they are going to work again). Negative influence of fitting/over-optimization (for example fitted factors with not equal weights within one theme that catch outliers at specific IS time moments) is the simple difference between average robustness test performance and fitted realized performance during whole IS time period - 15 years )
So first it helps with such systems.
And second you can check your robust model performance during different market conditions in the past even if it won’t be exactly the same again.
To make good systems that work fine after launch is primary task of designers (cause they earn stable money in that case) not yours (primary).
That would be enough if you close one IS realization graph (which was shown previously) but not completely cut IS info at all.
Smart alpha parameters set is not full enough. As Konstantin said form prevails over content and I agree in that case 
Hi Marco,
please do not take tkp’s comment personal.
I have been with P123 since two years and it has developed wonderfully!!! (TRADE, rolling backtests, Canadian Data, Fundamental Charts, Technical Charts, Macro Data/Backtesting, Custom Series, ETF screens and sims, combination/books, Smart Alpha, random function,…)
P123 definitely is the best site of its kind!!!
There is nothing out there, that is so easy to use, so powerful and so much value for money!!!
great customer service, great continuous innovation!!!
!!!I LOVE P123!!!
!!!THANK YOU MARCO AND YOUR TEAM!!!
There are loads of users that feel the same.
P.S. Your approach against over-fitting ( R2G performance on Smart Alpha only after launch ) is very commendable.
P.P.S
And for me as a Europe based investor, you are keeping the best to come. European DATA!!!
I hope we will get a beta for christmas.
PLEASE Santa, PLEASE.
That would be the best christmas ever. I feel like a little child again!!!
Marco, don’t take this to the heart, this is just mine subjective opinion and continue with your quoting:
[quote]
…I will explain: I prefer content over form. I see form development, but quite opposite move on content. And most frustrating is not a step forward to really protect investor.
[/quote]Do you really, really think that removing sim results (and letting them to be published by designers in the description if only they wish for the same marketing goals you intended eliminate by removing sim results) and according performance stats from R2G downloadable list is for investor protection? Do you understand you’ve just killed the most of investors trust?
I see a lot is done for designers and I see nothing for investors. I hear no investor voice asking for better models segmentation or risk assessment!? I see me shouting about fundamental over-fitting problem and designers crying for under-subscription having mostly burning OOS models with unrealistic performance and pricing.
OK, you took out sim results presentation. Not a big deal in visualization, but why hide the stats. Then, do introduce something in exchange! Why not to introduce IS vs. OOS performance. What to hide? We are loudly acknowledge models under-perform, let see investors this in stats with ease. Yes, now it is ugly stats, but who knows, maybe I am wrong and you are right about over-optimization and soon will see recovery, at least we will know it is possible and this could become a huge marketing power. If things get worse, then, what to hide? We need to learn and evolve.
While I do agree on most Marc is outlined above, I don’t understand… Do you, P123 and designers, have Time and Resources to wait-and-see strategy, having in mind most of the models OOS under-performed their sims the last 3 years? This mean 3 years spent just to understand that almost all current R2Gs most probably (what is a chance we will see in the future market conditions like in 1999-2013?) are not suitable for the future. I see a high probability current R2Gs will collapse in the future. What do you expect? How long will it take for designers to learn, redesign, test, run and collect OOS for new models? Should my joke to come back in 10 years become a reality? I have this time. You don’t!
Proposal btw: let designers to have as much private sims as they want, freeze them and let convert into Smart Alpha keeping OOS from private stance. If we don’t know how to build sustainable models and not event sure it is possible at all, let the luck do the job. Who knows, maybe at least 1% of those private ports will survive during the next 3-5 years, at least they’ll have OOS. Maybe it worth a try to let the designers not to graveyard their poorly performing models, but temporarily hide them for sure keeping the record public?
Marco, you asking me what to do? Haven’t you’ve seen my battles with Steve 6 and 1 month ago? I think you need a functional book as model-of-models for investors and for sale in line with Smart Alpha ASAP. It is getting clear to me that it is hard to outperform the market constantly by more than ~5% with single model, but we can try by diversifying and even timing/rotating a number of models in a book. At least subscribers will not be inclined to hurry up selling under-performing model from the book. Maybe this way designers come back to the ground and will start to model and price realistically. For sure this is not a crap-selling strategy.
I am with you TKP. Basically what we had yesterday was a lot more and what we have today is alot less. What subsriber does this make sense to. I guess we should more sensitive to the stress that the model designers are under in having to compete with each other.
As a subscriber, I like it a lot less.
Showing simulated results and , more importantly, making the simulated stats searchable and sortable, guarantees models will be curve fitted. Why? Because everyone wants their model to be listed on top. It becomes a vicious cycle of constant tweaking that is very easy to do. I’ve caught myself doing it many times. And the result is several models with 50% simulated alpha. Then they launch and don’t do much, but the market is up 40% in two years.
Will these models catch up and even pass the market in two more years ? Who knows. But, in the meantime, this is the fastest, surest way to lose customers, forever.
A customer called me asking which model I would recommend for a consistent 40% alpha. All I could say is that Warren Buffet alpha is 6%, if (a big IF) had you invested at the very beginning in 1964. What this is saying to us, loud and clear, is that we needed to do something.
The main thing now is that priorities will hopefully shift. Now there no incentive for a designer to squeeze every bit of performance, rather aim for robustness. Sure, they can advertise it any way they want, but it’s not even close to being the same as before.
It’s also very clear that there are two main types of customers: advanced and investors. We’re doing other changes in P123 to cater to investors. In essence a new presentation , ease of use, and separating them from the advanced/research side of things, with their own discussion, etc. The research side of P123 will basically be what you see now, with only P123 paying members. I see no problem in exposing the backtest results & stats to P123 paying customers which are more savvy. And I see no problem in creating a special category of rocket models only shown to advanced users.
[quote]
…I will explain: I prefer content over form. I see form development, but quite opposite move on content. And most frustrating is not a step forward to really protect investor.
[/quote]Do you really, really think that removing sim results (and letting them to be published by designers in the description if only they wish for the same marketing goals you intended eliminate by removing sim results) and according performance stats from R2G downloadable list is for investor protection? Do you understand you’ve just killed the most of investors trust?9/quote]
Sims are not content and we would be in danger of being shut down by regulators if we were to be seen saying otherwise. Sims are part of the designer’s process of creating.
Investors should not trust sims, and federal regulators make darn sure we clearly inform them they should not (the disclaimers).
But do they trust sims anyway? I hope not - for their sake. We believe not. But even if they’re wrong, does that mean we should support the wrong thing? I refused to pump internet stocks in 2000 even though investors trusted and loved them. I bashed housing in 2006 even though everybody trusted and loved the sector. I dumped junk bonds and put a junk bond mutual fund into treasuries before that market collapsed in the late '80s (and got chewed out big time by shareholders and fund directors – until the market turned and I became a hero).
Sometimes, doing the right thing is easy. Other times, it isn’t. Time will tell whether what we did now will work easily for the p123 community or not. But one thing is absolutely positively clear. We did the right thing. We know it. Federal regulators will know it (if they don’t already). And now, for the first time since R2G went live, I’m unconditionally proud to show it off to sophisticated knowledgeable industry professionals and media.
Hi Marco,
Nice work… Did we loose the interactive graph functionality where we can manually change the dates?
Ilan
Nice hack 
Investors want to see actual performance and not to be mislead with sims. Publishing sims only helps developers who either have poor short term performance or no short term performance at all. It does not help investors (customers). It just gives them a placebo.
My models are new, so I have little OOS data. But I am confident in my models and feel they will do well over the coming years. I am willing to wait. P123 obviously wants to build a brand that has credibility and integrity. Good for all of us.
When I go to the new site and look at Performance->Excess Return 2 years>0, I get 35 models that have been around for at least two years and each have overall performance that have beaten their particular benchmark. Most are open to new subscribers. In the next 3 years, this list will grow, I am sure, and the developers will have 5 years of good numbers to show. Investors, developers and P123 will all win.
I personally feel comfortable with sending my friends and acquaintances to the site now. I did not feel that way with the previous one which showed just 5 years of sims.
We need to give this thing some time to sink in. They are on the right track.
You said,
“It’s also very clear that there are two main types of customers: advanced and investors. We’re doing other changes in P123 to cater to investors. In essence a new presentation , ease of use, and separating them from the advanced/research side of things, with their own discussion, etc. The research side of P123 will basically be what you see now, with only P123 paying members. I see no problem in exposing the backtest results & stats to P123 paying customers which are more savvy. And I see no problem in creating a special category of rocket models only shown to advanced users.”
Okay , So how are you going to decide who is more savvy. Do you you really think you are saving less savvy invetors from themselves. So thats why you have taken the backtests of the P123 models off of 1999 starting dates. Remember those are mostly only oos since to 2007-8 when they were created. Were you trying to save unsaavvy investors from themselves 3 years ago until yesterday when you started R2G. What is a special category for rocket models? Does that mean high fying microcap models will be shone more data than lumbering large cap models. This making less sense to me the more time goes on. I have said in past posts that there are very few R2G models that do not have market timing in there backtests , that are outperforming oos. As time progresses that will become more evident. But if backtesting is now evil why does P123 exist? Backtesting data is the reason most subscribers,subsribe.
Is it not a little hipocritical that P123 ports show backtesting results that are in sample but subcribers of R2G now can be shown only OS data. How can this make sense?
Also I still cant figure how to find all the models by authors names Maybe i just cant see it after an trying . I also cant find how to get an anual return for all the models. Seems that only excess returns are given. Still trying
you can easily find models by developer names/userid by typing in either their name or userid in the upper left search box. works for me. I think you need to clear all the other search criteria first, though.