I Get different simulation results with exact same system / timeframe

yuvaltaylor · October 9, 2020, 3:40pm

Some thoughts…

@Yuval - considering the kind of results you are getting, would you be willing to do 1h “training” sessions for the relatively advanced users the same way that Marc used to do?
I am not sure what your exact background it (your profile seems to point to a being a writer but you have way too much Quant knowledge not to have done sth else as well). Many of us will surely benefit from your experience. I know I will.

I am conscious this could require some preparatory work but it could be minimized if caveated as “for advanced users only” and “not meant to be featuring pretty slides”. I .e. you could have one slide with an agenda and the rest could literally be running some demo screeners or orally explain your methodologies incl how your leverage Excel when P123 is not enough etc. This clearly would not be a basic training e.g. “here is how to write a rule in the screener”

You do not need to divulge your own systems. What is of real interest -and a recurring topic on this forum- is how you ensure that / test to make a system robust. For example, your post today above can effectively be your first (and possibly many more) session(s).

We’ve been doing some webinars. I could do a few if there’s enough interest.

yuvaltaylor · October 9, 2020, 3:42pm

Both.

Jrinne · October 9, 2020, 3:50pm

Yuval,

Thank you.

Can you tell us which provider is used for each of your designer models?

Marco seems to have some pretty strong opinions on the quality of data (CompuStat vs. FactSet). I do not have an opinion on this yet myself. Not an informed opinion.

But for people with a strong opinion (like the opinions expressed above by Marco and Quantonomics) would knowing which data provider a model uses be something Subs could look at? Maybe this is already provided. I do not subscribe to a model.

Just an idea/question. As I said, I do not have an opinion now. But the question of whether subs should be allowed to look at this if they think it is important aside, I would not mind being able to compare the 2 groups to get somewhat of an informed opinion on the best use of my $12,000 (per year).

That is real money for some people. And I could rent a pretty nice luxury car with that kind of money (or come up with enough money total to buy a small beach house/condo). I do not think this is a trivial question for most of the members.

For some people at P123 there may not be a choice and maybe they would not mind possibly being able to turn off the noise (and have some data instead) when people on this site tell them how bad their data is.

Best,

Jim

yuvaltaylor · October 9, 2020, 6:30pm

All my designer models switched to FactSet when P123 made the switch in late June and have been based on FactSet ever since. Prior to that they were based on Compustat data, like everyone else’s.

Jrinne · October 9, 2020, 7:24pm

Thank you Yuval,

So I will bet anyone a dollar that the best comparison I can come up will show FactSet data to be at least as good as CompuStat (and CapitalIQ).

In case anyone has not already noticed the Designer Models are already ahead of their benchmarks over the last 3 months. While over the last year the Designer Models trail significantly.

I will bet that the year after switching to FactSet data looks better (mean excess returns of all of he models with complete data) compared to the year before using CompuStat.

Hard to do worse frankly. And I will actually be up $12,001 over the CompuStat crowd if I win (so not too embarrassing for me). Yeah, I will take that bet.

Yuval, I suspect you might agree that the trashing of FactSet data without evidence could be a mistake. Maybe you do not agree, but I do think P123 did a good job on the FactSet data and I would like to learn more about that data (as illustrated in my above posts). If you are not particularly fond of the FactSet data I would be happy to take a dollar bet with you.

I am just looking for some real data on some of this stuff. And I am willing to bet that some of (a lot of) what we think is true or are told is true is not. I do not claim to know ahead of time where we are wrong. I would like to be exposed to enough data to eventually find some of the things I have been wrong about.

For those who already know the answers, I will take your bet.

Best,

Jim

RTNL · October 10, 2020, 2:24am

I’d like to give you my philosophy and my story in order to put all this into some perspective.

I first started using P123’s ranking systems five years ago. I backtested very heavily. I quickly realized that if I ran the same backtest on two different dates I would get slightly different results. I shrugged and accepted that. It took me much longer to accept that a stock that was ranked #6 on one day could be ranked #52 the next day and then back at #6 the day after that. I’m still having trouble with that one. But I accept that no data provider can be perfect, or even close to perfect. For me, Compustat and FactSet both have strengths and weaknesses. I love the increased coverage we have with FactSet and the speed with which they process domestic preliminary reports. I love the PIT-ness of Compustat data, with their effective dates and the fact that they don’t overwrite preliminary data. I like the way FactSet classifies industries (RBICS) better than Compustat (GICS), and Compustat’s methods of standardizing line items I find a little more intrusive than FactSet’s. On the other hand, I think Compustat may take a better approach to depreciation and amortization (though I’m not positive). It’s a fine balance.

I backtest a lot, and heavily. It has served me well. I trade a lot, and I’m up 52% YTD; my five-year CAGR is 36%. But I’ve learned that backtesting has to be done smartly and realistically. I don’t focus on whether a backtest gave me an alpha of 25% or 30%, or whether it was 25% or 30% last week. I focus on what backtests can tell me about factor choices and combinations. I try to look at things probabilistically. If system A consistently backtests better than system B, it is more likely to perform well out of sample. Maybe not always, and nothing’s certain. But all other things being equal, it is more likely. And more likely is the best I can hope for.

So my advice is as follows. Don’t worry about the actual numerical returns your backtests get. Don’t try to maximize those returns because they’ll change from week to week. Instead use backtesting to compare systems, factors, and approaches. Design your strategy according to what backtests best over various datasets, maybe using various performance measures. And maybe mix in some elements of strategies that performed second best, just to be safe.

If you focus on whether your backtests are completely accurate, you’re missing the forest for the trees. They’ll never be completely accurate. They can’t possibly reflect the fills you’ll actually get. They can’t possibly reflect the quirks of future stock pricing, or the way factors go in and out of favor. The only thing that’s accurate is your actual trading returns, and even there, five years of real-time trading is a very short time and won’t necessarily mean anything regarding the next five years. I have a great track record now but it could all go to hell extremely quickly. Maybe over the next five years I’ll consistently underperform my benchmark. It’s entirely possible.

But is it probable? That’s the question I want to ask. I believe that you can maximize the probability of strong out-of-sample performance by using comparisons and robust methods (try to break your strategy and see what it takes to do so), by investigating companies and strategies as much as you can.

There seems to be an assumption in this thread that backtesting should aim for precision, and if it lacks that precision, we should look for that precision elsewhere. That we should look at backtesting scientifically. That we need exactitude, not vagueness; consistency, not variety; hard, firm answers, not further questions.

I believe that in the financial world, such thinking leads to overconfidence and overfitting. Personally, I would rather get a lot of surprises from my data than none. I would prefer that my data present me with questions in addition to answers. I believe that my models will be strengthened rather than weakened by being repeatedly tested on slightly different data.

Especially lately, because of the switch in data providers, we are finding a lot of things that need fixing. And the impact of those fixes is not negligible. Last week it was interest expense and EPS actuals (which also affected EPS surprises and other estimate factors). We’re looking at other things that need fixing right now. I’m sure we’ll find more in the future. So our data will never be static. It will always be getting better and better.

I’ve spent too long talking about myself now. As product manager at Portfolio123, it is my job to see that YOU get the highest quality data available, and that you are happy with the data you’re getting. I’m trying to convince you that you should be, but if I fail, please don’t hesitate to tell me so, and to urge me to do better.

And if you think you see some fundamental mistake in our handling of data, please continue to point that out. Bugs are always cropping up.

I agree, one should not fixate on numbers, but rather on construction strategy. However, given that one thinks it was done thoughtfully, the discrepancy in returns is disconcerting.

RTNL · October 10, 2020, 2:25am

We’ve been doing some webinars. I could do a few if there’s enough interest.
[/quote]

I would look forward to that. I enjoy your posts.

judgetrade · October 10, 2020, 10:18am

@yual, yes that is the right approach, do not get spooked by the difference in the backtest, test a lot, digg into the factors and backtest. Then choose the ones with the best theoretical base and captital curve behaiviour and trade them…

Jrinne · October 11, 2020, 10:11am

Jerome,

I think you make an excellent point here. And I spent some time looking at Marc’s stuff at Chaikin Analytics (again) this morning.

I am not one who thinks the posts have to stay on the topic of the original post. Your posts on any topic are always welcome on any thread as far as I am concerned. I truly enjoy and find value in every one of your posts.

If you meant this post to be related to data at P123—Marc is at Chaikin Analytics now?

And if I wanted to blindly follow Marc’s methods that is where I would be now: Chaikin Analytics (with their data provider). It appears that Marc has a lot of input there.

And this is just a plain simple statement: That is were I would be.

Thank you for reminding be to go to Chaikin Anaylitic this morning. Marc is excellent. And he does not like to brag about it but his Finance Degree and long experience have served him well.

Expanding on that last a little. I do not want to necessarily want invest in the same way Marc does (although subscribing to that site is not off of the table for me). But Marc does–by long thought process or by natural instinct developed through years of training–arrive at some very good answers. Pretty much always good opinions that need to be taken into consideration. And I am sincere in that.

You make an excellent point. I do not see how it related to my desire to understand the data I am currently using here at P123 to make decisions on my retirement fund and that is all good. Thank you for your post.

Best,

Jim