That made me curious, so I ran a screen for S&P500 stocks on my TD Ameritrade account. It returned 499 stocks.
Hi, Marco,
S&P has been charging for constituent data for a long time now. While I am good with you guys wanting to be more resource efficient, I recommend you and your team refine the proxy concept before cutting us users off.
The proxy you present is a snapshot of a single point in time. I’d like to see more evidence that shows that the proxy’s performance is not significantly different than the S&P 500’s. As it stands right, I see too much divergence to accept your method as a good proxy. It seems like the current high rate of overlap between this proxy and the S&P 500 may be somewhat spurious.
See: https://www.portfolio123.com/app/screen/summary/228032?st=0&mt=1
Compared to your proxy that has an rsq value of .79 against the index, my attempt has a value of .96. My method is not important. What is important is that there is great room for improvement before your team hands us a worthy proxy.
SPY, the SPDR® S&P 500® ETF Trust seeks to provide investment results that, before expenses, correspond generally to the price and yield performance of the S&P 500® Index.
So why don’t we simply periodically download the holdings of SPY which have to be published daily. That will give us the composition of the S&P 500 at no cost. Number of holdings= 505.
https://us.spdrs.com/en/etf/spdr-sp-500-etf-SPY
Same for MDY, the SPDR® S&P MIDCAP 400® ETF seeks to provide investment results that, before expenses, correspond generally to the price and yield performance of the S&P® MidCap 400® Index.
https://us.spdrs.com/en/etf/spdr-sp-midcap-400-etf-MDY
The screen is flat with 0 holdings at the beginning , that’s the cause. It’s because I believe the Float factor is NA. With a start date of around 2004 it’s rsq is .97. Also my screen has “Universe(sp500)” as the last rule to see how many stocks match.
In any event this was just an exercise to demonstrate that offering a rule-based sp500 is not possible, and confusing.
Something along the lines of what Georg is suggesting, I’d like to direct your attention to this Quandl database ETF Global Constituents: https://www.quandl.com/databases/ETFC/documentation
The history only goes back 2 years, but you get constituent data for 2,000 ETFs. So you would not only get the SP500 (SPY) constituents and weightings but for many other ETFs. This would allow us to build fundamental aggregates for a whole range of ETFs…
They don’t say what the price for the database is however…
Steve
Would it be feasible to keep your PIT constituents data and run it forward with alternative data, such as the SPDR S&P 500 constituent lists?
Just to be clear on the issues:
SP500 as a benchmark is not in question We use SPY and that etf, along with the others, remain part of the p123 suite of licenses.
The issue is constituent lists. Algorithmically, they may be impossible to recreate because S&P Index Committee judgment is no small thing (What makes the active-passive debate so funny is that the S&P 500 is really an actively managed portfolio). Before we signed on with Compustst (S&P subsidiary), we manually created the constituent lists and mismatches likely occurred because of timing. Every now and then Marco would send me an IM saying something like: “When was the last time we checked the S&P constituent list for changes. Maybe we should do it again now.” And I’d respond with something like: “Oh #$&! That again, yeah i suppose we should do it now.” I’d do it manually by surfing the S&P web site hunting for announcements of changes and hoped to avoid making too many mistakes.
I can’t officially speak for Marco until he signs on the dotted line, but at the moment, our inclination is to sign on for now and continue to evaluate the impact on our user base. I suggest that those who are using SP constituent lists use this time to prepare for the possibility we may decide next year to drop these lists.
The fact that S&P is gouging . . . There really is no other word for it, the people with who we regularly deal are powerless to help because SP put its index licensing into a separate profit center group outside of their control or influence — is already starting to do what Economics 101 suggests ought to do; attract competition. I already know of one index provider, Solactive, that has a suite of similarly-targeted market indexes and is competing against S&P on price. Others may surface.
Are they the same as S&P? No. Will your strategies have different results is you switch? Probably. Will your strategies be worse if you switch? They shouldn’t be and if they are, you may have bigger issues to solve than your starting universe. Are the alternatives of lesser quality than S&P. Yes if you ask someone who works at S&P; Who the heck can say? For the rest of the world. But one thong is clear . . . Even the best of the best becomes mundane or crappy if the vendor raises the price high enough.
P123 is not unique. This is an issue impacting the industry as a whole. My long-term forecast: Either S&P will have to moderate their pricing dreams, or they may eventually cease to become the universally accepted “standard” they are today.
So my advice: Even if we do stick with S&P for now, take some time to wean yourself off of it. The laws of economics may wind up forcing your hand, so take whatever interlude you can get to address it gradually.
As to keeping PIT constituents and running forward with alternative info, unless something fell through the cracks as to our rights with the index, the answer is probably “no.” When a license terminates, the departing licensee is typically required to scrub historical data from its systems.
I’m confused to what is being lost… Are you talking about the universe itself? As in, when we’re doing backtesting or live strategies the option of using the S&P500 as a universe will go away?
Hi David,
Both the link to your screen and to your blog are broken for me. Just sayin’
Walter
I appreciate the fact that you are working to mitigate this issue and the heads up.
I don’t know the solution, but I found that fundamental data for S&P 1500 (and S&P 500) stocks is much cleaner than other stocks. It makes a big difference in my value systems.
At this moment, nothing is likely to be lost. But looking ahead, it is the SP500 universe that may go away to be replaced by a P123 approximation, as we now do for Prussell1000 fro Russell 1000, etc. ( No, that P at the start of the name is not a typo; its there to differentiate between our approximation and the real thing.)
I guess I’m having trouble understanding, but I can go to the SPY ETF website and pull up the holdings.
https://www.ishares.com/us/products/239726/ishares-core-sp-500-etf
Right now it’s showing 509 holdings, but it seems to be a pretty good SP500 tracker. I don’t understand the behind-the-scenes programming required to grab data, but a periodic pull of this data from one of the major index funds that track the SP500 other other index would be fine for my purposes. I’d really hate to see P123 held over a barrel, especially if it’s 20% of data fees, over something like this.
I agree with Chipper about the ‘cleanliness’ of the SP1500 universe versus just using PR3000, for instance. I feel more confident in using a base universe that has been pre-scrubbed than one that is just based upon marketcap, for instance. I know the S&P is doing this in effect for me.
Maybe this could be an area where P123 and its community comes up with some criteria that has added value.
For instance, I would like to see a base universe that use has something like Rating(“Quality”)>20 which I can use for operating, non-financial companies.
Or a similar one for banks (which I know has completely different criteria than an operating company).
Just an idea…
iShares Core S&P 500 ETF (IVV)
At iShares one can download historic monthly constituent data of their ETFs back to Sep-28-2012. So it is easy enough to construct a point-in-time universe from this information for the S&P 500.
We did this for USMV and are able to run a PIT strategy from 1/1/2013 which periodically selects 9 stocks from its holdings. The strategy has a turnover of only 58% and a CAGR of 24%, outperforming USMV which only produced 14%.
Georg, how are you importing the data into the simulation?
Walter,
We made inlists of the holdings every 3 months. Then you make a universe that holds all the inlists and future inlist which are empty at this time.
You start the sim on a specific date and in the buy rules use portbars to call up the specific inlist applicable to the time period of that inlist. That gives you a PIT simulation as it will only select stocks from the currently active inlist. Going forward you need to fill the predefined inlists every three months and also run a total on the universe so it updates.
The only inconsistency is that your universe gets bigger all the time because it will hold all the stocks dropped from USMV over time. But USMV has a turnover of only 27% so that’s not a big problem. If you are using RANKPOS in the sell rules you may want to change the position number every few years. For my strategy the universe now holds 292 stocks, while the most recent list holds 214 stocks. Of course you can also remove the redundant stocks from the universe as you update every 3 months.
Thanks Georg! I didn’t consider using portbars. Nice!
No issues from my end on using the SPY constituents.
Yuval, Marco and Marc, thanks a lot for the transparency and for asking our opinions. Much appreciated.
The proposed proxy screen reduces by 3 percentage points the annualized return and increases the drawdown by 7pp of my oldest screen in use on its out-of-sample-period (7 years, with a few forgotten revisions on the way so not 100% OOS). But, I’m not sure the difference between 23% and 26% CAGR on a 7-year period is really significant about the quality of your proxy or of my screen, or of any of them. I can live with this uncertainty.
- Maybe the proxy screen can be improved.
- No issue for me if you decide to use constituents of SPY or IVV to make a S&P 500 universe as PIT as reasonably possible.
- Anyway, I wouldn’t pay S&P for that. Even if it’s perfectly legal it looks like racketeering.
Additional clarifications.
What’s at stake here is the SP 500, 400, 600 and 1500 index universes and their historical data. They are currently included with our S&P data agreement and go back to at least 1999. The constituent data will disappear around August from our data service. Their index data will be marketed separately and they are asking around 20% of our current costs for it. Of course there are a lot more indexes than we need, but they don’t care that we do not use it.
If we do not accept their terms …
- We will not try to re-create them using rules.
- We will have a backup using ETF constituents data providers. Since the SP 500,400,600,1500 all have ETFs we should not loose any functionality. The only loss will be historical data that goes back to 2007 for the SP500 (not sure about the others yet) and maybe a little bit of “precision” as far as when stocks enter/exit the index.
This one is $240/mo http://masterdatareports.com/Content/Subscribe.htm
That sounds good to me and we will not have to raise prices! It will also give us something we do not have: ETF constituent data for all US etfs which we may want to use for other ideas.
Thanks