I was curious to tap into the knowledge of the community and see if anyone has faced the same question.
Maybe Marc G might chip in as well…
I am building systems that scale well i.e. large or mid-caps only. I usually start from the SP500 and then try to extend the Russell1000.
But it happened several times in the past that when I extend from the SP500 to the (P)Russell 1000, the additional ~500 stocks bring the performance down.
See for example → port ID 1528792 (very simple: SP500 universe and Greenblatt ranking) and compare with port 1528793 (same Greenblatt ranking but uses “the next ~500 stocks” = PRussel1000 - SP500 stocks)
The SP500 does perform better than the one with the “additional 500 stocks”
It is important to note that the SP500 does NOT represent the 500 largest stocks by market capitalization in the U.S. based on the interview in the AAII journal (Aug 2017) of Elisabeth Kashner (director of ETF research for FactSet).
I quote:
In any case, I was naively thinking that these additional ~500 stocks are similar enough to the SP500 stocks (i.e. large companies) but there seem to be some more fundamental differences.
I noticed that the perf of my systems degrades when there is a value component to it (e.g. the Greenblatt example above). It seems that I would need to add another component to a ranking that works well for SP500 to make it work well for the “additional 500”
I was about to embark on an investigative journey.
Any thoughts anyone?
Things I had in my mind to investigate in order to try to explain what happens and add the relevant variable(s) to convert a ranking system from SP500 to the “additional 500”:
Mktcap comes to mind but based on the quote above, it certainly cannot explain everything (since the SP500 goes as far as the 900th to 1000th rank to fetch its components). In any case, if size still matters even for already pretty large firms, then what criteria might help in a ranking for “the additional 500”?
Sector / industry weight i.e. the additional 500 might be more or less overweight in some sectors / industries
Based on the quote above: “Second, like the Dow Jones industrial average, it’s designed by a committee” → ah? what criteria do they decide on? Can this lead to the explanation?
It doesn’t answer your question directly, but I think it’s instructive to see how closely you can come to recreating the S&P 500 universe using only ranking criteria.
While Russell constituency criteria are very concrete, inclusion/exclusion in the S&P is largely qualitiative. However, if you do try to reverse engineer the investment board’s criteria, my experience is that you can get pretty close based on size, liquidity, ownership, market share, geography, and notoriety factors.
Anyhow, those latter factors other than market cap size may help explain the difference. It’s also possible that your model which works better on the S&P 500 may be an example of the unintended curve fitting which is pervasive in this business.
Curve fitting: impossible to exclude and probably contributes a small amount but I am pretty sure it is not the main thing. In the 2 examples above, I use the Greenblatt ranker made by Marc in 2009(?). Just quickly looking at the out-of-sample period (e.g. start, say, 01 Jan 2010), my point stills hold true with the SP500 >> {R1000 - SP500} over the period 2010-2018
In the back of my mind, I am trying to tie this back to the Gordon Dividend Growth model i.e. I suspect that the SP500 eligibility criteria do (inadvertently rather than by design?) set a homogeneous range for a number of the variables detailed by Marc Gerstein in this model, thereby leading to a degree of consistency across the ~500 stocks selected (probably mostly introducing some risk homogeneity and removing risk outliers e.g. “US only no ADRs”, “real businesses and not umbrella construct like MLPs, etc”).
That makes them comparable with each other when later using whatever ranking involving value or other criteria i.e. you are comparing apples with apples which is probably not the case when using only Mktcap as the sole selection criteria (R1000)
Jerome, I really have nothing substantive to add other than to say I’ve noted the frustrating aspect you mention of the R1000 not performing like the SP500. I’ve suspected that some of the subjective criteria utilized in selecting the SP500 stocks were adding elements of quality or momentum (they probably don’t toss out the quality and/or good performing companies). It could also be that being in the SP500, due to indexing, is itself a return enhancer? But in the end I haven’t investigated it much as my focus as an individual investor has been more on smaller caps. It is an unexpected characteristic of the R1000 though.
Primus has it right. With the SP 500, objective criteria are more of a guide, and the selection committee has latitude to use subjective judgment. In a sense, you could say that the S&P 500 is an actively managed large-cap portfolio (albeit one with low turnover) that can be measured against a R 1000 benchmark.
Also, if you’re bored or suffering from insomnia, you may want to dig deeper into disclosure documents to see the details of how index providers compute market cap. Some multiply price by float. Others multiply price by shares adjusted for float and others say price by shares with shares defined in a way that aims at some notion of float. And no two definitions of float are the same. Various providers look to exclude shares owned by institutions if they believe the blocks are not likely to trade and often, this is determined by index providers by getting on the phone and talking to institutions to supplement or replace objective algorithms.
The bad news: You’re rarely if ever going to be able to precisely match up or replicate any indexes.
The good news: You don’t ever have to precisely match up with or replicate anything (even tracking ETFs have prospectus language that takes them off the hook for this sort of thing) and the consequences of inaccuracies would have to work incredibly hard to claw their way up to becoming merely trivial.
After further review, of the stocks that are in the Russell 1000 but not in the S&P 500:
76% of these Russell 1000 stocks are too smaller for the S&P 500.
12% did not make money of the past twelve months.
3% are not USA.
Less than 1% (2/490) were MLPs.
Another 19% did not fit any of the above criteria.
27 stocks in Sp500 not in Prussell1000
But many remaining prussell1000 are part of S&P mid cap and even small cap: SP400 (overlap 264) and Sp600 ( overlap 7)
If you start with Universe “All Fundamentals” and then compare the ranked list to the S&P 500, the top 500 ranked stocks comprise 82.8% of the S&P 500. The top 1000 comprise 94.4%. For frame of reference, this compares somewhat favorably against a much simpler ranking system based solely on current float size in which the top 500 companies comprise 61.2% of the S&P 500, whereas the top 1000 comprise 97.6%.
By comparison, P123’s Prussel 1000 Universe contains 963 constituents as of 4/30/2018, altogether comprising 94.6% of the S&P 500. The top 500 ranks, according to this proposed system, comprise ~86% of the S&P 500.
The relatively high proportion of S&P 500 constituents outside the top 1000 ranks is curious. How and why are certain companies included in the S&P 500 when there are ostensibly more apparent candidates? In spite of its limitations, this particular exercise in futility demonstrates some of the ways in which the S&P 500 qualitatively and quantitatively differs the Russel 1000. In particular, the efficacy of additional criteria suggests that the S&P favors liquid issues with high share turnover, which can be seen as a kind of active management and could in turn be seen as exerting a factor interaction effect with other rank/screen criteria.
Alas, this is just a first draft; no doubt someone can improve the S&P 500 “hit rate” with minor tweaks. If one of you find any easy tweaks that improves the “hit rate”, I would very much be interested in that information.
You’re overlooking the key difference here: the reconstitution schedule. The Russell indexes are reconstituted only once a year, and the last time was June 23, 2017 (the next one will be June 15, 2018). The S&P indexes are reconstituted four times a year, most recently on March 16. So the Russell 1000 is based on the market prices of over ten months ago, while the S&P 500 is based on market prices about six weeks ago. You need to run your screens based on the actual reconstitution dates, not based on today’s price.
I noticed that Tesla is in the R1000 and not SP500.
Good example of having the market cap but not the other qualities that the SP500 committee is looking for.
[quote]
I noticed that Tesla is in the R1000 and not SP500.
Good example of having the market cap but not the other qualities that the SP500 committee is looking for.
[/quote]Tesla didn’t make it because it’s TTM earnings is negative and the criteria explicitly requires positive earnings.
… yet there are 42 companies in the index with negative NetIncBXor earnings (as reported???) over the previous twelve month reporting period.
The S&P 500 is like the Marine Corps. It’s hard to get in and even harder to escape.
//dpa
[/quote]As Yuval wrote, the S&P reconstitutes quarterly. Those 42 stocks must have had positive earnings at the time of reconstitution or they would have been excluded by the objective quantitative criteria.
To put it Marine Corps terms, it’s like having a young child apply to the Marine Corps. Even if the acceptance committee would want to take him they wouldn’t be able to.
It seems that you may have the impression that these criteria are purely mechanistic, when in fact the investment criteria are fuzzy.
For example, one of the historically strongest requirements is that the companies be based in the US, yet there are 24-25 companies not domiciled in the US. The other merits were forceful enough to cause the board to include these even though they failed to meet all the “boolean” criteria.
In other words, the Index Committee does have the final say in deciding if a stock counts as being truly in the USA.
Positive earnings on the other hand seems to be a absolute requirement for index additions. Other than two exceptions (index migrations and mergers with index stocks), I didn’t see a mention of committee discretion for index additions.
What is the positive earnings requirement? [quote]
The sum of the most recent four consecutive quarters’ Generally Accepted Accounting Principles (GAAP) earnings (net income excluding discontinued operations) should be positive as should the most recent quarter. For equity real estate investment trusts (REITs), financial viability is based on GAAP earnings and/or Funds From Operations (FFO), if reported. FFO is a measure commonly used in equity REIT analysis.
[/quote]
For index subtractions however, there is some discretion involved even in regards to earnings[quote]
S&P Dow Jones Indices believes turnover in index membership should be avoided when possible. At times a stock may appear to temporarily violate one or more of the addition criteria. However, the addition criteria are for addition to an index, not for continued membership. As a result, an index constituent that appears to violate criteria for addition to that index is not deleted unless ongoing conditions warrant an index change. When a stock is removed from an index, S&P Dow Jones Indices explains the basis for the removal.
[/quote]