Chart Average Rolling Alpha by Bucket

Please tweak the ranking system charts to make it much more useful:

In descending order of importance:

  1. Plot a range of returns with std bars or a box plot.
  2. Median average returns.
  3. Rolling periods.
  4. Alpha instead of raw returns.
  5. Benchmark should be the average (equal weighted).
  6. Benchmark should be limited to the universe.

1 - Why std bars?

Because statistically, the greater the magnitude of the out-performance, and the more consistent it is, the more likely it is that it’s not a fluke.

Magnitude is returns. So that’s already on the chart. But consistency is still missing from the chart. Std is one measure of consistency.

Here is jlittleton’s chart which illustrates a way of plotting std:
Avg Rolling Alpha by Bucket With std Bars

Alternatively, a box pot, (also known as a box and whiskers diagram), might be an even more useful way to see the consistency of returns for each bucket at a glance.

Generic box plot (from Wikipedia):
Generic Box Plot

Notice the red line? That could be the benchmark.

2 - Why medians? Because medians are less prone to outliers than averages. .

3 - Rolling periods incorporate more data.

For example, when the re-balance period is one year, a ten year back-test only incorporates ten periods.

If you would have a rolling back-test every week, even if the holding period is one year, it would incorporate many more data points and would show and greater signal to noise ratio.

4 - Why alpha?

There is more signal and less noise in alpha vs. raw returns.

True, you can’t eat relative returns. It’s absolute returns that matter.

Yet, when designing a ranking system you are measuring how well the system is able to pick better stocks from worse stocks. That’s why the same raw returns mean very different things depending on what the market did. If your top ranked stocks were up 20%, is that a good result or not? It depends. What did the average stock do during that same period? If the average was down -50% and your top ranked bucket was up 20% then it did fantastic! But if the market was up 75% during the same period, then being up 20% is not as good, is it?

Instead of alpha, alternatively, you can draw a horizontal line across the chart showing what the benchmark did during the same period, sort of like the red line on the generic box plot above.

5 - The benchmark should be the equal weight returns of the tested universe; not the cap weighted returns of some somewhat arbitrary subset of the market.

Again, the equal weight benchmark has a higher signal to noise ratio than market cap weights.

Please vote for this feature request! Let’s make the ranking system charts much more useful!


1 Like

For clarification of the problem P123 has, it was not that long ago that the use of standard deviation in any metric (including error bars) was a controversial issue. This has never been resolved in the forum (I am not sure how P123 feels about this controversial issue now). And cross-validation is either not understood or not accepted.

But magically, P123 will become a machine learning site in a few week now, we are told.

It is not like @Marco should not be aware. I truly hope so.

Just an observation. That would mean breaking a perfect record of ignoring any requests involving statistical metrics since Parker’s request on information ratios nearly ten years ago and perhaps every one of your requests have been made before (standard deviation is involved in the controversial metric, information ratio).

Happy to be corrected if P123 does not have a perfect record on this.

I particularly agree that nothing can be done without excess returns relative to the benchmark. Something that ChatGPT has helped make available to me with the DataMiner downloads. That has been a feature request multiple times before.

But here is my question on this: Are we suddenly going to get a huge number of our metric requests answered when the AI/ML arrives? You would think so, but we have heard nothing from the AI/ML expert.

Assuming the beta release of AI/ML is not perfect will she take a few requests for statistical metrics (if she posts in the forum for the first time)? Weird to say, but I am not sure. That would mean breaking a perfect record after all.

You are referring specifically to the “Rank Performance” which right now is an embarrassing bucket chart from 10 years ago. Correct?

We are re-doing Rank Performance right now with many new stats and charts. We’ll review your suggestions.

@Jrinne yes, we are close to the AI/ML beta launch (I was finally able to do a whole flow without issues)


Thanks Marco. Can I hold you to that? :slightly_smiling_face:

I want the alpha and its t-value under certain factor models as those in papers.