How do I reconcile these results?

Use case for lift charts: When predictions of the actual returns are made available it will be nice to know how much bias there is in the predictions. Especially, if predictions of returns are added to estimates of transactional costs, by the member, to determine whenever to buy, hold or sell an asset.. Plus, why not add bias estimates to the variance estimates as part of a basic understanding of a model just for completeness, if nothing else?

Wow. I don't even have to add a feature request for this. Nice work on this P123!!!!! :slightly_smiling_face:

To me it looks like the lift chart provides many of the usual metrics of machine learning such as RMSE, MAE, R^2 etc. P123 adds to this with a lift chart.

What I think the lift chart adds to this is that it shows any bias of the results. In Korr's first post, that I will not duplicate here, the predictions seem to be biased to more favorable predictions to the left and biased to lower predictions (compared to the actuals) at the far right.

As an aside, this seems to be some sort of kernel regression with some smoothing. Like a LOESS regression or someting.

TL;DR: Seems like the lift chart might be an addition to the standard machine learning metrics. An addition with the purpose of showing any bias. And it might be serving its intended purpose.

A nice addition if Korr verifies the numbers (accounting for the smoothing of a Kernel regression) are correct. I have no opinion on that. I'm not going to run the numbers my elf and I default to the idea that Marco probably got it about right for now.

Addendum. I checked my post for accuracy and got one correction. Claude 3 does not think the lift chart shows very much bias. A good thing if you think about it.

" Lift Chart Purpose: Your interpretation that the lift chart is meant to show bias in the predictions is insightful. The chart compares predicted values (blue line) to actual values (red line), allowing visualization of where the model over- or under-predicts.

  • Bias Visualization: While you mention bias to the left and right in Korr's first post, this particular chart doesn't show significant bias at the extremes. The predicted and actual lines follow each other quite closely throughout the range.
  • Smoothing: Your observation about smoothing is astute. The lines do appear smoothed, which could indeed be a result of some form of kernel regression or LOESS-like technique."