Why is the sorting by rank showing extra trees II higher than linear lasso? I can't make sense of it.
The Compare All tab has an Edit button under 'Rank & Columns' which allows you to specify which stats to use when calculating Rank. What is yours currently set to?
The Compare All tab has ranks that are independent for Validation and for Testing depending which of those you selected. The Lift, Portfolio and Return tabs only show the Validation ranks even it you choose to show the Testing results in the charts.
Ah - ok. Do you think we ought to have a rank based on the delta between test and validation or also on testing? It's confusing to me that I'm looking at testing but other data on page is not corresponding to it.
If not, a label that the ranks are only on validation would help a bit to clarify things.
Thanks for that.
I agree that it can be confusing. I think the idea was that on the Lift, Portfolio and Return tabs the most common use case for Validation vs Test is that the user wants to pick a model and flip back and forth between the Validation and Test charts. Reranking them each time the user switches between Validation or Test doesnt seem ideal either. Also the list of models that have been validated and the list that was tested can be different ie not every model that was validated was tested and vice-versa. So switching between Validation or Test would also make some disappear/reappear in the list.
I think the best solution is just to add the label that mentions the list is based on Validation results and not Test.