In this topic we will provide suggestions for improvements.
My first is related to 'Features Stats' tab.
We need to do some work here for sure. It's missing key information.
For example the problem with Surprise%Q is that proper coverage starts around 2004 (not sure the exact month) , but at least 1 stocks has a value back in 2001. This makes the Min Date column useless if you want to keep that feature and just adjust your dataset start date to around 2004.
Second problem is UX problem that is very hard to delete features you don't like. You have to jump back and forth or something.
Thanks , we'll get right to it.
Also a problem for earnings estimate revision up until about 2005, I think. And others features. I had to remove a few features (two features). And/or change the data period for earnings estimate revisions.
i learned something about some of my factors and NAs are a problem and ways to deal with them, e.g., impute etc are important. .Something I am usually too lazy to do.
But I think this is also true: NAs are not as much of a problem with tree models as it is for a linear regression model.
Sorry to admit this but I asked ChatGPT about this. It would go along with "less of a problem" with tree models but I could not get it to say "no problem."
Still you could imagine a random forest (or HistGradientBoosting being a better example) making some of its splits right at the NAs bucket or empty bucket. Ideally, at times, splitting the data into NAs and not NAs (or zero buckets), And even recognizing the times where being NA is useful information.
For example, a small stock with no analyst coverage and therefore giving an NA for Surprise%Q could actually do better because of the lack of analyst coverage (and P123's AI using other data that is not being followed by a lot of people).. A tree model could, at least in theory, recognize this.
Anyway, no conclusions on my part but I am not convinced that the threshold for removing a factor should not be at least different when using a tree model.
@Pitmaster and @Marco, I have this idea for discussion. Why not give NAs a NEGATIVE rank and give the NA returns for those buckets. Therefore a tree model can always, consistently split on the NAs. as the number of NAs changes throughout the years. And use that information.
So again, embarrassed to admit but I wanted to get a first impression of my idea. As per ChatGPT: "
Negative Ranks for NAs:
"Assigning a negative rank to NAs could ensure that the tree models consistently recognize and split on NAs, treating them as a distinct and potentially informative category.
Consistency Over Time:
By giving NAs a specific rank (e.g., negative), you provide a consistent way for the model to handle missing values across different time periods, which can help in making more reliable splits."
Conclusion: You should probably do something like that for tree models unless I am missing something. Forget imputation etc.
Keep and use any information that the NA category or classification provides.
Jim
The intended workflow here was to favorite models you want to add as predictors on the Validation/Models page, then when you're ready, you can use the ⋮ menu to copy them over one at a time.
Show Used is meant to let you add the same models multiple times for validation, useful only when when randomness is involved in the model.
The AI Factors User Guide is a nice introduction for individuals with no machine learning experience.
But it doesn't show up when you initially enter the AI Factors aera. I was using the site for 4 days
before I stumbled onto it.
@marco What do you think about adding a metric that shows the delta between validation and testing results?
It's sensible to look for consistency between those time periods. Perhaps the model that is most consistent in it's validation performance and testing performance is most robust.
Agree:
A few thoughts:
Is there any way to expose the AI Factor? There may be valuable insights there. For example, out of 300 factors it might be primarily using 3 or 4 in a certain way.
At a minimum, it allows me to robustness check using certain elements of it. In a traditional ranking system I like to adjust weights, remove some factors and so forth to see if the core idea holds up or how dependent it is on the optimized weighting. Doing this allows me to use my training period for robustness checking in addition to out of sample periods.
What would the code even look like?
Assuming you literally wanted to look at the code, once you have run an extra trees regressor you could run this to get an output of which features were most important. Something similar can be done with XGBoost. For linear regression you can output the coefficients.
Specifically, feature importances tells you how many times the program thought that a particular feature was the most important feature in a split and how much the split reduced a metric—often mean squared error.
I believe Marco said he will be doing all of the above at some point.
# A regressor example
etr = ExtraTreesRegressor(n_estimators=100, random_state=42)
# Get feature importances
feature_importances = etr.feature_importances_
# Create a DataFrame to display feature importances
feature_importances_df = pd.DataFrame({
'Feature': X.columns,
'Importance': feature_importances
}).sort_values(by='Importance', ascending=False)
# Display feature importances
print(feature_importances_df)
Jim
I was reviewing Chapter 8 of Advances in Financial Machine Learning by Marcos López de Prado, which discusses several approaches to feature importance (e.g., MDI, MDA, and SFI). I’m not entirely sure which of these methods is most closely aligned with how feature importance is computed or interpreted in Portfolio123. Any idea? @Jrinne @AlgoMan @marco
For the sklearn-based tree models, I believe the feature importance being reported is MDI (Mean Decrease in Impurity). The banner suggesting that only certain algorithms expose feature importance makes me think Portfolio123 is simply using the native importance provided by each algorithm, rather than computing a separate importance metric (as would be required for something like MDA / permutation importance). MDA / permutation importance could be applied to any model, so P123 would not need to restrict which models support feature importance if that were what they were doing.
This is from Sklear’s ExtraTreesRegressor documentation:
feature_importances_
The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance.
MDI and Gini importance are used synonymously as the underlying code is the same. Historically the term “Gini” comes from classification trees, but the same mechanism applies to regression trees, where the criterion is MSE rather than Gini impurity. So despite the name, this is best interpreted as MDI.The XGBoost and LightGBM models use something similar to MDI.
Linear models use the coefficients.
I could not find any additional confirmation in the forum or the documentation.
Massive info! Thanks @Jrinne
I haven't read the book in its entirety, but I have researched the three approaches mentioned. You are correct: the feature importance shown in Portfolio123 is MDI. I want to highlight a few critical points regarding those results.
If you have several features providing the same type of information, the importance can be disguised or inflated in the MDI output. This usually happens in one of two ways:
Dilution: The importance is split between the correlated features, making the underlying information appear less relevant to the model than it actually is.
Inflation/Overfitting: Worse yet, the model may place correlated features in the same decision tree, focusing on the extreme differences between them. For example, if you use ROC(150) and ROC(170) as features, the model might fixate on the performance specifically between day 150 and day 170—information that is likely just noise.
Since we are discussing correlated features, it's worth noting that a simple MDA (removing one feature at a time to measure OOS impact) can also be misleading. If you remove ROC(150), the ROC(170) feature will likely "take over" the predictive work, and the performance won't drop. While this correctly identifies the individual feature as redundant, you might mistakenly conclude that momentum as a style is not important to the model.
To truly see the impact of a feature “style,” you would need to identify all features within that same style, remove the entire group, and then run a performance test. I might actually add a Dendrogram to my Python correlation program to help visualize these "Information Clusters."
I know @Jrinne has brought this up a few times. But to give the option to pass NA values as NaN in the normalization prosses is important and should not be difficult to add as an option before normalizing.
At them moment I’m very consistently removing features with high NA’s, which is a shame, losing so much valuable information.
I got a good answer from Gemini when comparing the P123 way and the “normal” way of NA handling.
Question:
*I'm mainly using LightGBM. When Normalizing, what is the better NA handling, give the NA's a 0 value as result after Normalization or pass it as NA?
*
Answer:
In the context of LightGBM, the answer is definitive: Pass them as NA (NaN).
Imputing NAs as 0 (especially after normalization) is one of the most common ways to accidentally "poison" a tree-based model's logic. Here is the technical and practical breakdown of why passing NAs natively is superior.
LightGBM does not just "ignore" NAs; it treats them as a distinct category during the split-finding process.
The "Best Side" Logic: For every single split in every tree, LightGBM calculates whether the gain is higher if all NAs go to the left child node or the right child node.
Predictive Missingness: In finance, the fact that data is missing is often a signal in itself. For example, if a small-cap stock is missing a specific institutional ownership metric, that "missingness" might be highly correlated with lower liquidity. By passing NaN, LightGBM can learn: "If this value is missing, the stock is likely to behave like X."
If you replace NAs with 0 after normalization (where 0 usually represents the "average" or "median" of the dataset), you are telling the model: "This missing value is exactly average."
The Blur Effect: You are forcing the model to treat a "True Average" stock and a "Missing Data" stock as identical.
Mathematical Distortion: If your feature is something like "Debt-to-Equity," an NA might mean the company has no debt (very good) or it's a distressed entity not reporting correctly (very bad). By forcing it to 0, you destroy the model's ability to distinguish between "Average Debt" and "Unknown/No Debt."
| Scenario | Feature | Effect of Imputing 0 (Normalized) |
|---|---|---|
| Growth Stock | R&D Spend | Model thinks the company spends the average amount on R&D. |
| Value Stock | Dividend Yield | Model thinks the company has an average yield (masking the fact it might be 0). |
| Momentum | 12m Return | Model thinks the stock price stayed flat (relative to the mean). |
Use NaN in Python: Ensure your dataframe uses np.nan. LightGBM recognizes this automatically.
Check zero_as_missing: LightGBM has a parameter zero_as_missing.
If you have features where 0 is a valid, meaningful number (like "Total Debt"), keep this False.
If 0 is just a placeholder for missing data in your source, set it to True.
The Only Exception: If you are using a feature where "Missing" literally means "Zero" (e.g., "Number of Analyst Upgrades" where a blank really means there were zero upgrades), then you should impute 0 before normalization.
By passing NAs as they are, you allow LightGBM to decide the optimal "default" path for missing data based on the training labels. This almost always leads to a more robust model than forcing the data to the mean.
One more suggestion, think it has been mentioned before. Give us the option to flag a feature as a categorical feature.
