What exactly does AIFactor's numerical output represent?

AI Factor Discussions
1

First post here! Before I get to my questions, I just wanted to say I'm a huge fan of Portfolio123, the AI Factor tools, and this community. The amount I've learned over the past few years has been incredible. A big shoutout to Andreas, @judgetrade for helping me through the learning curve.

I'm trying to better understand what the numerical output value returned by AIFactor() actually represents.

For example:

AIFactor(...) will return values like:

0.198
0.214
0.256

A quick way to see this is in a Screen by increasing the threshold (e.g. AIFactor(...) > 0.20, then > 0.21, etc.) the higher the value, the less the totals.

I have a few questions:

  • Is the value returned by AIFactor() the model's raw prediction, or is it some kind of probability-like or normalized score?

  • Is the AI Factor output calibrated, or are the absolute values arbitrary and only intended to be ranked?

  • Is there any advantage to using an absolute threshold (e.g. AIFactor(...) > 0.20) instead of Rank > X?

  • Does a higher AI Factor output value indicate greater confidence in the prediction, or simply a stronger prediction?

The reason I'm asking is that I've noticed different AI models produce different output value ranges, and from what I’m seeing, models with higher validation results produce higher AI Factor predictor output values.

Eg: a validation Avg% High of say 95% will top out at 0.26, whereas a validation model with an Avg% High of 75% will top out at 0.18.

I've also experimented with using AIFactor(...) > X in buy rules and < X in sell rules. (They take ages to run a sim)

Within a single AI Factor model, the output seems to behave much like a rank gate. However, seeing how the output ranges differ across models, and appear to increase with stronger validation results, this has me wondering whether those values are actually comparable across models.

If the AI Factor score is calibrated, it seems like there could be some interesting applications. For example, could a multi-AIFactor strategy combine different targets, say 2M, 3M, and 4M Total Return AI Factors, and only select stocks where all models exceed a score threshold such as 0.20?

Or combine a linear model and a tree-based model trained on the same target, and require both to exceed a given output threshold. Other ensemble approaches too, weight formulas, etc.

This post could save me hours of experimenting, as the output could just be another expression of rank for all I know, and using AI Factor in buy/sell rules takes the simulations 5-10 minutes to load, vs seconds with rank.

Thanks in advance. Looking forward to learning more.

2 Likes
2

I'm not a ML expert, but ...

That's my understand of the ML environment. Users who want a deeper understanding of what their specific predictor's values represent would benefit from further research into the underlying algorithms.

A good starting point is the AI Algorithms documentation (AI algorithms - Portfolio123 - Help Center), which describes each supported model and links directly to the reference documentation for the underlying libraries, including scikit-learn, XGBoost, LightGBM, Keras, and DeepTables.

3

According to the documentation, the AI Factor numerical output is not a probability or confidence score, it is the expected value, or in our case the expected return? Does P123 do anything with that output on their end? I would be nice to have confirmation.

This is pretty cool because it is the only output that allows multi-AIFactor within one simulation, as far as I know. And its not rank. There's meat on the bone here but I'm not entirely sure how to take a bite, other than a filter. Maybe AIFactor output's growth over time? Growth over time plus filter?

I'm working on an off-platform 2MTotReturn in parallel with a 3MTotReturn run.

The "overlap", the stocks that both models choose, dramatically outperform the ones that arent in overlap. To the tune of ~20% avg return on the stock pick overlaps, and ~6% on the non-overlaps. Its very tough to figure out how to test this in platform though. Fortunately I can run a bespoke sim with LLM tokens, but I lose the accuracy of Sharpe, DD, etc., its looking at total return and annuals a this point.

Also the data suggests that "confirmation", meaning a stock was purchased by one model, then another model picks it later is the key signal. This later date confirmation is what I'm targeting to backtest, but I think its only possible off platform. When both models pick a stock at the same time, this avg expected yield is still better than the overall average, but later date confirmation seems to be where the lions share of the alpha lies, in my run at least. Anyhow, I'll share progress if anyone is interested.

1 Like
4

It is the expected value of the target.

So if your target is 3 month excess return compared to the S&P 500, then the numerical value that you see will be exactly that.

5

But the target is normalized - either rank or z-score, no?

6

Hmmm sometimes you grow so used to your thinking habbits that you don’t question them anymore. That is what I might be doing just here.

I think you might be right and the target value gets normalized.

7

The cautious thing to do is to interpret the output as a rank ordering. What you have posted looks a lot like raw model output. A lot of models, like XGBoost, have a tendency to produce poorly calibrated results. There are ways to calibrate them, however. And some models are better at producing calibrated output right off the bat (thinking specifically of logistic regression w/o regularization).

Much of Portfolio123 is ranking-based anyway: Pick the top N stocks, etc.. These outputs are very compatible with that.

1 Like
8

Hi walter, just a related follow up question about this, given the way p123 aifactor currently work, do we have any way to donwload the trees say for xgboost or lightgbm and run an interpretability analysis on the tree? Are we able to inspect every tree if that is something we want to do?

9

No, there's currently no way to download the trained model file for any AI Factor algorithm, including XGBoost and LightGBM. The trained model stays on our servers, so inspecting individual trees isn't possible today. We also don't currently offer SHAP output, which is the usual way to get per-stock interpretability without the model file. If either of these would be useful to you, I'd encourage you to post a feature request so it can get consideration.

2 Likes