I was trying to tweak the universe back and forward, revalidating and backtesting all night to make the backtest with AIFactorValidation be more similare to my Predictor Rankings backtest... but I came to a point that I'm backtesting something completly different from what I actually will trade live... very frustrating
Backtesting with AIFactorValidation is fine for anyone trading a more static universe like SP500, but backtesting micro cap strategies with AIFactorValidation, really don't see any value in it, it's too far off.
@marco I am getting the following error with the Screen Backtest.
ERROR: Error relating to predictor 'linear ridge-26' in AI Factor '1MRel-1w-Unicorn++AI-SR-200F': No predictions are available on 11/28/2020. Check the date/frequency in which AIFactorValidation() is being used. Saved predictions cover 01/04/2005 to 11/30/2021 every week.
I did the rerun for my model. Are there restrictions in the Settings and Validation Method?
This is the formula: AIFactorValidation("1MRel-1w-Unicorn++AI-SR-200F", "linear ridge-26")
@ marco. About the predictor. I am thinking that once we are in PROD for the AI predictor, we will want to keep the predictor up to date (re-train once per month / or once per quarter / etc.)
When updating the predictor, a good new feature would be to use a rolling window approach, where you retrain the model on the most recent data (e.g., last X years) rather than the entire historical dataset. This can help the model focus more on recent trends while still being informed by historical data.
Which model? Most models have an element of randomness (they have the #random tag). Impossible to get the exact same results from two trained models for the same dataset. And, if your system buys for example the top 10 stocks, then any single stock will impact the difference greatly.
I would not try to predict tot return, relative is better. But that's a different story.
To test a model in a screen or backtest you need to reference a Validation model, not a Predictor. There's a clipboard icon next to the validation model. Try clicking that.
May not be what you want. We'll release soon a way to have many more folds so you can test what happens with more frequent model retraining. Didn't seem to help in the tests I did.
@sraby You will need to do a Save As of the AI Factor and start fresh with a new one. The older version is incompatible with validation predictions backtests.
Too many backtests were failing. We'll see if we can improve the reliability. Maybe just allow it with restrictions, like no more than one AI Factor for entire strategy.
Maybe I am missing something but if I build a model one one universe I cant backtest this model on another universe. This is option is probably not feasible in back testing if you are only doing 1 year.
One option that I was thinking that could help with the compute is to have option to score at validation other selected universes that would either cost API or just compute and storage $ passed down onto consumer. B/c AI training is on demand anyway this probably easier to manage an on demand cost. (maybe free tier storage up to xxx GB and passed that $xxx per 1TB or something).
We discussed this internally and we can add another option when you do a Model validation to not only "Save predictions" but also "Save Predictor".
This way you will be able to access the same exact predictor used for the validation, and use in backtesting, use it with different universe, and use the "force into universe" flag. It will be slow though. And would still need to have some limitations.
We'll also have to have some safety mechanism so that you don't , for example, run 100 validations with 10 folds with "Save Predictor" ON, as this will use up a lot of disk space (a predictor can easily be 2GB).
It's a bit of a project, few days of work . Anybody else feel free to voice your support for this.
Yes, I agree with you. It would be great if you could provide alpha returns using a risk model or volatility-adjusted returns. I'm currently trying to do that manually with P123 terminology.
Just to clarify, I mentioned that as an example of a target you automatically provided as part of the AI feature. What I'm really asking for is if you could share a public example of how to create a long-short test with a prediction horizon longer than 13 weeks. I initially requested 1 year total return (an arbitrary request), but I'd be more than happy with 12-month relative returns.
Randomes is not the issue here. I had to train with way lower liquidity requirement and way higher market cap restriction to make sure that my stocks don't fall out of the universe. Then backtest on a more restricted universe, but now I'm traing a model on a universe that I don't wont or can't trade.
I support that idea of backtesting on a saved predictor, or limit backtesting to use one predictor instead of multiple, or anything to be able to use "force into universe".
@marco with this moving out of beta, it is possible to share a full blown step by step of this request so that we can understand how to use all these forecast horizons? Thank you.
I did another check. On my test with Predictor Rankings, 1% was sold with a Rank below 25. With AIFactorValidation more than 30% of my stocks was sold with a Rank below 25. I assume that it's stocks that is not in the universe that is given NA and ends up in the bottom of the rankingsystem.
If I use Sell Rule -
Rank < 90 and Rank > 25
with my AIFactorValidation backtested strategy to fore the stocks that is falling out of the universe to not be sold, I get a better result than when I just use
So traditionally with AI prior to this update, my workflow was to take a final AI factor, put it in a screen, and layer some rules on top to further filter the predicted stock list. I'd then run this screen for the current date to get the list of top screen holdings.
Now it seems like this isn't possible - I get this error:
ERROR: Error relating to predictor 'abc' in AI Factor 'xyz': No predictions are available on 08/28/2024. Check the date/frequency in which AIFactorValidation() is being used. Saved predictions cover 02/23/2013 to 05/04/2024 every week.
The issue here is I am not trying to use AIFactorValidation(), I'm also not concerned with doing any backtests in this screen. I only want to be able to access the current predictions in order to layer more rules on top of them.
I think this would also make it tough to view the daily predictions for an ensemble model?