What kind of live performance is typical with AI Factors?

[rtelford]

Hello all,

I’ve been working on the AI factors, and developed quite a few feature sets, with different models etc.

But the results seems to be the same. I can find some great results on say basic holdout, but when I start testing using rolling CV (what I understand as being more rigorous, but correct me if I’m wrong), and then test in a sim, results usually fall apart.

So if I can’t get results to work in more rigorous tests, I’m obviously not comfortable making these strategies live.

So far, my traditional ranking strategies have worked out better in sim, and in live performance, at least for the factors and universe of stocks I play with (growing microcaps).

I’m hoping I’m missing something, as I really do want AI to work, but I just continue to hit the wall.

So I’d like to ask if anyone is comfortable sharing actual, live performance using AI factors? Preferably at least one or two years of performance. And if so, are there any fundamental learnings you’d like to share (without giving away your secret sauce).

As an example - is “basic holdout” good enough? Is Rolling CV too pessimistic?

As I said, if I’m doing something wrong, I’d like to figure it out, but I'm curious whether my experience is typical, and what others have actually been able to achieve in live performance.

Thanks all,

Ryan

[AlgoMan]

I don't mind sharing some results. I have slowly been transitioning over to AI models because they perform better than my classic models, and a few of my classic models had very high correlation with each other. Believe it or not, I actually have more confidence in my AI-trained models than my classic ones.

None of my live AI strategies are misbehaving—quite the opposite, in fact. I deployed my 9th one the other day. I also run a whole bunch of paper-trading AI models. None of them are performing worse than the benchmark, though a few are performing worse than their backtested results. So far, the only common thread I can see is that those lagging models hold 25 stocks or fewer; all strategies holding more than 25 stocks are performing as expected.

I'll share the live results of my two longest-running AI models. I've posted about these two before, so no cherry-picking. Both are trained on the same features and universe, but one is a linear model and the other is a LightGBM model. Of all the live models I run, the LightGBM model below is my best performer and the linear model is the worst—so the rest of my live models fall somewhere in between.

What I have learned: I skip the last 5 years in my training dataset so I can test on truly unseen data. For the most part, I skip the validation phase entirely at first and jump straight to the prediction phase. I check to make sure the predictor isn't relying too heavily on some odd, random feature that I'm not comfortable with. Then, I step outside the AI factor environment to test the predictor. I also always test it on a completely different universe than the one it was trained on—for me, that means Eastern European stocks and stocks that didn't pass my initial liquidity requirements. If the predictor works across other universes too, I feel very confident. I don't spend time on hyperparameter optimization; I basically just stick to two algorithms that work for me.

After that, I run my sanity checks. I validate with a few additional algorithms just to make sure I didn't get lucky (which is almost never the case). I always do one training run with K-fold cross-validation just to get a sense of how it might have performed during the GFC. It’s not a perfect method, but it’s what we have to work with.

That said, the two models I showed results for above—being my first deployed models—were built the traditional way: validation, hyperparameter tuning, and then generating the predictor.

Next for me, since the AI2.0 seems to be low on the priority list here, is to build a pipeline outside P123 so I can deploy my own custom algos.

[Whycliffes]

I have gradually resumed using AI-factor more and more, after a break for some time. Finally, the tests are starting to look consistently better. Alogman, both what he has shared on the forum and directly, has been an enormous help.

However, I had two questions:

• Why is the holdout period at the end of the test period so important? For example, I mostly use Time Series CV; isn't the idea that each validation period with a holdout functions almost similarly to pure holdout testing? And if that's not the case, how do you then ensure that the last 3-5 years of testing aren't just a stroke of luck or specific market conditions that make it perform exceptionally well during that period?
• It doesn't seem like the choice of algorithms is the most important factor, but rather the features you select?
• Do you have any recommended machine learning books that have given you an advantage when using AI-factor here?

Can anyone provide more information on when we can expect AI 2.0 to be released?

[willx7890]

Regarding books, I think Lopez de Prado’s ML book is the most well-known one and most recommended one. But at the same time, you have to be careful of publication bias, in the sense that he is one of the very few practioner that published a book. Most practitioners at top quant firms have no reason to give out their secrets. The best book on ML would probably be inside the head of the Lead ML Researcher at Jane Street. Just anecdotally, having talked to a Jane Streeter, they basically had to rebuild ML from the ground up at the theoretical level and currently use plenty of algorithms that don’t exist publicly. Would be really interesting if they released some of their internal research one day, but that would probably never happen.

[rtelford]

This is great stuff AlgoMan, thanks, and the performance is very motivating. I’ve checked some of your other posts as well, very helpful. I’ll keep digging and get back to you with more specific questions.

[willx7890]

Just wondering, if you are building your pipeline, are you also getting your own historical dataset (factset , compustat) and live data and coding your own backtester?

[AlgoMan]

I got my back tester already and I'm only using the data download form P123.
I have not found any other data source that I think I would benefit from using, yet.

The only thing left to do is to automate the download of data and upload of predictions (in the format as a stock factor) to P123 (to keep all my models on one platform). Was hoping the function to upload our own pickle files (predictor) straight to P123 was coming soon, would make the process super smooth.

[ScifoSpace]

Some AI results in Small Cap Universe USA, this is and OOS for example, while my Linear Small Cap (US) Systems are struggling with some inverted factors...AI models are doing well...of course there are a lot stuff involved as @AlgoMan pointed out and I made my own research through python . Personally I am running Linear and AI models. Good luck @rtelford enjoy the process and read a lot