I have spent a few evenings building a Bayesian Hyperparameter Search using Optuna, and it works like a charm.
One very powerful feature is the ability to use different optimization objectives that push your hyperparameters toward different goals. For example, instead of just maximizing IC, you can focus on the tail ends using Conditional IC, or optimize for risk-adjusted returns with Top Bucket Sharpe. The learning objective (what the trees train on) stays the same — the trees are still just trying to predict returns. But the optimization objective tells Optuna which set of hyperparameters produced the most useful predictions, and that changes everything.
These objectives can be improved even further. There has been a lot of discussion about the issue of turnover cost when the model picks high-spread micro-cap stocks. With this grid search you can take turnover cost into account. I did a simple trial with a Turnover-Adjusted Sharpe — it adds a cost each time a stock enters or leaves the top bucket, which forces Optuna to steer the hyperparameter search toward parameters that produce more stable, cost-aware rankings. So the target never changes, the tree structure doesn't change for any given hyperparameter set — but the selection pressure on which hyperparameters win does.
If/when we can build Streamlit apps on P123, I would be able to use real time-varying spreads per stock instead of a fixed cost, which should improve the cost awareness even further.
To guard against overfitting the search itself (meta-overfitting), I also added a chronological holdout test set that Optuna never sees. After the search completes, all trials are evaluated on this holdout data, so you can see whether the top-scoring CV trials actually held up or just got lucky. A scatter plot of CV Score vs Holdout Score makes it easy to spot — points on the diagonal are robust, points below it are overfitted.
There are still wok to do, but here are some screenshots of the workflow I built for the hyperparameter search. I have been using Claude to build a Streamlit app for this — so much easier when you don't have to start from scratch learning a new GUI framework.
