These tools seem too complex to me and may increase the risk of overfitting to the data.
If you prefer to focus on factor selection and economic intuition rather than factor weighting, you might consider a custom discrete optimizer.
Such an optimizer can:
Set factor weights on a fixed grid (e.g., min = 0.025, max = 0.10, step = 0.025).
Accept a dictionary of fixed weights for core factors you always want in the model.
The advantage is avoiding wasted computation on meaningless precision — in practice, a weight of 0.0487566486845644 will not perform differently from 0.05 out-of-sample.
And you can do that with the API and/or downloads, right? I imagine it gives quick and useful results — especially if you include cross-validation, which I assume is possible with both the API and downloads. Very nice!
In a way, it feels a bit like a “brute-force” evolutionary algorithm — and I mean that in a good way (simple and just works while using the same theory). The name doesn’t matter much as long as it works!
Phase 1: Exploration with a Genetic Algorithm (GA)
Problem: The search space for choosing 100 criteria from 3000 is astronomically large. A simple search would get lost. We need an algorithm that can efficiently explore this vast, discrete landscape.
Algorithm Used:Genetic Algorithm (GA) .
Why GA is Used Here:
Population-Based: A GA works with a "population" of many potential solutions at once. This allows it to explore many different areas of the search space in parallel, reducing the risk of getting stuck in a single, poor-performing region (a local optimum).
Handles Discrete Choices: GAs are naturally suited for combinatorial problems like selecting a subset of items (the criteria). The concepts of "crossover" (mixing two good solutions) and "mutation" (randomly changing a criterion) are perfect for discovering novel, high-performing combinations.
Goal of Phase 1: To perform a broad search and identify a small "shortlist" of the most promising combinations of criteria . The exact weights are not the focus here; the goal is to find the right ingredients.
Output: A JSON file - Phase1
Phase 2: Exploitation & Refinement with CMA-ES
Problem: Now that we have a few promising sets of 100 criteria, we need to fine-tune their numerical weights to squeeze out the maximum possible performance. This is a continuous, high-dimensional optimization problem.
State-of-the-Art for Continuous Problems: CMA-ES is widely considered one of the best algorithms for optimizing continuous parameters (like weights) in complex, "black-box" scenarios where you don't know the underlying mathematical function.
Learns Parameter Dependencies: Its key strength is the "Covariance Matrix." This allows the algorithm to learn the relationships and correlations between the weights. For example, it might learn that increasing the weight of "P/E Ratio" is often effective only if the weight of "Debt-to-Equity" is also decreased. It adapts its search direction and shape intelligently based on this learned information, making it far more efficient than random search.
Efficient Local Search: It excels at homing in on the precise peak of a promising region, which is exactly what we need for fine-tuning.
Goal of Phase 2: To take the best criteria sets from Phase 1 and perform a deep, surgical search for the optimal weights.
I’ve also experimented with Bayesian Optimization, which you mentioned in your original question. My early runs produced strong results, but over time I found it to be quite sensitive to the hyperparameters. I often had to re-tune just to get decent convergence. It was always a bit hit or miss, and at times the hyperparameter tuning itself felt like a form of overfitting.
That said, I really appreciate the elegance of the explore/exploit framework it uses (e.g., upper-confidence bound). But the Gaussian assumptions baked into the surrogate model may not hold in real-world return data — and I suspect that’s part of what makes it so tricky to tune reliably.
Your two-phase approach — using a GA for selection and CMA-ES for fine-tuning — really resonates. It’s pragmatic and well-structured.
I might add that @pitmaster’s method has no such Gaussian assumptions at all — assuming he’s not using a parametric fitness metric (which I’m guessing he’s not).
There are certainly several good methods worth exploring, but your approach already looks excellent.