Automatically test and optimize ranking systems?

[Whycliffes]

I have 90 nodes that I believe in, which I want to (over)optimize the weight of in a ranking system. I want to test it 4 sub-universes.

I can then weight up-down nodes by ±2-12% by creating thousands of ranking systems. (See the description of Yuval below.) It will probably take months to do this manually.

(I can use Excel and " [text editor]" and " =RANDBETWEEN([LowerLimit],[UpperLimit])" to speed up this a bit:

Is it possible to do this automatically? Has anyone created a Phyton code and API that can do this automatically? Will OPTIMIZER work to do this?

I don’t need to automate the process for the screener or the simulator. A performance test for the ranking system will suffice. Much like the Danp spreadsheet that applies to one node at a time, but something similar for multiple, but entire ranking system: Single factor testing tool developed using the API - #26 by Whycliffes

It doesn’t look like I can test multiple ranking systems with Dataminer: DataMiner Operations - Help Center

Otherwise, I there are several good threads that deal with the questions of automation, weighting, and optimisation below:

Running full simulations through the API

Thanks Dan Well, now I feel a bit dumb I’ve actually tested the optimizer at bit, but I don’t think I realized how powerful it is, and I had a some problems with it - the web page would “hang” if I ran more than 15ish combinations. Returning to it now, and trying another browser (Firefox instead of Safari), it now works fine and seems to be able to do 90% of what I need at the moment. I’d still prefer doing this through the API, it would be easier and faster to work with, and i…

Best way to optimize weight of nodes?

I don’t think there’s a “statistics best practice.” I’ve been trying to come up with one for years. Here’s what I’d recommend. Assign every node a weight divisible by 4. Make changes in the weights to see if returns improve, but always keep every node’s weight divisible by 4. Once you’ve optimized the weights through slow trial-and-error, do it again but make each node’s weight divisible by 2. Stop there. I optimize my systems using this method on subsets of my universe and subsets of my time …

“Here’s what I’d recommend. Assign every node a weight divisible by 4. Make changes in the weights to see if returns improve, but always keep every node’s weight divisible by 4. Once you’ve optimized the weights through slow trial-and-error, do it again but make each node’s weight divisible by 2. Stop there.”

Running simulations in API

Is there a possibility that the API will be able to run simulations in the future? Thanks Tony

Over-Optimization

I have three related questions for P123 users. What are the warning signs that a system is over-optimized? Given two roughly comparable systems (i.e. with a good number of common elements), one of which performs much better in a backtest than another, in what circumstances would you choose the lesser-performing system? Are strategies subject to regression to the mean? In other words, given ten strategies, will the one that performs the strongest be more likely to perform more poorly …

[Jrinne]

P123,

So, I have no problem with people doing the above.

If you want to attract any machine learners in the future you will want to consider using best practices for machine learning. Probably 5-fold cross-validation with an embargo period would be well accepted.

I appreciate Wycliffe’s humor when he says he is “(over)opmiizing.”

But it is no joke that a neural net will give you the perfect solution to the problem with highdsight if you are going to use 150 nodes without cross-validation. More than a few machine learner will find that not useful AT BEST. They may have additional thoughts about that. Not all of them kind, I would guess.

I think you will have to do something like what Wycliffes suggests (I am on your side), I would do it differently I think (which is okay I hope). But you should not look at just my input (if P123 takes input on this). Maybe find what the machine learning community would be drawn to and accept as standard practice (consider your marketing to new members) IN ADDITION TO WHATEVER YUVAL LIKES TO DO WITH MOD().

But you will have to do something if you adopt neural nets, I think. You cannot do neural nets without something called cross-validation which is sort of like what Whyclisses is suggesting. He is not wrong that something needs to be done. People (including Yuval) are already doing it at P123 but SOMETHING will have to be automated if you want to attract machine learners.

There is another way to put this: The present P123 community with constantly revisit the question of overfitting (what it even is and certainly how to address it) and never arrive at a consensus. The machine learning community looking at P123 already have a two-word PRACTICAL ANSWER : Cross-validation and Regularization. If you actually want to attract the machine learning community (which I question at times) you might want to use those words yourself or hire someone to use them in a sentence for the first time–in the forum or the documentation

TL;DR: Whycliffes has a point (even if I would do it differently) and It is not actually optional for neural nets anyway if you are going to go ahead with ML/AI.

Best,

Jim

[Whycliffes]

Thank you Jrinne.

I understand you know Python, have you ever seen or made a ranking weight optimization code that can be used by others?

Or do you know if the Optimizer in p123 can be used for this?

[Jrinne]

Yes, definitely. But it is along the lines of what you are suggesting. And the spreadsheet that I sued was made by Steve Auger. It has been a while. The output from the spreadsheet can be formatted to be copied and pasted into the the optimizer. I suspect you are already doing that.

But as I recall the spreadsheet did what you have already suggested in your post: TAKING THE BEST SOLUTION FROM THE OPTIMZER AS INPUT and randomly adjusting the weights and normalize them with the spreadsheet. Multiple new randomized ranking systems are the result.

Thise are then cut and pasted into the optimizer. Repeat until things stabilize…

Also note I have not used the optimizer in years so I am probably not describing that right and it may have changed to but I think the principle I am trying to describe remains sound.

This is a well known machine learning method by the way!!! I do not think you are against machine learning I just think you are slowly developing what has been done in the past. Your method may be up to about 1980. I hope P123 hires some pros to help us all get into the new millennium…

You might call what you are doing a genetic algorithm!!! Very nice. Also for sure you are really just doing gradient descent which is the center of all machine learning. So again, very nice that you, on your own, are developing the techniques of the last millimiim.

The only thing I would add to your tiechnique, if time is an issue, is that you could start with large jumps initially (I an not sure what the ideal large jump is). Not only will this speed things up but it address the problem of finding a “local minimum.” Which just to summarize is not a good thing to be stuck in and you would prefer to find a “global minimum.”

Once you are close to a “global minimum” (as apposed to one of the local minima which by definition is not optimal) with your larger jumps you can make them smaller. Maybe even +/- 0.5% then or maybe even 0.2%.

But actually what you are doing is pretty awesome!!!. And that is all of the actual advice I have.

Sorry about the preaching or advocacy of advanced machine learning techniques. But I do think you could ultimate arrive at an optimal solution for a linear model in about 5 minutes with an Elastic net. AND EVEN HAVE ENOUGH COMPUTER TIME LEFT OVER TO CROSS-VALIDATE THE HYPER-PARAMETERS!!!

Anyway, this last is basically what Duckruck is suggesting. I think you should support him if you are serious about optimizing a large number of nodes. Or all least let him develop his ideas IN THE FORUM AND NOT LET ANYONE SAY HIS POSTS DO NOT BELONG IN A THREAD ABOUT AUTOMATED SOLUTIONS and see where his ideas lead.

Anyway, none of this is optional for neural nets. It has to be done somehow at P123 if they are going to do machine learning.

Jim

[WalterW]

That reminds me, is anyone using Ranking System optimizer? The last time I tried, I found it very cumbersome. Does it have an api endpoint?

[Jrinne]

I guess I need to say this. The problem with neural nets finding the optimal solution in hindsight is that it will not perform well out-of -ample no matter how good it looks!!! Not an entirly new problem at P123 but we have the opportunity to make it much, much worse (or better).

You will find the same problem with boosting to a less degree and with anything that does not have cross-validation and perhaps regularization to a lesser degree still.

No doubt there will be debate about this but only at P123, I think. The truth is we already accept MOD() can help so anyone debiting this would have to accept its value if they do not want to contradict themself.

BTW, the use of MOD() is subsampling and not cross-validation per se. More in the category of regularization, I think. Or just model averaging if you prefer. But not the final perfect solution, I would guess. Not if you asked someone like Hastie over at Stanford unless I imagined his discussions on cross-validation and regularization. He has nice lecture on Boosting if you intend to use it.

[InspectorSector]

@WalterW

That reminds me, is anyone using Ranking System optimizer? The last time I tried, I found it very cumbersome. Does it have an api endpoint?

I haven’t used it for years. But it is awesome when used with a particular EXCEL spreadsheet. It is a manual process and can be time-consuming. I used it to develop my original Cloud Computing Designer Model and it performed exceptionally well.

If you want to use this approach, then start with a large number of factors and continuously prune until you arrive at a minimum number of nodes with the maximum level of optimization you can achieve. Repeat this at least 5 times, you will end up with different solutions. Then use the ranking systems in parallel to diffuse the over-optimization. This worked well for me. The only reason I stopped was because I found four independent factors that work on growth stocks and I have no need to use the optimizer approach.

As for machine learning, I tried this a couple of years ago and if done properly can result in some minor improvements. However, at the time we were in the middle of the pandemic, and I found that the near-term out-of-sample results were not good because the pandemic had such a radical effect on the markets. I think it would be good to go back and revisit the API and XGBoost now that the pandemic is behind us.

[Jrinne]

SteveA,

You shared a version of your spreadsheet, I believe. Thank you for that. I can vouch for its effectiveness.

Thank you again for the spreadsheet. And I think this addresses Wycliffes question directly.

Edit: It does not matter, as what matters is it that it just works. But do you have a name for your method? Is it a genetic or evolutionary algorithm, for example? A partly-manual version of gradient descent? Do you call it something else?

Jim

[InspectorSector]

Jim - I’m not into the technical buzzwords. I developed the technique and Ralph Nohe provided a nifty VBA-based spreadsheet. I can’t take credit for the spreadsheet. Let’s just call it the Auger-Nohe method of ranking system optimization.

[Jrinne]

Steve,

I hope you will take this in the totally positive way I mean it: “A rose by any other name….”

Jim

[Jrinne]

So again a rose by any other name……

Explanation for why one might call this an evolutionary algorithm and a machine learning tool: You are essentially randomly mutating the ranking weights and testing their fitness in the "environment " with the rank performance test and selecting the one that is best adapted. Then randomly mutating the "fittest"again. Only the strongest survives in the end. Or selection of the fittest as Darwin would put it.

Here is how Wikipedia puts it: Evolutionary algorithm

BTW, did I mention that cross-validation—as a “fundamentally analyst’s technique” if you wish—will not be optional with neural nets? There is no rational debate to be had on that.

**Cross-validation–when you automate it as you will have to–could be made available for many algorithms including an automated Auger-Nohe method (or similar method)—whether you want to call it machine learning or not.

BTW, a “nested cross-validation” would even begin to give you some idea of what to expect out-of-sample as it has a way if giving you a large-sample holdout test set. Not that one should give up using MOD() if they like it.

Jim

[InspectorSector]

@Jrinne

BTW, did I mention that cross-validation—as a “fundamentally analyst’s technique” if you wish—will not be optional with neural nets? There is no rational debate to be had on that.

What worked for me was to choose a market niche that I felt had a strong future. Then “over-optimize” for the previous 5 years (or some time period representative of how you think the future might look). Cross-validation and OOS testing doesn’t do it for me. I think the best way is to choose a small market segment and jump on recent trends. There isn’t much out there that works on all markets for all time periods and trying to find such a system results in mediocre results at best.

[Jrinne]

Steve,

I accept that what you do works for you without comment other than it does work (I have seen it) and I am glad.

But neural nets are like black magic (or black boxes at least) that can turn truly evil if you do not do cross-validation.

You remember probably: cross-validation for early stopping at a minimum maybe? But dropout or other regularization methods (e.g., L1 or L2 regularization) and usually k-fold validation to tame the beast.

Nested cross-validation being computer intensive but giving you a better idea of the out-of-sample performant. For a neural net anyway.

Even for me, there are some techniques less needing of cross-validation (mostly linear techniques for example). But I think cross-validation should be an option (and necessary at times) for people as long as P123 plans on neural nets.

I do mean truly evil when I refer to unchained or uncontrolled neural nets. You could try to put a stake through its heart later if you want—assuming members do not become hypnotized by the beauty of the in-sample results and want to keep believing them.

Jim

[Jrinne]

@duckruck

using the same weights to test outside of the universe intended to be used.

I call that cross-validation or a holdout test sample myself.

If you are changing anything at all after you “test outside of the universe intended to be used” then it is in fact cross-validation.

If you have manipulated everything done everything and then just for fun moments before you fund you “test outside of the universe intended to be used” and make no changes based on the results then I would call it a holdout test sample.

Either way, I like it!!!

So if you did do an elastic net ever, how would you determine your shrinkage hyper-variables (α, alpha and λ, Lamda you call them on your programs)? Not a grid search using cross-validation? Seriously, how do you set these hyper-parameters then?

You mentioned deep learning. You do not use early stopping (which uses cross-validation usually I thought)? I have never done a neural net or boosting ever in my life without cross-validation except as satire to show people the “perfect model” which of course does not work out of sample. It does LOOK good; I wil say that. It doubled every year with no drawdown, Made the perfect pick every time. It is called curve fitting. The curve can be fit exactly with enough factors, layers and time-running with a neural net. Hmm…I thought that might convince people. Interesting.

Jim

[InspectorSector]

@Jrinne

But neural nets are like black magic (or black boxes at least) that can turn truly evil if you do not do cross-validation.

NNs require a Run set and Test set as a minimum. Typically, additional data is held back for out-of-sample (OOS) testing. While this is all great for operating NNs, the truth of the matter is it all amounts to optimization, including the throwing out NNs based on OOS data. Why not cut to the chase and throw all of the data into the NN and optimize across the entire data set without holding back data? I know this is a bit of a simplification, especially for stopping the NN training but essentially captures my thought process.

In any case, I don’t view cross-validation with other universes or sectioning of a universe as fruitful. My opinion is that if you find a couple of meaningful factors that span universes then cherish them, it is a once-in-a-lifetime occurrence. In most cases, you will either be fooling yourself into thinking you have something that isn’t there or end up with a truly mediocre system that has marginal results across multiple universes. I prefer highly targeted systems and lots of them to spread out risk.

[Jrinne]

@InspectorSector

Why not cut to the chase and throw all of the data into the NN and optimize across the entire data set without holding back data? I know this is a bit of a simplification, especially for stopping the NN training but essentially captures my thought process.

Have you tried that? Maybe you should. I got an ETF model that doubled each year (on average).

Basically when we talk about curve fitting a neural net is flexible enough that it can fit the curve perfectly. Generally thought to be a bad thing.

IN-SAMPLE THE FACTORS DO NOT EVEN HAVE TO HAVE ANY REAL PREDICTIVE VALUE! THE NET CAN STILL FIT TO THEM AND FIND THE PERFECT SOLUTION!!!

I would joke and ask if you want to buy my model but someone might not be sure if I was using satire when I said it I would have been.

Jim

[yuvaltaylor]

In response to the initial post by Whycliffes: While it’s not entirely clear to me from your post what exactly you’re trying to do, have you tried the optimizer? Or the DataMiner? I think most of what you want can be done with one or both of those. Or perhaps you can be more specific about what exactly you want to do? The part I don’t understand is what you mean by “test.” Personally, I can generate dozens of ranking systems and “test” them with the DataMiner. Also, the optimizer offers a range of testing options.

[Jrinne]

@duckruck

I didn’t say I don’t use cross validation, I said what I mentioned before wasn’t cross validation.

For example, the cross-validation you refer to for determining hyperparameters is not the out-of-sample testing I’m talking about.

I think that time series split is the typical solution for cross validation of this type of financial data.

The hold-out sample is still a randomly selected sample in the universe intended to be used, not a sample not randomly selected and outside the universe.

Duckruck,

My bad. One of my post above was talking about what Steve is doing but was sent as a reply to you. I believe this caused some confusion. Some of what I said clearly does not apply to what you are doing (as you already know).

I will have to be more careful about who I am replying to in the future. I had been occasionally posting by replying to the last post in a thread.

Best,

Jim

[InspectorSector]

@Jrinne

Basically when we talk about curve fitting a neural net is flexible enough that it can fit the curve perfectly. Generally thought to be a bad thing.

Any kind of optimization, neural net or otherwise, is an art not a science. There are a few general rules of thumb that can improve your results:
(1) Garbage in, garbage out. If you start with junk factors then your results will be bad.
(2) The topology of the neural net should be such that memorization is minimal (minimum internal nodes relative to the data points)
(3) Delete inputs that aren’t contributing to the output. Minimize the number of inputs while maximize the output.
(4) Generate multiple neural nets and combine the results.
(5) The NNs should have different input factors.

Follow these rules and you might just get positive results. The goal is certainly not to memorize the dataset.

What you call cross-validation, examining the NN against a different dataset and deciding whether to keep the NN or toss it, in fact it is just another level of optimization. Every decision you make in developing a trading system is an optimization.

[InspectorSector]

@duckruck

However, this may or may not improve performance. I still prefer models that at least exhibit mediocre performance outside of the universe you want to trade in

This is a little twist on what I was saying. You will likely get mediocre performance “inside of the universe”. I don’t really care what you get outside the universe.

Everyone has there own strategies. That is why the stock market works… I am past developing universal systems that work on everything. I don’t believe that exists, not now and probably never did. I try to capitalize on trends in market niches. For example, it is hard to lose on AI stocks right now. Tune the right combination of factors that are outperforming at the moment and you could have a goldmine on your hands. (Same thing with cloud computing 5 years ago.)