If a dataset is on QuantConnect, it’s safe to assume that we could also make it available to our subscribers.
I have no experience w/ Estimize but I would be open to trying. However, given the expected charge, it better provide some amazing utility. And how likely is that?
I would personally subscribe to this if it is available. Tons of potential new factors can be created if we can combine these with existing P123 factors.
Marco,
I agree with Azouz (and others) that I would to look at this. But also, I could easily make a decision as to a factor’s usefulness in about an hour (less really), IF THE DATA IS PIT.
So I guess that would be a question for me: Is the data PIT?
Assuming that it might be, I note that perhaps with a trial-period (or short subscription) your AI specialist could make the same determination as I might make. They are trying to sell this to you? Will they let you look at some of the data?
You would think that they could make some past data available. If they are hyper-paranoid about their past data they could encrypt it (i.e., mask the ticker names).
Many might be able to make this determination in short order using a variety of methods. Methods might vary by preference for P123 members. BUT HOW LONG WOULD IT TAKE YUVAL OR DAN TO MAKE AN ASSESSMENT OF THE FACTORS? Dan, for example, has a spreadsheet of a huge number of factors and seems to make an assessment of the usefulness of each of these factors in a timely manner. Timely enough to be useful for his investing is all I know for certain.
P123 staff’s assessment (including you, Dan, Yuval and the AI specialist) might be different than the one I would make or Azouz or SteveA or Feldy. BUT ALSO MAYBE NOT TOO DIFFERENT.
If we are to make the choice independently, could you (somehow) give us a single-factor rank performance test for some of the factors? Maybe compare the different offerings that you mention? As a start.
If I have to wait until P123 offers this as a subscription can I get a one month trail offer with some comment on whether the data is PIT after any manipulations by P123 (e.g., lag if warranted)?
TL;DR: You could consider a detailed look at the data rather than relying on random, relatively uniformed, comments from the forum. I count my comments as being uninformed at the time. I cannot predict what I would do once I became more informed.
Jim
It would be the wholy grail (and therefore I am sceptical!) if we can get very good estimates as a data set. Jim is right, it needs to be pit.
All my eps estimate strategies outperformed 20 and 21 but had a real bad streak in 22 and so far in 23.
I think what would be needed is, the community can test it and then let the strategies run for some time before we go live. If the data is bad (non pit or other reasons) strats can fail fast and we see this before we put capital in.
If the test phase is well (could be tested via public R2G portfolios) I am in.
Andreas
From their FAQ:
Is Estimize data point-in-time?
Yes, we keep point-in-time data for all estimates contributed to the platform. We never delete estimates or change estimates. The sanctity of this data is extremely important to us because our clients need to be able to trust that the historical data they are backtesting is what they would have seen at the time in production.
To be clear, it’s not just about estimize data. What we need to decide is where to spend our resources, and adding datasets to our simulator requires some retooling. We have something rudimentary for Imported Data Series and Imported Stock Factors, but it’s not scalable.
Estimize would be the first one. Many others seem interesting, like twitter sentiment and many more that use natural language AI, etc.
In that context I think it is definitely worth while for our next project.
Marco,
That sounds great!!! I have mentioned natural language AI myself. Link to ChatGPT language processing ChatGPT
I do not mean this to imply that ChatGPT is the best natural language AI for possible consideration by P123.
My only addition, really, is some confidence that the forum AS WELL AS you, Dan, Yuval and the AI specialist and anyone you may consult with can make a well- reasoned decision.
Jim
Sounds interesting.
But to be honest, I would much rather have Asian or Pacific stock data available or be able to backtest further back in time.
The impact of testing strategies over long term data in multiple regions makes a much larger difference to me at the moment then adding another dataset (even though that would be valueable as well of course).
In my opinion an equity strategy backtest is truely solid if it is (1) univeral, (2) timeless and (3) makes sense from a financial or business perspective.
Adding new data from for example Estimize might give a few new effective factors, but adding a region or extending the backtest period would potentially give information about the effectiveness of way more factors (the ones that P123 already offers).
I give higher priority to extending the ways to test the robustness of current factors, than to add new factors.
Best,
Victor
We will decide about Asia soon. The European ROI is about breakeven now (but a lot of upfront work is out of the way so Asia will be easier)
2 Likes
Might be best to determine the desirability of this proposal by conducting a user poll. The quantity and intensity of votes could help determine whether this is a productive use of resources. My $.02 anyway.
Clearly that will be a huge addition in my opinion. Any single alternative data factor could be translated into so many new factors (using formulas, regression, interaction with other factors…).
The European ROI is about breakeven now
You laid the groundwork with no cost. Now, you will be ready. When international stocks outperform, there should be a surge in subscribers and a surge in ROI.
1 Like
Why would this be any better than what one can get out of the P123 Core: Sentiment ranking system?
1 Like
It is different data. Another way to measure sentiment and complements what we already have.
Reason I like alternative datasets (in a broad sense: top-down, bottom-up, extraneous) is that they would be ideal with AI/ML model. Plug them in and let the training factor them in or not.
So maybe after the AI launch it’s a good next project.
Here’s a site that lists them Database - AlternativeData with interesting industry stats (from 2018 so a bit stale).
Marco,
Let me stipulate that you can do that. That you could input a bunch of factors into—say XGBoost—and not overfit. That the program will “learn” (in the machine LEARNING sense) which factors are most important and weigh them properly. How well that works could be a topic for another post. Again, just accept that for now.
Again using XGBoost as an example (it also works with random forests) you can see how valuable a factor is by using “feature importance.” This is provided in the XGBoost program.
Just one link: Feature Importance and Feature Selection With XGBoost in Python
Okay another: The Multiple faces of ‘Feature importance’ in XGBoost
And documentation from XGBoost documents (a few paragraphs down): Feature importance
My simple point is this: There are some opinions above as to how valuable Estimize data and/or other data might be above. You can get a quantitative answer to this question.
TL;DR: Pretty simple to get a quantitative answer in XGBoost on which factors will work best and ultimately to calculate which one will give you the most “Bang for your Buck” in a business sense if you plan on doing much machine learning. Once you see if a factor is worth the price then let the machine do its thing. But see how much you are willing to pay first—especially if you are going to then use it in XGBoost.
Jim
Hm…
I am wondering if this is not a very sophisticated form of overoptimizing the past, with mostly dismal performance in the future ??
Werner,
Edit: Extreme overfitting is possible with a lot of different methods. I agree that machine learning can DEFINITELY BE MISUSED. I have also been able to overfit some models using the P123 optimizer in the past. There are ways to mitigate these problems.
My present models are performing well and if I were to detail them one would have to agree that they are resistant to overfitting despite being heavily reliant on Python programs and other platforms.
We have heard little about what will be done at P123 as far as cross-validation etc in that regard. But the tool I suggested could actually be helpful if used properly.
Or could be misused as you accurately point out.
My point was different. Assuming it is possible, one might want to see what they are paying for. In as objective of a way as is possible. And if someone suggests using machine learning then I do not think it is too radical to suggest a machine learning method for that.
Jim
I have a real world example over the past couple months that’s caused me to shift my priorities for 3rd party datasets. News sentiment would now be at the top of my list.
On Jan 10, my short model recommended shorting CDZI, which at the time was a 150 MM market cap stock with no analyst coverage. From what the ranking system could see, the stock was overbought, but ironically there’s a news story that shows up on the P123 timeline feed announcing a development partnership that’s clearly driving the price movement.
The problem is that my P123 ranking system is completely oblivious to the news – all it can see is a large upwards price/volume move. At the time, this was a 150 MM market cap stock with no analyst coverage, so I couldn’t rely on any analyst related factors in P123 to explain the price move.
My next thought was to check Estimize, but there’s no crowd coverage for CDZI, so no luck there either. As mentioned above, it looks like the universe coverage tends to skew larger cap, and a lot of folks here tend to play more in the small cap arena, so the intersection may not be great.
A news sentiment data source could be immensely valuable here, especially for Biopharmaceuticals where there’s often price movements driven by FDA approvals/rejections. These news sentiment data sources likely have wide universe coverage and may be informative, timely, and orthogonal to existing factors.
Some news providers listed on Quantconnect like Tiingo or Benzinga provide raw news which is not really feasible for consumption on P123, but others like Brain build their own NLP models and provide sentiment scores which could integrate nicely. I’ve seen others on Factset Marketplace like Sentifi, but have not tried any of these out personally.
There are other providers of sentiment scores based on Twitter (e.g. S-Factor) or StockTwits, etc., but I worry about susceptibility to manipulation and lack of history compared to news feeds, so they’re not as intriguing to me.