I am not getting any data. Have a look
I think this happens when you try to save historical data. If you run the screen with today’s date, you should get up to 500 lines? This also happens when there are too many columns of data but it looks like you only have a few.
I do understand the P123 has contracts and obligations with FactSet.
But we can get this data—without the 500 line limit—thru the API and/or DataMiner correct? It seems artificial and random that we cannot get it through the screener but can thru the API. It is possible that FactSet may not really think this limit or distinction is important to them. It could just be something P123 has not looking into changing even though it has been requested by members in the past but not followed up on in the forum.
If P123 does not object to requests from member and seriously considers request from members, I wonder if P123 might consider whether it is possible to increase the number and quality of downloads directly into Excel without having to use the API? That would include looking into changing the 500 line limit.
Jim
Hi Jim,
Regarding the API - The API and DataMiner do not allow fundamental data to be retrieved unless you have a data license from FactSet. You can only get pricing related data and ‘derived’ data like ranks, etc. The screen run endpoint in the API will return all the rows from the screen, but the output only contains P123 UID, Ticker, Name, Last, MktCap, SectorCode and IndCode.
Dan,
Thank you. I understand much of this I think and very much appreciate any clarification.
If I am not mistaken, P123 undertook the API when I made Marco aware in a post that fundamental data is NOT required or even helpful for XGBoost, random forests etc. Fundamental data is simply redundant and completely unnecessary if you have the ranking data for many machine learning models. Why would I want another row of completely redundant data?
I have no need for fundamental data. I am not sure where you got the impression I wanted it.
Ranks are good for me. Sometimes I seem to be able to get ranks from the screener.
I had thought that there was a 500 row limit on the number of ranks from the screener but more rows could be obtained from the API. I could be mistaken.
Am I missing something still? Again to be clear I only need ranks. XGBoost just needs ranks as the person developing AI/ML at P123 already knows, I hope.
More rows for rank in the screener would be appreciated. I also understand that this may not be immediately possible—largely due to any negotiated agreements with FactSet. I am not asking anyone not to honor any contracts. Just to look at whether this is important to FactSet—since they are already giving out a lot of rank data in the API–next time you talk to them.
Thank you.
Jim
1 Like
Jim,
Regarding ranks specifically, you are correct that the API and DataMiner do not have row limits. But keep in mind that the API/DM do not use the screener operation to get that data. Changing the screener on the website to return rank data for all the rows seems like it would make it even more confusing for typical users (not somebody only looking for ML data). Say a typical user downloaded data from a screen with a current As Of Date that returns 2000 tickers. There are already various rules for how many rows and columns of data we can return depending on if the As Of date is current or not. Currently the NAs would start appearing at row 500 but with the proposed change there will be some columns (FRank, ZScore, price related factors, etc) that have data for all the rows. That would be confusing.
And currently if you try to download data for a non-current As Of Date, you get all NAs like in the case at the start of this thread. With the propose change you would have all NAs except for any columns that have ranks, etc. Also confusing.
DataMiner has the DataUniverse operation that was designed for the purpose you mentioned. It is simple to use with no programming skills and you can pull ranks, etc for a range of dates for up to 100 formulas and automatically save that to a file. I would think that most users that want ML/AI data have basic programming skills and could also use the API if they prefer that over DataMiner.
I will look at DataMiner. I did not like the API much myself. initially I did not find it useful at all with the example downloads. Maybe that is better. I am sure it is in fact.
BTW, I do not think I am the worst Python programmer in the forum (certainly not the best though).
I get that some people might prefer to just not have the data but then why would they be looking at the 501st row then? Does everyone look at the 501st row if they are not hoping to find data there?
Or even trying to download more than 500 row in the first place?
I guess that would be very confusing to some and difficult to put a message there. Or answer in the forum.
I do think P123 is headed to just being an API where sometimes what is download into the API can be put back into a ranking system and sometimes not. Generally requiring Python to process the downloads.
I appreciate your response. Truly appreciated. well-reasoned and I do not have to always agree.
I can probably move to the API if I need to (guess I will have to if I want to develop a new port or maybe use DataMiner) and I definitely use P123 every day the way it is.
There are other ways to do this that work and are less work, I think. I look forward to seeing if the AI/ML makes it easier for some.
Best,
Jim