Downloading large files of Factors fails

General
1

Hi,

The message says that one cannot download files that are greater than 500 million data points but the downloads repeatedly fail at smaller file sizes for smaller stocks but will allow larger file sizes for larger stocks. Is there a different file size or another method (ex: exclude a certain factor so that the download doesn’t fail) that allows downloads of large files of small stocks to work ?

2

I see a failed communication relating to the data request for 2021-07-24, exactly an hour after the download preparation started. I'm not sure why the particular error I see would have come up an hour in. You're consistently seeing Download preparation failed with this one?

3

Yes. I tried it many times (approximately 7) slightly altering parameters and continuously get a download failed message

4

Followed up by email. Let me know if that helps.

5

We've made the following adjustments to better support this download preparation that was failing.

  1. SetVar and : can now be used in more places. This means one may rewrite a formula like Eval(LoopMedian("Eval(X > LoopMedian(`X`, 500), X, NA)", 500), NA) as Eval(@xmed:LoopMedian("X", 500), LoopMedian("Eval(X > @xmed, @xmed, NA)", 500), NA) to accelerate computation, cutting execution time drastically. This can be leveraged in ranking system nodes, Factor List formulas, and anywhere else that would have previously emitted an error, including custom formulas used in such contexts.

    Note that variables bleed between ranking system nodes or Factor List formulas, so be careful with your variable names to avoid picking up the wrong value. (It works just like variables in rules, but the order of execution is more implicit.)

  2. The backend service fulfilling Factor List download preparation previously timed out after an hour. This timeout has been increased to two hours.

2 Likes
6

On this note, please can you consider slightly extending the timeout for AI factor prediction loads as well?

I’ve got a couple AI factors where I get timeouts loading the predictions occasionally and have to try again a few times before it works. I’d think it might be a bit more server-efficient to allow a little bit more time so that extra loads aren’t necessary. Thanks

7

A different subsystem is handling those requests. There is active development to improve the performance and reliability of AI predictions. We're currently finalizing the work for production and hope to have it up very soon.

1 Like