Do the AI servers go to sleep during US nighttime?

AI Factor Discussions
1

I don't know if this is a coincidence, but this is now the third day in a row that I’ve started running validations in the European mornings, and it doesn’t work. I’ve tried running a simple linear training model that would normally take less than a minute to complete, or LightGBM models that usually take 10-15 minutes, but after 45 minutes, I have to cancel because they make very little progress (finishing only 2 out of 4 splits). When I restart in the afternoon, it works as expected.

Do the AI servers go to sleep during US nighttime? Is it only me or does others have problems european mornings too?

2

We're investigating . The kernel of the server is getting overloaded for some reason

Until we figure it our we will not charge you for tasks that you cancel. Only completed tasks will be billed.

We'll also examine the logs to and give credit for canceled tasks.

Thanks for reporting this

1 Like
3

We may have found the culprit. Linear models have a different way of controlling parallelism which we did not account for. We're going to disable linear models until we patch the backend since they affect everything else that runs on the same server. I'll post something specific shortly.

1 Like