Hi everyone,
We have been facing the following error a few times in the last week in one our tool based on the SG Python API: “developerMessage”:“Quota limit exceeded”
Since this morning, this error is occurring a lot, to the point that it is preventing a part of our team to use our tool.
We recently our SG server to the new Autodesk user system. Could this be the reason for this kind of error? From my current knowledge, we never experienced this error before the migration.
Hi @Francois
I assume that this is occurring at login time ?
Is this in a script that runs without user supervision or is actively interacting with a human being ?
There IS a rate limit for calls to the authorization endpoint. And no way for you to see what your current usage is.
Your code should have a hold off period of a few seconds when this occurs, as calling again immediately only makes the problem worse.
-Patrick
Hi Patrick,
We are getting exactly the same issue, and has only started happening this morning. We use the python api.
Our script allows users to log in via gui to use shotgrid-connected tools. It’s not something that is firing off multiple requests a second. We also only have a handful of users who use the tools.
If we are naturally hitting the rate limit based on the above usage, then we might have a problem.
Additionally, one user reported that an upload failed partway with the following error:
<urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>
Has something changed on the back-end within the last 24 hours?
-Mike
Hi @patrick-hubert-adsk, thank you for your answer.
To give you more details, this error was flagged for the first time on our SG server 10 days ago.
I assume that this is occurring at login time ?
No, this is occurring when making Update/Create requests through the API.
Is this in a script that runs without user supervision or is actively interacting with a human being ?
The error happens in a custom app that runs trough SG Desktop. SG users are logged in this app thanks to their Shotgrid credentials, so API request are done as the currently logged user.
As the migration of our SG instance is already done, we are currently working on switching the authentication method for this app to rely on the Autodesk credentials instead. Could the error be linked to the fact that we are still using SG credentials?
There IS a rate limit for calls to the authorization endpoint. And no way for you to see what your current usage is.
Could you provide us with more details about what you call the authorization endpoint and this limit?
It would help us a lot to know more about how this limit work and what it is affecting exactly. A hold period of a few seconds would probably work as a workaround, but I’m sure we could come up with a better solution would we have a better knowledge about this limit.
If we are naturally hitting the rate limit based on the above usage, then we might have a problem
Same conclusion here. We did change the way our app is making requests. This is the same usage we have had in the last past years.
Hi @Francois
No. Under the hood, we use the Personal Access Token of the user to make a request to Autodesk Identity. So using either set of credentials may (and I say may because I am not sure exactly yet what is happening) result in hitting the same limits.
Every call to ShotGrid triggers a series of checks. And depending on the results, we will call Identity to get information about the user, obtain a new Autodesk Identity token, etc.)
Everyone one of these types of call has a rate limit.
The rate is global to all ShotGrid, and is per minutes.
The current situation is due to a bug on our side, which makes undesired additional calls and allows one script to over-consume the allowed budget. Thus impacting all of ShotGrid users.
This is obviously a situation that should not happen : one piece of rogue code somewhere should not impact all of ShotGrid users for all sites.
All of this must be implemented by us in a way that does not require clients to make changes to their code or take extra precautions. This should be done in a way transparent to API applications.
That being said, on the client side (meaning here your code), it is always a good network programming practice to throttle retries after an error condition (like this Quota limit exceeded). Using an initial random delay, and increasing the delay between re-attempt, to avoid all of your processes hitting the server again at the same time.
Immediately re-attempting the call is not a good practice.
But the TL:DR; version is : this error should not have occurred, and it is on us to fix. But there is a way for clients to not make the problem worse when it happens again (trying to be realistic, not pessimistic).
-Patrick
You are right, we did not implement any limitation regarding how often users can make attempts. We are going to take this into account in the future.
Thank you very much for the detailed answers and thanks to your team for the quick reaction time on fixing this issue! We really do appreciate.