Shotgrid Upload Issues

Starting late yesterday we started having issues with uploading files. We’re getting connection time outs. Smaller file sizes seem to be making it but any of the bigger file sizes are failing. I’m not seeing any service problems listed on the status site so reaching out here to see if others are experiencing it or is just me.

How big are the files?
Have you tried looking at the Site activity Monitor?

Hi Ricardo, all.

Adding some more info to this (I work with Don).

EDIT: SOME of the files that are failing are reasonably small MP4/H.264s for individual shots, so nothing out of the ordinary. It doesn’t seem to be directly related to large uploads.

The uploads are being pushed through the API in an SG Desktop app.

The failures seem to happen randomly, but it seems most of the time they go through.

Here’s a sample traceback:

2023-12-13 10:16:56,232 [83076 ERROR sgtk.ext.tkimpbda273b8eea140f3946c1d4d16daed98.tk_desktop.site_communication] [PROXY] Traceback (most recent call last):
  File "/Applications/", line 1346, in do_open
    h.request(req.get_method(), req.selector,, headers,
  File "/Applications/", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Applications/", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Applications/", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Applications/", line 1040, in _send_output
  File "/Applications/", line 980, in send
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 4216, in connect
  File "/Applications/", line 946, in connect
    self.sock = self._create_connection(
  File "/Applications/", line 844, in create_connection
    raise err
  File "/Applications/", line 832, in create_connection
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/", line 324, in _publish_updates
    is_shots_updated =
  File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/", line 179, in run
  File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/", line 665, in _process_shot_versions
    self._create_version(sg_shot, edit_in, edit_out)
  File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/", line 722, in _create_version"Version", version["id"], movie_tempfile, field_name="sg_uploaded_movie")
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 2431, in upload
    return self._upload_to_storage(entity_type, entity_id, path, field_name, display_name,
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 2468, in _upload_to_storage
    self._upload_file_to_storage(path, upload_info["upload_url"])
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 3979, in _upload_file_to_storage
    self._upload_data_to_storage(fd, content_type, file_size, storage_url)
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 4077, in _upload_data_to_storage
    result = self._make_upload_request(request, opener)
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 3597, in _make_upload_request
    result =
  File "/Applications/", line 517, in open
    response = self._open(req, data)
  File "/Applications/", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Applications/", line 494, in _call_chain
    result = func(*args)
  File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/", line 4235, in https_open
    return self.do_open(self.create_https_connection, req)
  File "/Applications/", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 60] Operation timed out>

Is there maybe a firewall / blocking in play?
Something that could interfere with the network connection?

Usually in studios there is internet filtering on the firewall to block various services.
If thats the case the systems admin will need to check that they have followed the Ecosystem document:

If the above is not at play then it might be worth to test your site’s connectivity:

Ricardo, I’m also the firewall guy as well and that isn’t happening. An engineer is looking into it.

1 Like

For anyone running into this in future, if it is your own SG code that’s uploading to SG (or a hook you have control over), you can help prevent this from crashing your app by doing something like this:

def upload_with_retry_on_error(self, version_id, movie_path, max_attempts=5, wait_time_on_fail=3):"Uploading %s to Version %d" % (movie_path, version_id))
    attempt_n = 0
    while attempt_n < max_attempts:
        attempt_n += 1
  "Version", version_id, movie_path, field_name="sg_uploaded_movie")
        except Exception as e:
            if attempt_n >= max_attempts:
                "Upload failed (attempt #%d). Waiting %d seconds to try again: %s"
                % (attempt_n, wait_time_on_fail, str(e))

It’s also noted in the API documentation that this is a good idea :slight_smile:

“Make sure to have retries for file uploads. Failures when uploading will occasionally happen. When it does, immediately retrying to upload usually works”

… Today “occasionally” was more like “pretty frequently” for us. But replacing the upload with the code above has gotten us through I think.


1 Like

Good to check what version of the Api your running because as far as I know the api should now have retries built in. (recently added)

But not sure if that was also added for uploads specifically.

1 Like