Starting late yesterday we started having issues with uploading files. We’re getting connection time outs. Smaller file sizes seem to be making it but any of the bigger file sizes are failing. I’m not seeing any service problems listed on the status site so reaching out here to see if others are experiencing it or is just me.
How big are the files?
Have you tried looking at the Site activity Monitor?
Hi Ricardo, all.
Adding some more info to this (I work with Don).
EDIT: SOME of the files that are failing are reasonably small MP4/H.264s for individual shots, so nothing out of the ordinary. It doesn’t seem to be directly related to large uploads.
The uploads are being pushed through the API in an SG Desktop app.
The failures seem to happen randomly, but it seems most of the time they go through.
Here’s a sample traceback:
2023-12-13 10:16:56,232 [83076 ERROR sgtk.ext.tkimpbda273b8eea140f3946c1d4d16daed98.tk_desktop.site_communication] [PROXY] Traceback (most recent call last):
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/urllib/request.py", line 1346, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 1040, in _send_output
self.send(msg)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 980, in send
self.connect()
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 4216, in connect
http_client.HTTPConnection.connect(self)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/http/client.py", line 946, in connect
self.sock = self._create_connection(
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/socket.py", line 844, in create_connection
raise err
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/socket.py", line 832, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/dialog.py", line 324, in _publish_updates
is_shots_updated = shot_updater.run(True)
File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/shot_updater.py", line 179, in run
self._process_shot_versions()
File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/shot_updater.py", line 665, in _process_shot_versions
self._create_version(sg_shot, edit_in, edit_out)
File "/Volumes/SHOTGRID/_pipeline/toolkit/apps/tk-multi-edupdater/v0.2.9/python/app/shot_updater.py", line 722, in _create_version
self.sg.upload("Version", version["id"], movie_tempfile, field_name="sg_uploaded_movie")
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 2431, in upload
return self._upload_to_storage(entity_type, entity_id, path, field_name, display_name,
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 2468, in _upload_to_storage
self._upload_file_to_storage(path, upload_info["upload_url"])
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 3979, in _upload_file_to_storage
self._upload_data_to_storage(fd, content_type, file_size, storage_url)
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 4077, in _upload_data_to_storage
result = self._make_upload_request(request, opener)
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 3597, in _make_upload_request
result = opener.open(request)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/urllib/request.py", line 517, in open
response = self._open(req, data)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/urllib/request.py", line 534, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/Users/stopmo/Library/Caches/Shotgun/shadowmachine/p187c470.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 4235, in https_open
return self.do_open(self.create_https_connection, req)
File "/Applications/Shotgun.app/Contents/Resources/Python3/lib/python3.9/urllib/request.py", line 1349, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 60] Operation timed out>
Is there maybe a firewall / blocking in play?
Something that could interfere with the network connection?
Usually in studios there is internet filtering on the firewall to block various services.
If thats the case the systems admin will need to check that they have followed the Ecosystem document:
If the above is not at play then it might be worth to test your site’s connectivity:
https://help.autodesk.com/view/SGSUB/ENU/?guid=SG_Administrator_ar_troubleshooting_sites_ar_connection_test_html
Ricardo, I’m also the firewall guy as well and that isn’t happening. An engineer is looking into it.
1 Like
For anyone running into this in future, if it is your own SG code that’s uploading to SG (or a hook you have control over), you can help prevent this from crashing your app by doing something like this:
def upload_with_retry_on_error(self, version_id, movie_path, max_attempts=5, wait_time_on_fail=3):
self.logger.info("Uploading %s to Version %d" % (movie_path, version_id))
attempt_n = 0
while attempt_n < max_attempts:
attempt_n += 1
try:
self.sg.upload("Version", version_id, movie_path, field_name="sg_uploaded_movie")
break
except Exception as e:
if attempt_n >= max_attempts:
raise
self.logger.error(
"Upload failed (attempt #%d). Waiting %d seconds to try again: %s"
% (attempt_n, wait_time_on_fail, str(e))
)
time.sleep(wait_time_on_fail)
It’s also noted in the API documentation that this is a good idea 
“Make sure to have retries for file uploads. Failures when uploading will occasionally happen. When it does, immediately retrying to upload usually works”
… Today “occasionally” was more like “pretty frequently” for us. But replacing the upload with the code above has gotten us through I think.
Cheers,
-Dashiel
Good to check what version of the Api your running because as far as I know the api should now have retries built in. (recently added)
But not sure if that was also added for uploads specifically.
1 Like