Hi all,
This is a strange situation I am experiencing. I made an AMI which can download a Version’s media from Shotgun’s Amazon S3 storage. The AMI seems to download media without issues at our studio’s building. However, if I work from home via a VPN then I encounter the below error randomly when attempting to download media:
https://sg-media-usor-01.s3-accelerate.amazonaws.com/…(rest of URL has sensitive info)
HTTP Error 403: Forbidden
The error seems to increase as the selected number versions increases. For example, I had no issue downloading 4 iterations of 25 versions (total of 100 unique versions) simultaneously via the AMI but a single selection of the same 100 Versions had some failures.
I could probably get around this issue by enhancing my tool to retry N times if the exception is detected. Just wondering if the issue is experienced by others.
PS - Currently I can’t perform tests without being connected to a VPN. Just in-case anyone wonders if the issue happens without VPN.
Thanks, Adam Chrystie
1 Like
hi @achrystie! It’s been a while, hope you’re doing well. I’m pretty sure those s3 urls are temporary and expire after a 5 min window or so for security reasons. If you’re at all able to download via shogun’s API instead I think that would probably resolve it. Otherwise, you’ll likely need to re-query shotgun for a fresh version url which would force you back into the API regardless.
Hope this helps,
Cheers
3 Likes
Hi @nico.vandenbosch. We’re doing ok. I hope you and the family are well. I thought about your suggestion regarding the s3 urls expiring. I was gathering the info for all selected versions in one call via the APIs find method and iterating over the versions. However my internet connection is very slow temporarily and my selection list is very large so this could explain the s3 errors; basically the url expired by the time it was the version’s turn to be processed. So I changed my code to iterate over the selected IDs and obtain the version info one at a time. Now I encounter no issues when downloading the same 100 versions even when connected by VPN plus slow download speeds of 2.4 Mbps. Previously this was the most problematic scenario. I stress tested further and had no issues downloading the media belonging to 600 versions.
I just noticed method get_attachment_download_url() in the API which might have allowed my original code to work. Seems like the method might be able to construct an updated URL if the dictionary passed to the method lacks a URL key.
Thanks for helping me solve this issue! Looking forward to catching up in person once the pandemic subsides…
Cheers, Adam
3 Likes
Just leaving this for my future self and anyone else with this issue. You can pass in the attachment id to the attachment argument and it will look up the URL for the attachment on the fly and avoid this issue.
1 Like