Python API (< 3.0.37) deprecation & Direct Upload to S3

Flow Production Tracking
, , ,
1

Shotgun is standardizing the way we handle file uploads for all hosted sites, as we work on our architecture to enable improvements in reliability and quality of image transcoding.

One change coming soon is direct upload to S3. This feature will be progressively enabled on hosted sites over the next months. A dismissible banner will be presented to each site’s admin users with notice of the change a couple of weeks beforehand. See Deprecation of old versions of Shotgun API, RV and iOS for the deprecation schedule.

  • Python API versions 3.0.37 and above are compatible. Support for earlier versions will be dropped in 2020.
  • Thumbnail “image” fields will have an additional possible value to indicate transient state. (Exposed to both REST and Python APIs).

Read on for a complete overview of changes.

What is changing?

Previous Upload Operation

Behind the scenes, Shotgun’s original system architecture meant that any (web UI or API) file uploads resulted in files first being uploaded to the Shotgun server, and then sent to S3 for storage.

FsPNFITyrFFl4PQ8asHen3dupKFQBOAFVhlQkhMfsaqR2xSgoQp2Swj62Ty3U7PttnW5jw4F0fHe_Ch9afFJu9EqLTDxpPasVrqWyZye4_nlpHjVYtm2j_KGy3TFCnPmvhL4p-DO

New Upload Operation

In the new upload operation, supported by Python API versions 3.0.37 and newer, the client application asks Shotgun for an ‘upload link’ and then uses this link to upload content directly to S3.

Shotgun is not involved in the upload process, but is informed once it is complete.

cr92Hot_lDio6gH1Z99bt_WAgAwfCgEgzDZF7tz67IcbrguMD9wfNY-DxS7nAJvHVXVLXU0qjsk-KqBdGBRYvb2EpZ4WkjThahuGQfc6uHIDicEjzj5bXtEP6ilEpZOprehKa54f

Why this Change?

Durability

Since media does not need to hop between servers, the upload chain is greatly simplified and reduces risks of media being lost in interim steps.

Users are guaranteed (by S3 policies) that when an upload succeeds, their media will not be lost.

Performance

Since media upload requests do not transit through Shotgun servers, more bandwidth is available to process other user requests.

Security

With content going directly from its source to its target destination the attack surface is greatly reduced.

Do I need to update my code?

From an API client perspective, you should be aware that this change introduces an additional ‘state’ of values returned when querying thumbnail URLs (“image” fields).

Three possible states for values returned are:

  • null: No thumbnail image uploaded, or thumbnail generation has failed.

  • transient URL: Serves a placeholder icon. Returned when image is requested after upload, but before thumbnail has been created and made available on S3. Will be a constant string for your site, with the form: <protocol>://<domain>/images/status/transient/thumbnail_pending.png
    NOTE: Other upcoming features are likely to require the use of other transient thumbnails.
    For this reason, it is highly recommended to use the prefix part of the placeholder path (e.g. https://yoursubdomain.shotgunstudio.com/images/status/transient/) to detect any transient URLs rather than use the full path of the thumbnail.

  • signed URL for S3 object: Provides access to final processed thumbnail. Returned after transcoding is complete and media is available on S3.

This implementation provides a seamless experience for dynamic user interfaces, and in many cases, no changes will be required from API calling code.

You will need to take action in the following cases:

  • You are running a Python API version below 3.0.37 (Toolkit Core API below v0.18.149).
    Python API versions below 3.0.37 (released July 2018) will be deprecated, and support dropped in 2020. Deprecation notices will be posted to site admins. Please update to the latest version as soon as possible.

  • You have not built in retries to your file upload operations.
    As with the Shotgun-conduit architecture, failures will sporadically happen when uploading directly to S3. A retry on the spot typically works.

  • You persist image URLs, assuming they are immutable.
    This could include:

    • generating reports which include thumbnail images
    • caching results of image field queries based on entity information only

Developers can explicitly deal with this new behavior in different ways:

Let us know if you have any questions or concerns.

15 Likes
4

Hey Zoe!

One question I’m not clear on. Is this going to be a new option to upload with, or is this entirely replacing the old upload functionality? I’m not clear on if it’s required that we update our scripts or just that we could change them to use this new functionality.

1 Like
5

Hi Gary, we’re entirely replacing our architecture behind uploads, and if you utilize the Python API, it will be necessary to run a newer version in order to be compatible.
After that, your scripts themselves will only require revisiting to ensure they can deal gracefully with upload failures (something you may already accommodate) or if you are reporting or caching the values returned for image fields without first inspecting for validity (not a situation we anticipate, because of expiration time for signed URLs).

3 Likes
6

Hi Zoe, how does this change affect local-installs i.e. the Shotgun app and database and files/storage are all on-premises, with no S3? I’d imagine this is a silly question and the answer is “no change whatsoever”… but it’s nice to be sure. :slight_smile:

1 Like
7

Hi Tony, correct, “no change whatsoever” is required for local installs with no S3, but we’d still encourage people to upgrade to the latest Python API for its support of Python 3, amongst other niceties :slight_smile:

4 Likes
8

Hi,
Can you give little correlation between tkcore version and pyhton api version.

2 Likes
9

Core v0.18.149 bundles python API v3.0.37:


Thanks for the question. Main post now edited to include this info.

3 Likes
10

Hi Zoe,

Only just noticed this deprecation warning pop-up in the Shotgun web UI today.

Out of curiosity, roughly when in 2020 will the old transfer method be deprecated?

Cheers
Dan

2 Likes
11

Hi Dan, see Deprecation of old versions of Shotgun API, RV and iOS - March 24 2020 for the deprecation schedule.

2 Likes
12

Followup FAQs:

Q: How do I check which version of the Shotgun Python API I am using?
A: Example from python interpreter:

>>> import shotgun_api3
>>> shotgun_api3.__version__

Q: How do I upgrade?
A: Download and installation options are outlined here. You’ll then replace your existing API with the new download.

Q: What does this deprecation mean for Python 2.7 support?
A: Shotgun’s Python API remains Python 2.7 compatible and there is currently no timeframe to stop supporting Python 2.7. The API was updated to be Python 3 compatible in July 2019 as of v3.1.0 (release notes here). So upgrading to the latest API version will also enable a switch to Python 3.

Q: What about Deadline’s Shotgun Integration?
A: Deadline 10.1.2.2 ships with Shotgun API v 3.2; or you can manually upgrade the packaged SG API to the latest.

13

This is such a major change, I think a few code examples would be helpful.

3 Likes
14

Hi @jamie,

Welcome! What sort of examples are you looking for, specifically? If you can let us know we’ll do our best to direct you.

1 Like
15

@brandon.foster How are versions uploaded now? Are the thumbnails still automatically generated? What is returned when you submit a version for upload? How do you confirm a file has been uploaded? How do you confirm a file has failed uploading. What does resubmission look like? Does the version id of the failed upload still persist?

1 Like
16

Using the Python API as an example, uploads are handled through the Shotgun.upload() function:

https://developer.shotgunsoftware.com/python-api/reference.html#shotgun_api3.shotgun.Shotgun.upload

On success you’re returned the ID of the attachment entity that was created for your uploaded File. On failure a ShotgunError is raised. None of how this works is changing.

The major difference with this upcoming change is instead of your file first being transmitted to the Shotgun application servers, and then to S3 storage, your file will instead go straight to S3. No extra steps by end-users is required (aside from upgrading to the latest API), just continue to use the upload() function as you have. Everything will continue to upload/transcode as it has.

4 Likes
17

Hi Zoe,

Maybe my questions are too long, but just wanna make sure I clearly understand that. :bowing_man:

We’re using several Shotgun infrastructures:

  1. SG Webpage -> SG Desktop -> SG API
  2. RV -> SG Desktop -> SG API
  3. DCC Integration -> SGTK (tk-config-default) + SG API
  4. ActionMenuItem -> SG API

Our system is using Win10 x64 + Google Chrome/Firefox.

Some questions here:

  • Upgrading SG Desktop and RV will solve? As I know the Shotgun Desktop in charge of running the API command. It has a python interpreter embedded. It downloads the API and auto-update when logging in the app each time. RV has some python code in itself. So is it true that upgrading SG Desktop and RV already upgraded SG Webpage and RV to Python 3 compatible?

  • Is it required to change the DCC to Python 3 compatible before upgrading the integration? The python interpreter in DCC is in charge of executing the SG commands. So should I first activate the Python3 mode of DCC, then upgrade the SG Toolkit?

  • Upgrading system Python interpreter before upgrading ActionMenuItem? We add our studio core path to system EnvVar PYTHONPATH, and as far as I know, the system python is in charge of running the ActiveMenuItem commands. Should I upgrade the system python and also the Studio core we’re pointing to?

  • Should all the above be finished before 24th March? As in your article, the upload logic will be deprecated after 24th March. But for us as I know, all of above involve uploading logic (thumbnail upload). Does it mean that I need to finish them before 24th March otherwise some infrastructure will not work properly?

Thanks and sorry for my long questions! :bowing_man: :pray: :bowing_man: :pray:

#pipeline #shotgun #rv

1 Like
18

Hi @danielwong!

  • Yes, upgrading Desktop, RV, and the Python API will prepare you for this change.
  • The newest release of the Python API is compatible with both Python 2.7 and 3. You don’t have to upgrade your apps to Python 3 until you are ready to do so, but when you are, the Shotgun API will be ready for you.
  • There should be no need to update your interpreter, just your Shotgun API, since it is still Python 2.7 compatible.
  • Yes, this should all be completed by March 24th, otherwise uploads via the API will fail.
  • No code changes are required unless you are persisting thumbnail URLs (see the main post for how to work with the thumbnail updates).

Regards,
Brandon

1 Like
21

Hi everyone!
We just published some REST API samples showing the file upload (and download) process: https://developer.shotgunsoftware.com/rest-api/?python#code-samples

These samples also show the ‘token renew’ process in action (see ‘setup_auth_headers’ function).

We hope these make it easier to use the API!

Regards,
Stéphane

5 Likes
22

Hi Zoe!

Couldn’t that be implemented into the Shotgun API itself?
Really not looking forward to cluttering my code with retries and teaching every new coder to not use the “upload” function directly.

If it helps I could submit a Pull Request for this.

Cheers,
Fabian

2 Likes