[This post is part of a series of articles about working from home in Shotgun.]
Running from the cloud, Shotgun is well suited to distribute your pipeline to your users around the world. You can move a centralized config, which lives on-prem and is accessed directly by users from a shared filesystem location, to a distributed config, which is uploaded to your Shotgun site, and cached locally on remote user machines as needed. You can learn more about centralized and distributed configurations here.
This article assumes you currently have a pipeline configuration set up with Shotgun.
A couple of definitions:
- Pipeline Configuration: A Shotgun entity that specifies where your pipeline can be accessed from.
- Bundle: A Toolkit application, framework or engine.
- Configuration: A folder that contains your environment files and custom hooks. The environment files enumerate which bundles are available depending on the context.
Finding your configuration folder
The first step is finding where the configuration is located for a Pipeline Configuration you would like to make available to your users working from home.
Centralized Configuration
If you are using a centralized configuration, you’ll need to retrieve the location (the Mac Path, Windows Path or Linux Path field) from the Pipeline Configuration entity in Shotgun. At this location you will find the following:
In such a setup, the configuration is stored under the config folder.
Descriptor-based configuration
If you are using the Descriptor field on a Pipeline Configuration, chances are you are using the git, git_branch, dev or path descriptor. Simply browse to it’s location (or clone the repository if it’s in git), and you’ll find your configuration’s environment and hooks.
Preparing your distributed configuration folder
Whether you are using a centralized or descriptor-based configuration, we recommend you copy the files we’ve identified above to source control, as you’ll likely be making many changes to them. The root of the repository should be laid out exactly like our basic and default2 configurations.
Fixing includes
The first step is to make sure that all includes found in your environment and template file refer to files that are part of the configuration.
If an include references a file outside the configuration, as /mnt/projects/common/frameworks.yml does in the example above, you’ll need to make sure that this file is moved inside the configuration and the include is modified to point to the new location, or that this file is accessible on your user’s machine at that exact location.
Distributing custom applications, engines and frameworks
If you’re already using git (tag or branch) and/or shotgun based descriptors for your configuration’s custom Toolkit bundles, you’re ready to move to the next step. If you are using dev and path descriptors, you can move these to a git repository or upload them to Shotgun.
Another option is to move them inside the configuration. If you use the {CONFIG_FOLDER} token in the path field of the descriptor, you can then point to where the application is on disk relative to the configuration root.
Locking the core version
The third step consists of locking down the core version to use with this configuration. We recommend creating the core/core_api.yml file if it doesn’t already exist to lock the version of the Toolkit core to use with this configuration. If you are using a centralized configuration, you can find your current core version inside the pipeline configuration at install/core/info.yml.
Packaging all dependencies in a zip
For the next step, you need to zip the configuration. We recommend you use the script at tk-core/developer/bake_config.py and point it to the configuration folder. The script was integrated in the tk-core release v0.19.5.
The following command should be sufficient for our needs:
python bake_config.py /path/to/the/config /path/to/output/folder --zip --skip-bundle-types=app_store,shotgun,github_release
This will copy the configuration in the destination folder and cache every single custom Toolkit bundle that require the git executable. Finally, it will generate a zip file that you will be able to upload to Shotgun. Look for something similar to
Zip archiving the baked configuration...
Zip archive available here: /var/tmp/baked/tk-config-default2-latest.zip
at the end of the script output to retrieve the path to the generated zip file.
Bundles coming from the Toolkit AppStore, your Shotgun site, or available as Github Releases on Github.com, can be downloaded by Toolkit straight from the source as it only needs an internet connection, which is why we’ve configured --skip-bundle-types as such.
Caching git based bundles is essential if they are normally stored on a private git server on your local network, since these will not be accessible to your users working from home. Caching them even if your custom code is stored in a public repository on Github or any other git host can still be useful. By packaging them with the configuration, you’ll remove the need for your users to have access to the git executable or an ssh key pair. Note that the github_release descriptor uses the Github REST API so it does not require the git executable.
We’d like to give a shoutout to Stéphane Déverly who contributed this script.
Uploading everything to Shotgun
Finally, you can upload this configuration to Shotgun and assign it to the right projects. In the Shotgun web app, browse to a
Pipeline Configuration entity page, and hit the Add Pipeline Configuration button. You need to:
- Make sure you can see the
Uploaded ConfigsandPlugin Idssettings. You can add them via the gear icon at the top right of the pop-up. - Upload the zip file you created using the
Choose Filebutton on theUploadtab. - Set the
Plugin Idsfield is set tobasic.*.
Note that during development, you can use the User Restriction field to prevent other users from using your work in progress by mistake.
Accessing from Shotgun Desktop
Now that you’ve uploaded your config to Shotgun, when users launch Shotgun Desktop, they can pick this pipeline configuration from the project’s dropdown menu if multiple configurations are available. If only one is available, then Shotgun Desktop will default to it.