Distributed Config and Render Farm Setup

Pipeline Integrations
3

Hi

Welcome to the forums and thanks for posting here!

So I’m actually working on a doc/guide on this very thing right now, though unfortunately, it’s not yet in a place where I can properly share it with you. However, I do have an earlier unreleased draft of a different doc I was working on which I can share.

I just though I’d point out that the tk-core (sgtk API) comes bundled with a copy of the Shotgun python API, and you can access it through the sgtk API. For example you can get an authenticated instance of Shotgun via the engine:

import sgtk

# get the engine we are currently running in
currentEngine = sgtk.platform.current_engine()
# get hold of the shotgun api instance used by the engine, (or we could have created a new one)
sg = currentEngine.shotgun

Here is an extract taken from one of my much earlier drafts on bootstrapping:

Distributed configs are handled a bit differently from centralized configs as you don’t necessarily have a config stored on disk for your project yet. The approach here is to use a standalone copy of the sgtk API to bootstrap into an engine.

The bootstrap process will take care of ensuring everything is cached locally and swap out the previously imported sgtk package for the one belonging to the project you’re bootstrapping.

(Note for this to work you need to have download a standalone copy of the sgtk API.)

The bootstrap process will start an engine, and you will want to pick the engine appropriately for the environment your running this script in. If you’re running the script in a python interpreter outside of some software like Maya or Nuke then the tk-shell engine serves this purpose nicely. Here is an example of how to do that:

(Note this also works for centralized configs.)

import sys
# import a standalone sgtk API instance, you don't need to insert the path if you pip installed the API
sys.path.insert(0,"/path/to/a/sgtk/api")

import sgtk

sa = sgtk.authentication.ShotgunAuthenticator()

# get pre cached user credentials
# user = sa.get_user()

# or authenticate using script credentials
user = sa.create_script_user(api_script="MYSCRIPTNAME",
                             api_key="MYSCRIPTKEY",
                             host="https://mysite.shotgunstudio.com")

sgtk.set_authenticated_user(user)

project = {"type": "Project", "id": 176}

mgr = sgtk.bootstrap.ToolkitManager(sg_user=user)
mgr.plugin_id = "basic."
# you don't need to specify a config, the bootstrap process will automatically pick one if you don't
mgr.pipeline_configuration = "dev"

engine = mgr.bootstrap_engine("tk-shell", entity=project)

# As we imported sgtk prior to bootstrapping we should import it again now as the bootstrap process swapped the standalone sgtk out for the project’s sgtk.
import sgtk

print ("engine",engine)
print ("context", engine.context)
print ("sgtk instance", engine.sgtk)
print ("Shotgun API instance", engine.shotgun)

If this is running on the farm I would recommend you use script credentials rather than user credentials, as obviously there won’t be a user present to log in.

I would recommend against bootstrapping in each render frame/task, as it will, (A) slow each render task down, and (B) potentially DDoS your site, if you have many workers running this simultaneously.

Instead, it’s better to have a pre or post job that can perform any Toolkit processing so that it is limited to once per farm job.

Let me know if you have any further questions!

Thanks
Phil

4 Likes