Recommended way to share/update pipeline configurations between projects?

[Ricardo_Musch]

Hi again!

Our studio does a lot of small projects that currently all get a pipeline set up individually (copied from the latest developped project at that moment).

Is there a better way to maybe share most of the config but still allow for changes (like nuke writenode customisation)?

Maybe in combination with github?

I’m doing a lot of work on hooks at the moment and would really appreciate it if I didn’t have to manually keep track of what hooks on what project are the latest.

We don’t need to create different workflows for any project, I would like a master project and all projects to derrive their functionality from there (but keep things like colorspaces etc configurable)

[Ross_Macaluso]

Hey Ricardo,
I can explain what I do to solve this exact issue. I work at a small studio and we also get a lot of small projects, sometimes we have projects where only 1 or 2 people work on it, but I like to still have them access our full tool pipeline so what I have done is the following.

All our projects are not localized, they are instead sharing a core, the core stays on a consistent version for 2-3months on average then all of those projects also share a configuration that is attached to a central repository. I have 2 of these configurations running in parallel currently, one for the old style shotgun config, and one for shotgun config-default2, this is so I can have a Primary and secondary config setup on each project encase certain futures are not working as intended under config-default2 and more work needs to be done, it’s sort of a WIP config that is stable enough to test, but isn’t guaranteed to be fully functional.

We do, in fact, use Github, but there are certainly other options

Then within any given project I still have the option to create a branch on either config repository for any small deviations a particular show requires. Those deviations then can be merged back into the master branch or can live as a deviation branch for the life of the project.

If I need to take over any of shotgun’s apps/engines/frameworks I simply fork shotguns repo, then create a custom repository to hold my deviation of that shotgun fork. From there I can hook my repo up to my config and easily distribute it to all shows.

I have a pipeline project where I do all my sandbox testing, it is linked to both the stable active config, and a parallel copy “sandbox” config that has branch deviations of it’s own waiting to either be merged, or worked on further.

This is the basic outline of how I maintain multiple projects while keeping a streamlined and simple to understand config setup.

Let me know if you want more info on any of the outline steps.
-Ross

[Ricardo_Musch]

Thanks for the very detailed explanation @Ross_Macaluso!

What software do you use to track branches on github since I have found it to be very confusing to manage a branch in Github Desktop.
It doesn’t seem to be able to save a branch somewhere else? (or at least, not in a very elegant way).

and I assume you link these manually to the folder of the branch on disk?

[Ross_Macaluso]

I use powershell for initial remote setup, and initial new repo initalization…

But I use ATOM code editor, there are numerous packages for github specifically, ATOM is a totally modular code editor that allows the install of user created packages. However, Github has also authored a few core Github specific packages that handle commits, branch management, merge conflict resolution and numerous other features that really help visualize your branch/master relationship.
I would take some time and look at ATOM if you aren’t familiar with it already. I have friends that swear by Sublime, but with the integrated Github package support… I really have fallen in love with ATOM haha…

A branch will never have a physical local folder location, it will instead change all the altered files to the current file state held on that branch. Think of a branch just storing the differences, and applying those differences when that branch is checked out (made the active state) That way you can have one projects app repo on a branch, and another project still on the master, both will have the same unedited files, but any files that have deviated will ONLY be on the project where you have checked out that new branch.

I would take some time and look through Github training materials, they have quite an extensive database of how repositories work locally, and virtually. If you do go with ATOM, there is also extensive documents about the ATOM Github packages.

Hope that makes sense,
-Ross

[Ricardo_Musch]

Thank you!

You have given me some ideas!

[philip.scadding]

I thought I’d chime in with a suggestion as well.

Option 1

Another approach would be to have a site config (a PipelineConfiguration entity with no project set), using a distributed setup. This will mean that all projects that don’t have their own config will use the site config (you will still need to set a tank_name value on each Project entity).Then to take over for a specific project, you can just create a PipelineConfiguration for that project. That will be a full config take over though.

Option 2

If you just want to handle taking over specific hooks or environment yaml files, then you could maybe make use of environment variables. Here is an example:

Here is a config where I just mocked up a test: garden_show.zip (286.4 KB)

In this config, if you were in a project called garden show then it would not show the software launchers in Shotgun Desktop, but for all other projects it would.

In the config, I used environment variables to define the includes path.

I also added code to the pick_environment.py core hook, that contained the following code to set the environment variable based upon the project the configuration was linked to. If a project-specific settings folder was found (named after the project) , then it would define the PROJECT_SETTINGS env var to that. If one isn’t found it would define the env var to settings (which is the default folder.)

        if context.project is None:
            os.environ["PROJECT_SETTINGS"] = "settings"
            return

        # set an environment variable that will later be used to resolve the include path to the project
        # specific settings inside the generic config.
        settings_folder_name = "{0}_settings".format(context.project["name"])
        # remove any spaces from the name
        settings_folder_name = settings_folder_name.replace(" ", "_").lower()

        # Check project settings exist
        config_path = self.sgtk.configuration_descriptor.get_path()
        settings_folder_path = os.path.join(config_path, "env", "includes", settings_folder_name)
        self.logger.debug("checking for project specific settings here: %s" % settings_folder_path)
        if not os.path.exists(settings_folder_path):
            # no project specific settings found use the default location
            settings_folder_name = "settings"

        self.logger.debug("Using settings folder name: %s" % settings_folder_name)

        os.environ["PROJECT_SETTINGS"] = settings_folder_name

I also took over the bootstrap.py core hook and added os.environ["PROJECT_SETTINGS"] = 'settings' so that if the config is used as a site config then it will take the settings block.

This is just an example though, there are probably a number of different ways to set up something similar.

[Ricardo_Musch]

omg, just realized this is what I asked for in the other topic!

[Nik]

Hi Philip,

New to shotgun so still getting up to speed on the lingo. I, too, would like to switch to a distributed setup. Currently we have a single project that we use as a template for all other projects. So every time we make an update to that template, we have to manually copy those files/changes over to every project. Not so cute anymore. So I’ve set up a pipeline configuration without a project in shotgun, along with a couple of test projects to see how the distributed setup works.

Questions:

1. Now that I have this global pipeline configuration, do I still need the project I’ve been using as a template?

2. You mentioned I need to set a tank_name on each Project entity. In one of your examples, you named it ‘projects/some-name’. Currently all of my projects are named ‘some-name’. What’s the significance of adding the ‘projects/’ prefix? Is that relevant to it picking up the global configuration?

3. In order to get the existing projects to use the global configuration, do I need to delete each configuration from each project? Right now they each have a single configuration called ‘Primary’. Delete it?

   3a) Is there anything else I need to do to the existing projects for them to pick up the global config?

4. Uploaded Config vs. Descriptor: On the global config entity, if I opt to upload the config as a zip file, do I still need a descriptor? If so, why?

Thanks in advance!

[philip.scadding]

Hi Nik, welcome to the forums. Great questions!

1. It’s up to you really. Certainly, you won’t need to copy the files to each project, you will just need to upload a copy of the config to the site PipelineConfiguration entity. However, it may be beneficial to have a test project, with a config, so you can test changes outside of production.
   Personally I would be using git/GitHub where I would keep my source config and track the changes, then zip up the config from the repo and upload that.

2. Yeah, its a good point, perhaps that example is a bit distracting. You don’t need a projects/ bit in front of the name. You can optionally define a nested structure in the name, by separating it with /s. Doing so allows you to add additional folders after the root but before the main project folder.
   For example, you might wish to split your project folders out by client, or type of project: vfx_projects/my_vfx_project and game_projects/my_game_project.

3. Yes, that is correct, you will need to delete the PipelineConfiguration entity from your Shotgun site for that project. Maybe hold off on deleting the config on disk as well at first, so that you can check it all goes smoothly, otherwise, you can always restore the PipelineConfiguration back from the trash, and it should go back to how it was.
   3a. No, the projects should already have a tank_name if they were previously set up, clearing out the PipelineConfiguration entity should be all you need to do (assuming the site config is exactly the same in terms of schema). Though I recommend testing against the process with a dummy project just to make sure you feel comfortable with it.

4. If you upload the config, you should not need to set the descriptor. If you do it will take precedence over your uploaded config.

I hope that helps? You’ve probably seen these already, but I just thought I’d link them here as well:

Shared project configs

By default when you run the Toolkit advanced set up wizard for a project via Shotgun Desktop, you end up with a configuration created that is specific to that project. However, in certain setups you may wish to share your configuration with all your projects by default. To do this you will need to setup a site-level distributed config. Find the configuration you want to use. You can use the latest tk-config-default2 config (which is the default when using the setup wizard) or you can use a c…

Distributed config management workflow

I’ve seen the question come up a few times around how to test configuration changes and push them to production when using a distributed config setup. My personal recommendation would be to do the following: Assuming your project already has a “primary” PipelineConfiguration entity with an uploaded config, I would create a second one called dev. Set a dev descriptor to define a location to your config on disk. (If you haven’t got a config on disk you can download the primary config and unzi…

[Nik]

Nice one. Thanks for clarifying. Got a couple more questions for ya:

1. Now that my config is global and lives is in shotgun, my projects will need to download apps from shotgun’s remote app store correct? Where does it download it to?

2. I’d like to set up a custom app store as well to be distributed via shotgun. Could you tell me more about how that works? For example, I have a package “maya_pkg” that lives on disk on a path where all the maya packages/scripts/etc live, “path/to/maya/scripts”. Would it be possible to distribute maya_pkg via shotgun so that remote users could access it who may not have easy access to “path/to/maya/scripts”? If so, how does that work? Does the user download it from shotgun and access it the same way it accesses the built-in shotgun apps?

[philip.scadding]

1. By default it will download into the bundle_cache folder inside you cache folder. The whole cache location can be changed via the SHOTGUN_HOME env var. The location should be unique to the user and not on network storage (for performance reasons). You can separate out the bundle cache from the rest of the cache using SHOTGUN_BUNDLE_CACHE_PATH and that could be stored somewhere centrally so it can be shared.

2. Unfortunately I don’t have the time right now to write out a complete guide on how to do this, but I think the same thing was discussed here: DCC plugin distribution via Toolkit which contains a few details. Let us know if you still have questions though and I’ll or someone else will try and get back to you when we can.

[brhoades]

Hi @philip.scadding,

I’m trying to implement a variant of your proposal, but utilizing environment variables at the templates.yml level.

I’m struggling because adding the environment variable to pick_environment.py and bootstrap.py work when a descriptor is pointing at a local config location, and it works the first time that the configuration is downloaded if it’s been uploaded to a distributed PipelineConfig entity. However, once it’s been downloaded, it seems like neither bootstrap.py nor pick_environment.py is called before it tries to access the templates.

Do you know of any workarounds? Why isn’t bootstrap.py called after the first time? Is there another core hook that’s called earlier that can be used instead?

[Ricardo_Musch]

Maybe this helps,

I have various template paths in project specific templates.yml files (example proj1_templates.yml, proj2_templates.yml)

The templates.yml main file has an include that references an _templates.yml

So if you set the env file in pipeline_configuration_init core hook then the templates.yml include will redirect to the file you want.

[brhoades]

Hi @Ricardo_Musch ,

Thanks, pipeline_configuration_init was precisely the core hook I needed.

[nico.vandenbosch]

I’m attempting a slight variation on Option 2 here, using an environment variable to specify the external location of app settings yml file. Only, I’m adding one more level of indirection by leaving the original settings.yml file with an include to the external settings file. This approach minimizes the number of places we’d need to edit where the standard location of the settings yml file is still referenced. The setup is like so:

./includes/settings/tk-3dsmax.yml

includes:
- ./includes/settings/tk-multi-loader2.yml
settings.tk-3dsmax.shot_step:
  apps:
    tk-multi-loader2: "@settings.tk-multi-loader2.3dsmax"

./includes/settings/tk-multi-loader2.yml

includes:
- ../app_locations.yml
- $TK_MULTI_LOADER2_SETTINGS_PATH
# the environment variable resolves to an external path like /proj/foo/settings/tk-multi-loader2.yml

/proj/foo/settings/tk-multi-loader2.yml

settings.tk-multi-loader2.3dsmax:
  action_mappings:
    3dsmax Scene: [import, reference]

The only problem is, toolkit chokes on this and is unable to apply the settings. Turns out core’s env parsing does not properly allow for this many levels of recursion on includes. Specifically if a @setting is asked for in file A (tk-3dsmax.yml), then the recursion algorithm expects that setting to be immediately defined in a directly included file B (tk-multi-loader2.yml). If the setting is defined one level down in an indirectly included file C (/proj/…/tk-multi-loader2.yml) it gets filtered out by the include recursion because it’s not immediately referenced. I’m assuming this is not by design, but arguably a bug that may not have gotten much daylight?

[FilSus]

Hey, a bit late to the debate,

I am trying to execute the same setup of pointing the toolkit to project specific templates.yml files.

I have added the pipeline_configuration_init hook to setup the environment varaible, but have no clue how to make it project specific since I cannot import sgtk. Probably too early in the process.

os.environ[“PROJECT_TEMPLATES”] = “{project_name}_templates.yml”

Also even if I set it up manually by providing it with the exact file like project1_templates.yml
it does not get registered.

I have simply added this code to the original

core/templates.yml

includes:
– ./templates/$PROJECT_TEMPLATES

But after launching Nuke it is still picking up the template.yml instead of the project1_templates.yml

---

UPDATE

---

So the Toolkit actually picked up the project1_templates.yml but the paths got overwriten by the default templates.yml, looks like I need to leave this file empty so that it would not overwrite the included paths.

[FilSus]

OK, this post is the solution to my problems.

Template.yml per-project includes

Hey Patrick - I added this line to templates.yml and it worked: include: "${INCLUDE_PATH}/test.yml" so we definitely do support env vars in template includes. As for when to set the env var, pipeline_configuration_init is the right place, and it looks like you can get to the project name via self.parent. Here’s where the project is getting set and the pipeline_configuration_init core hook is getting executed (conveniently adjacent to one another ). You can see that the core h…