# Recommended way to share/update pipeline configurations between projects? **URL:** \ **Category:** Pipeline Integrations\ **Created:** [January 20, 2020, 5:56pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609 "2020-01-20T17:56:16Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Ricardo\_Musch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ricardo_musch/32/405_2.png) [@Ricardo\_Musch](https://community.shotgridsoftware.com/u/Ricardo_Musch)\ **Post date:** [January 20, 2020, 5:56pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/1 "2020-01-20T17:56:16Z")
Hi again! Our studio does a lot of small projects that currently all get a pipeline set up individually (copied from the latest developped project at that moment). Is there a better way to maybe share most of the config but still allow for changes (like nuke writenode customisation)? Maybe in combination with github? I’m doing a lot of work on hooks at the moment and would really appreciate it if I didn’t have to manually keep track of what hooks on what project are the latest. We don’t need to create different workflows for any project, I would like a master project and all projects to derrive their functionality from there (but keep things like colorspaces etc configurable) ---
**Author:** ![Ross\_Macaluso](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ross_macaluso/32/873_2.png) [@Ross\_Macaluso](https://community.shotgridsoftware.com/u/Ross_Macaluso)\ **Post date:** [January 20, 2020, 6:32pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/2 "2020-01-20T18:32:21Z")
Hey Ricardo, I can explain what I do to solve this exact issue. I work at a small studio and we also get a lot of small projects, sometimes we have projects where only 1 or 2 people work on it, but I like to still have them access our full tool pipeline so what I have done is the following. All our projects are not localized, they are instead sharing a core, the core stays on a consistent version for 2-3months on average then all of those projects also share a configuration that is attached to a central repository. I have 2 of these configurations running in parallel currently, one for the old style shotgun config, and one for shotgun config-default2, this is so I can have a Primary and secondary config setup on each project encase certain futures are not working as intended under config-default2 and more work needs to be done, it’s sort of a WIP config that is stable enough to test, but isn’t guaranteed to be fully functional. We do, in fact, use Github, but there are certainly other options Then within any given project I still have the option to create a branch on either config repository for any small deviations a particular show requires. Those deviations then can be merged back into the master branch or can live as a deviation branch for the life of the project. If I need to take over any of shotgun’s apps/engines/frameworks I simply fork shotguns repo, then create a custom repository to hold my deviation of that shotgun fork. From there I can hook my repo up to my config and easily distribute it to all shows. I have a pipeline project where I do all my sandbox testing, it is linked to both the stable active config, and a parallel copy “sandbox” config that has branch deviations of it’s own waiting to either be merged, or worked on further. This is the basic outline of how I maintain multiple projects while keeping a streamlined and simple to understand config setup. Let me know if you want more info on any of the outline steps. -Ross ---
**Author:** ![Ricardo\_Musch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ricardo_musch/32/405_2.png) [@Ricardo\_Musch](https://community.shotgridsoftware.com/u/Ricardo_Musch)\ **Post date:** [January 20, 2020, 6:37pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/3 "2020-01-20T18:37:23Z")
Thanks for the very detailed explanation @Ross_Macaluso! 🙂 What software do you use to track branches on github since I have found it to be very confusing to manage a branch in Github Desktop. It doesn’t seem to be able to save a branch somewhere else? (or at least, not in a very elegant way). and I assume you link these manually to the folder of the branch on disk? ---
**Author:** ![Ross\_Macaluso](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ross_macaluso/32/873_2.png) [@Ross\_Macaluso](https://community.shotgridsoftware.com/u/Ross_Macaluso)\ **Post date:** [January 20, 2020, 6:43pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/4 "2020-01-20T18:43:16Z")
I use powershell for initial remote setup, and initial new repo initalization… But I use ATOM code editor, there are numerous packages for github specifically, ATOM is a totally modular code editor that allows the install of user created packages. However, Github has also authored a few core Github specific packages that handle commits, branch management, merge conflict resolution and numerous other features that really help visualize your branch/master relationship. I would take some time and look at ATOM if you aren’t familiar with it already. I have friends that swear by Sublime, but with the integrated Github package support… I really have fallen in love with ATOM haha… A branch will never have a physical local folder location, it will instead change all the altered files to the current file state held on that branch. Think of a branch just storing the differences, and applying those differences when that branch is checked out (made the active state) That way you can have one projects app repo on a branch, and another project still on the master, both will have the same unedited files, but any files that have deviated will ONLY be on the project where you have checked out that new branch. I would take some time and look through Github training materials, they have quite an extensive database of how repositories work locally, and virtually. If you do go with ATOM, there is also extensive documents about the ATOM Github packages. Hope that makes sense, -Ross ---
**Author:** ![Ricardo\_Musch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ricardo_musch/32/405_2.png) [@Ricardo\_Musch](https://community.shotgridsoftware.com/u/Ricardo_Musch)\ **Post date:** [January 20, 2020, 6:49pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/5 "2020-01-20T18:49:43Z")
Thank you! You have given me some ideas! 😉 ---
**Author:** ![philip.scadding](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/philip.scadding/32/30_2.png) [@philip.scadding](https://community.shotgridsoftware.com/u/philip.scadding)\ **Post date:** [January 21, 2020, 1:43pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/6 "2020-01-21T13:43:21Z")
I thought I’d chime in with a suggestion as well. # Option 1 Another approach would be to have a site config (a `PipelineConfiguration` entity with no project set), using a [distributed setup](https://developer.shotgunsoftware.com/tk-core/initializing.html#distributed-configurations). This will mean that all projects that don’t have their own config will use the site config (you will still need to set a `tank_name` value on each `Project` entity).Then to take over for a specific project, you can just create a `PipelineConfiguration` for that project. That will be a full config take over though. # Option 2 If you just want to handle taking over specific hooks or environment yaml files, then you could maybe make use of environment variables. Here is an example: Here is a config where I just mocked up a test: [garden\_show.zip](https://community.shotgridsoftware.com/uploads/short-url/apmyrnLdFujZN4PDTDU7cCNnbfj.zip) (286.4 KB) In this config, if you were in a project called `garden show` then it would not show the software launchers in Shotgun Desktop, but for all other projects it would. In the config, I used environment variables to define the includes path. ![environment](https://global.discourse-cdn.com/shotgunsoftware/original/2X/6/63f128b8f157098cb914b33c18ea72dfd7bb4122.png) I also added code to the `pick_environment.py` core hook, that contained the following code to set the environment variable based upon the project the configuration was linked to. If a project-specific settings folder was found (named after the project) , then it would define the `PROJECT_SETTINGS` env var to that. If one isn’t found it would define the env var to `settings` (which is the default folder.) ```python if context.project is None: os.environ["PROJECT_SETTINGS"] = "settings" return # set an environment variable that will later be used to resolve the include path to the project # specific settings inside the generic config. settings_folder_name = "{0}_settings".format(context.project["name"]) # remove any spaces from the name settings_folder_name = settings_folder_name.replace(" ", "_").lower() # Check project settings exist config_path = self.sgtk.configuration_descriptor.get_path() settings_folder_path = os.path.join(config_path, "env", "includes", settings_folder_name) self.logger.debug("checking for project specific settings here: %s" % settings_folder_path) if not os.path.exists(settings_folder_path): # no project specific settings found use the default location settings_folder_name = "settings" self.logger.debug("Using settings folder name: %s" % settings_folder_name) os.environ["PROJECT_SETTINGS"] = settings_folder_name ``` I also took over the `bootstrap.py` core hook and added `os.environ["PROJECT_SETTINGS"] = 'settings'` so that if the config is used as a site config then it will take the `settings` block. This is just an example though, there are probably a number of different ways to set up something similar. ---
**Author:** ![Ricardo\_Musch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ricardo_musch/32/405_2.png) [@Ricardo\_Musch](https://community.shotgridsoftware.com/u/Ricardo_Musch)\ **Post date:** [January 27, 2020, 11:27am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/7 "2020-01-27T11:27:21Z")
omg, just realized this is what I asked for in the other topic! 🙈 🙈 🙈 🙈 ---
**Author:** ![Nik](https://avatars.discourse-cdn.com/v4/letter/n/6de8d8/32.png) [@Nik](https://community.shotgridsoftware.com/u/Nik)\ **Post date:** [March 17, 2020, 12:47am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/8 "2020-03-17T00:47:40Z")
Hi Philip, New to shotgun so still getting up to speed on the lingo. I, too, would like to switch to a distributed setup. Currently we have a single project that we use as a template for all other projects. So every time we make an update to that template, we have to manually copy those files/changes over to every project. Not so cute anymore. So I’ve set up a pipeline configuration without a project in shotgun, along with a couple of test projects to see how the distributed setup works. Questions: 1. Now that I have this global pipeline configuration, do I still need the project I’ve been using as a template? 2. You mentioned I need to set a tank\_name on each Project entity. In one of your examples, you named it ‘projects/some-name’. Currently all of my projects are named ‘some-name’. What’s the significance of adding the ‘projects/’ prefix? Is that relevant to it picking up the global configuration? 3. In order to get the existing projects to use the global configuration, do I need to delete each configuration from each project? Right now they each have a single configuration called ‘Primary’. Delete it? 4. Uploaded Config vs. Descriptor: On the global config entity, if I opt to upload the config as a zip file, do I still need a descriptor? If so, why? Thanks in advance! ---
**Author:** ![philip.scadding](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/philip.scadding/32/30_2.png) [@philip.scadding](https://community.shotgridsoftware.com/u/philip.scadding)\ **Post date:** [March 17, 2020, 9:12am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/9 "2020-03-17T09:12:52Z")
Hi Nik, welcome to the forums. Great questions! 1. It’s up to you really. Certainly, you won’t need to copy the files to each project, you will just need to upload a copy of the config to the site `PipelineConfiguration` entity. However, it may be beneficial to have a test project, with a config, so you can test changes outside of production. Personally I would be using git/GitHub where I would keep my source config and track the changes, then zip up the config from the repo and upload that. 2. Yeah, its a good point, perhaps that example is a bit distracting. You don’t need a `projects/` bit in front of the name. You can optionally define a nested structure in the name, by separating it with `/`s. Doing so allows you to add additional folders after the root but before the main project folder. For example, you might wish to split your project folders out by client, or type of project: `vfx_projects/my_vfx_project` and `game_projects/my_game_project`. 3. Yes, that is correct, you will need to delete the `PipelineConfiguration` entity from your Shotgun site for that project. Maybe hold off on deleting the config on disk as well at first, so that you can check it all goes smoothly, otherwise, you can always restore the `PipelineConfiguration` back from the trash, and it should go back to how it was. 3a. No, the projects should already have a `tank_name` if they were previously set up, clearing out the `PipelineConfiguration` entity should be all you need to do (assuming the site config is exactly the same in terms of schema). Though I recommend testing against the process with a dummy project just to make sure you feel comfortable with it. 4. If you upload the config, you should not need to set the descriptor. If you do [it will take precedence over your uploaded config](https://community.shotgridsoftware.com/t/if-i-have-both-an-uploaded-config-and-a-descriptor-on-a-pipelineconfiguration-entity-which-is-used/163). I hope that helps? You’ve probably seen these already, but I just thought I’d link them here as well: > [@Shared project configs](https://community.shotgridsoftware.com/t/shared-project-configs/3284): > > By default when you run the Toolkit advanced set up wizard for a project via Shotgun Desktop, you end up with a configuration created that is specific to that project. However, in certain setups you may wish to share your configuration with all your projects by default. To do this you will need to setup a site-level [distributed config](https://developer.shotgunsoftware.com/tk-core/initializing.html#distributed-configurations). Find the configuration you want to use. You can use the latest [tk-config-default2](https://github.com/shotgunsoftware/tk-config-default2/releases) config (which is the default when using the setup wizard) or you can use a c… > [@Distributed config management workflow](https://community.shotgridsoftware.com/t/distributed-config-management-workflow/232): > > I’ve seen the question come up a few times around how to test configuration changes and push them to production when using a [distributed config](https://developer.shotgunsoftware.com/tk-core/initializing.html#distributed-configurations) setup. My personal recommendation would be to do the following: Assuming your project already has a “primary” PipelineConfiguration entity with an uploaded config, I would create a second one called dev. Set a [dev descriptor](https://developer.shotgunsoftware.com/tk-core/descriptor.html#pointing-to-a-path-on-disk) to define a location to your config on disk. (If you haven’t got a config on disk you can download the primary config and unzi… ---
**Author:** ![Nik](https://avatars.discourse-cdn.com/v4/letter/n/6de8d8/32.png) [@Nik](https://community.shotgridsoftware.com/u/Nik)\ **Post date:** [March 18, 2020, 8:30pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/10 "2020-03-18T20:30:21Z")
Nice one. Thanks for clarifying. Got a couple more questions for ya: 1. Now that my config is global and lives is in shotgun, my projects will need to download apps from shotgun’s remote app store correct? Where does it download it to? 2. I’d like to set up a custom app store as well to be distributed via shotgun. Could you tell me more about how that works? For example, I have a package “maya\_pkg” that lives on disk on a path where all the maya packages/scripts/etc live, “path/to/maya/scripts”. Would it be possible to distribute maya\_pkg via shotgun so that remote users could access it who may not have easy access to “path/to/maya/scripts”? If so, how does that work? Does the user download it from shotgun and access it the same way it accesses the built-in shotgun apps? ---
**Author:** ![philip.scadding](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/philip.scadding/32/30_2.png) [@philip.scadding](https://community.shotgridsoftware.com/u/philip.scadding)\ **Post date:** [March 19, 2020, 9:08am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/11 "2020-03-19T09:08:44Z")
1. By default it will download into the bundle\_cache folder inside you [cache folder](https://developer.shotgunsoftware.com/7c9867c0/). The whole cache location can be changed via the [`SHOTGUN_HOME`](https://developer.shotgunsoftware.com/tk-core/initializing.html#environment-variables) env var. The location should be unique to the user and not on network storage (for performance reasons). You can separate out the bundle cache from the rest of the cache using `SHOTGUN_BUNDLE_CACHE_PATH` and that could be stored somewhere centrally so it can be shared. 2. Unfortunately I don’t have the time right now to write out a complete guide on how to do this, but I think the same thing was discussed here: [DCC plugin distribution via Toolkit](https://community.shotgridsoftware.com/t/dcc-plugin-distribution-via-toolkit/7697/2) which contains a few details. Let us know if you still have questions though and I’ll or someone else will try and get back to you when we can. ---
**Author:** ![brhoades](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/brhoades/32/3721_2.png) [@brhoades](https://community.shotgridsoftware.com/u/brhoades)\ **Post date:** [May 13, 2021, 10:45pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/12 "2021-05-13T22:45:10Z")
Hi @philip.scadding, I’m trying to implement a variant of your proposal, but utilizing environment variables at the `templates.yml` level. I’m struggling because adding the environment variable to `pick_environment.py` and `bootstrap.py` work when a descriptor is pointing at a local config location, and it works the first time that the configuration is downloaded if it’s been uploaded to a distributed PipelineConfig entity. However, once it’s been downloaded, it seems like neither `bootstrap.py` nor `pick_environment.py` is called before it tries to access the templates. Do you know of any workarounds? Why isn’t `bootstrap.py` called after the first time? Is there another core hook that’s called earlier that can be used instead? ---
**Author:** ![Ricardo\_Musch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/ricardo_musch/32/405_2.png) [@Ricardo\_Musch](https://community.shotgridsoftware.com/u/Ricardo_Musch)\ **Post date:** [May 14, 2021, 9:55am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/13 "2021-05-14T09:55:35Z")
Maybe this helps, I have various template paths in project specific templates.yml files (example `proj1_templates.yml, proj2_templates.yml`) The templates.yml main file has an include that references an \_templates.yml So if you set the env file in `pipeline_configuration_init` core hook then the templates.yml include will redirect to the file you want. ---
**Author:** ![brhoades](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/brhoades/32/3721_2.png) [@brhoades](https://community.shotgridsoftware.com/u/brhoades)\ **Post date:** [May 14, 2021, 4:10pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/14 "2021-05-14T16:10:31Z")
Hi @Ricardo_Musch , Thanks, `pipeline_configuration_init` was precisely the core hook I needed. ---
**Author:** ![nico.vandenbosch](https://sea1.discourse-cdn.com/shotgunsoftware/user_avatar/community.shotgridsoftware.com/nico.vandenbosch/32/4949_2.png) [@nico.vandenbosch](https://community.shotgridsoftware.com/u/nico.vandenbosch)\ **Post date:** [July 2, 2021, 8:22am UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/15 "2021-07-02T08:22:10Z")
I’m attempting a slight variation on Option 2 here, using an environment variable to specify the external location of app settings yml file. Only, I’m adding one more level of indirection by leaving the original settings.yml file with an include to the external settings file. This approach minimizes the number of places we’d need to edit where the standard location of the settings yml file is still referenced. The setup is like so: ./includes/settings/tk-3dsmax.yml ```auto includes: - ./includes/settings/tk-multi-loader2.yml settings.tk-3dsmax.shot_step: apps: tk-multi-loader2: "@settings.tk-multi-loader2.3dsmax" ``` ./includes/settings/tk-multi-loader2.yml ```auto includes: - ../app_locations.yml - $TK_MULTI_LOADER2_SETTINGS_PATH # the environment variable resolves to an external path like /proj/foo/settings/tk-multi-loader2.yml ``` /proj/foo/settings/tk-multi-loader2.yml ```auto settings.tk-multi-loader2.3dsmax: action_mappings: 3dsmax Scene: [import, reference] ``` The only problem is, toolkit chokes on this and is unable to apply the settings. Turns out core’s env parsing does not properly allow for this many levels of recursion on includes. Specifically if a @setting is asked for in file A (tk-3dsmax.yml), then the recursion algorithm expects that setting to be immediately defined in a directly included file B (tk-multi-loader2.yml). If the setting is defined one level down in an indirectly included file C (/proj/…/tk-multi-loader2.yml) it gets filtered out by the include recursion because it’s not immediately referenced. I’m assuming this is not by design, but arguably a bug that may not have gotten much daylight? ---
**Author:** ![FilSus](https://avatars.discourse-cdn.com/v4/letter/f/e79b87/32.png) [@FilSus](https://community.shotgridsoftware.com/u/FilSus)\ **Post date:** [January 29, 2024, 6:02pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/16 "2024-01-29T18:02:03Z")
Hey, a bit late to the debate, I am trying to execute the same setup of pointing the toolkit to project specific `templates.yml` files. I have added the `pipeline_configuration_init` hook to setup the environment varaible, but have no clue how to make it project specific since I cannot import sgtk. Probably too early in the process. > os.environ[“PROJECT\_TEMPLATES”] = “{project\_name}\_templates.yml” Also even if I set it up manually by providing it with the exact file like `project1_templates.yml` it does not get registered. I have simply added this code to the original `core/templates.yml` > includes: > – ./templates/$PROJECT\_TEMPLATES But after launching Nuke it is still picking up the template.yml instead of the project1\_templates.yml * * * UPDATE * * * So the Toolkit actually picked up the `project1_templates.yml` but the paths got overwriten by the default `templates.yml`, looks like I need to leave this file empty so that it would not overwrite the included paths. ---
**Author:** ![FilSus](https://avatars.discourse-cdn.com/v4/letter/f/e79b87/32.png) [@FilSus](https://community.shotgridsoftware.com/u/FilSus)\ **Post date:** [January 29, 2024, 7:12pm UTC](https://community.shotgridsoftware.com/t/recommended-way-to-share-update-pipeline-configurations-between-projects/5609/17 "2024-01-29T19:12:30Z")
OK, this post is the solution to my problems. > [@Template.yml per-project includes](https://community.shotgridsoftware.com/t/template-yml-per-project-includes/3681/7): > > Hey Patrick - I added this line to templates.yml and it worked: include: "${INCLUDE\_PATH}/test.yml" so we definitely do support env vars in template includes. As for when to set the env var, pipeline\_configuration\_init is the right place, and it looks like you can get to the project name via self.parent. Here’s where the project is getting set and the pipeline\_configuration\_init core hook is getting executed (conveniently adjacent to one another slight_smile ). You can see that the core h…