Keeping pipeline configurations current among various projects

I’m in the process of re-engineering our studio configurations to better keep up with changes coming from Shotgun. Our current method of working involves a basic config and core package that we keep in a git repo which we clone onto our network and branch off for each new project that starts up.

What is the best way to keep those configurations current to the latest version of tk-config-default2? Do you expect those to change frequently?

How would I keep the core/apps/install files current if I just have the configs in a git repo? Use them to create a new project with every time we onboard another project? I suppose I’m a bit confused by what is the “core” and tk-core. It looks like there’s some code overlap there. I’ve been using a separate tk-core to bootstrap distributed configurations I have been testing but so far all of our current projects are centralized in that they have their own install files/bundle_cache.

When tk-core is referred to by version, does this go for the whole “core” as well? That is all the install files?

I don’t expect the files in the core to change. All of our overridden hooks live in the base configs git repo anyway. I just want to be certain that the core/tk-core will always be the latest when we bring on a new project.

Hopefully this makes sense. I think I might be favoring a distributed structure but I want to make sure I have a good sense of all the components involved.

Hi, we also use git for managing our configurations. core is the main part of toolkit; there are several other modules. tk-core if I understand correctly is the package of all that, which gets cloned to each local centralized project’s directory and gets loaded from there for the project.
This is basically toolkit.

We have a central location - a bare git repo - for the tk configuration, from which everyone else pulls. Something can be changed in one project, and pushed to the central repo, from which e.g. other projects can pull, new projects can clone.

For new projects, if your central config git repo is up to date, they will automatically be the latest version. In the Shotgun destkop wizard, for config source select “git”, and point it to your central repo. This clones the config. The project dir also contains core, under install/core/python.

The best workflow for updating I found is this - there is a test project, where I perform upgrades and test.
You use the tank command:
tank updates - for apps and engines
tank core - for updating core

Then I commit the changes and pull to all other projects with a script.
Finally, you have to run
tank cache_apps
in each project to pull the apps that have been updated in the new configuration.
This is easy with a bit of automation. The nice thing is you have a lot of control over each project.

This became a bit longwinded, and I cannot list all relevant info, so feel free to ask further questions.

Edit: this recent thread is relevant

This helped visualize the process quite a bit. Thanks!

Looks like I can sync the core/configs updates via the shotgun desktop which is useful if I don’t have a specific tank executable available for the project.

Question about how it updates configs: Let’s say I modify some yaml files in the env/includes directory to not include engines that I won’t be using (after effect, flame, etc).
I’m assuming these changes will be overwritten when I update the configs.

Is there a way to have an override for things like this that won’t affect updates? Even though I may not be using certain engines that wouldn’t mean I want to exclude all updates to the yaml files.

This is sort of related to my question here where I don’t want to interfere with updates to roots.yml yet I want to make my own additions.

We already do this to some degree with our templates.yml file. We keep the default templates that are unchanged in a separate yaml file and then include them in the templates.yml file that has our overrides and new templates.

I think the roots question is separate, since they are managed differently. If you remove apps/engines from your env config, you should be fine - they will not be added in again, afaik. Updating engines is related to what you have in the env config only. Someone correct me if I’m wrong.

This is where the power of git comes into play. I would start off with reading Advanced git usage: branches and then looking online at images or videos of how git rebase works, as well as “resolving merge conflicts”.

This is one of the routes you can go down to apply your changes on top of new shotgunsoftware/tk-config-default2 changes

1 Like

I’m thinking that keeping it as a centralized configuration makes our lives a bit easier. But I do have some questions there.
We’re going to maintain a base git config that we will run create project from shotgun desktop/tank to build subsequent projects. Let’s say we have a new branch that we want for some project that isn’t the base config.
Can you specify a branch when creating a new project from git or does it always create from the master (and then I have to fetch said branch after the fact)?

So far we check out a project-specific branch after the project is created. This is fine since e.g. templates are not used until artists start saving files, etc.
But not sure if you can specify a branch during project setup.