Am I alone? Single pipeline config with per-project overrides?

[Patrick]

I’ve been moving towards using a single config for all projects in a studio, with the obvious benefit that there’s far less code to maintain.

One of the hurdles to this is that there are ARE certain configurations that currently cannot be abstracted from the shared config on a per-project basis without forking the pipeline-config. The tk-writenode profiles being an obvious one.

I’ve almost implimented a solution that allows us to have optional ymls residing in a config folder in each projects folder where setting overrides can be defined. We’ve managed to do this by adding an additional include in the nuke.yml definition that points to a path with an environment variable for the project folder eg :

includes:
- ./tk-nuke-writenode.yml # Pipeline-config based writenode settings.
- "$PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml" # Project folder based writenode settings. 

This would work if it weren’t for the fact that tk-core implements a validation of include paths, and throws a tankError if the path doesn’t exist. This behaviour prevents SG Desktop from loading as there is no project context when launching SG Desktop, so no project_path.

Am I alone in seeing this approach as being beneficial for how to configure a studios pipeline? Am I storing up problems for myself by complicating where configurations are located?
As I see it, any disadvantage of this approach is easily outweighed by the benefits of having a single config to maintain across an entire studio.

I’m curious to hear your thoughts!
Thanks in advance for your contributions.
p.

[Romey]

Your not alone. A global config with project overwrites is ideal. In the past I have lobbied for this. It’s a challenging thing to implement at a studio that has legacy configs and tools. Please keep sharing your progress on this topic. Would love to see SG/TK migrate to this. All roadblocks you mentioned should be prioritized in the roadmap ASAP in my opinion. As a result this would empower studios with more flexibility to manage the technical debt that builds up with per project configs.

[Patrick]

Thanks for your input. Eloquently put!

[philip.scadding]

I know the scenario is a bit different here, but you could use the template core hook, a bit like I suggested here:

Workfiles tab order

You can do either, but it does get a bit messy here. Obviously it’s nicer if it is defined in the yml file so all your settings are kept together. But to do this you would need to pass the settings as a string args to the hook. Each : after the hook name denotes args. So you could, in theory, pass the settings through as string args. Here is an example In the yaml file I’m passing two parameters through to my custom hook called example_template_hook, though the hook can be renamed to somethin…

You could then provide different setting depending on the project, though it’s maybe not quite a elegant, and is a bit more error prone. Basically you could pass a list of settings through, and maybe add a project list to each, and then in the hook filter out those settings that don’t apply to current context project.

[Patrick]

Hi Phil,
Oh my word… interesting suggestion!, but that really calls on me to up my yml-fu!

“maybe add a project list to each” - It sounds like this wouldn’t really address the issue of increasing technical debt… quite the opposite in this case?

Thanks for sharing this though, I never knew you could gather settings from a hook… although I’m struggling to understand quite how your example works. Is this documented anywhere else?

I think what I, and I assume others, are looking for, is an elegant solution. Being able to have optional yml files living in the project folder structure would appear to be a good candidate for this.

“Optional” being a key requirement here… we don’t want to have to have redundant files in the schema to get this to work. Plus, if it’s not optional, then backwards compatibility would fail.

[Patrick]

I’ve updated a fork of tk-core that appears to work; supporting non-existant includes without any adverse effects. It logs missing includes as log.warning() so it’s clear to any TD that there may be a problem with the config.
I’ll add a pull-request to the official repo so others can test this out and get some feedback.
I’ll also post here with details on how to include your project path in include paths.

[Patrick]

Ok, I’ve created a pull-request with a simple, and largely untested implementation of this.
There are instructions for how to get this working in our config in the pull-request description.

Allow per-project overrides of env setting ymls.

Let me know if you have any trouble getting this to work, or if you discover any bugs.

Thanks!

[Patrick]

@sg-team can someone take a look at this? I’ve introduced this update to a production environment without issue, but would prefer not to work from a forked core.
Is there scope for the devs to contribute here? If my approach isn’t deemed to be valid, I’d be glad to hear of other approaches that might achieve what I’m looking for.
Thanks!

[philip.scadding]

Hey @Patrick thanks for submitting this, I have logged it internally.
I can’t say when it will get looked at but I’ve flagged it to discuss.
Whilst I appreciate the amount of lines changed is fairly small, the change itself could potentially have big implications and we’d need to discuss them.

[Patrick]

Thanks Phil,
If it’s any help, your reservations can be very easily avoided.
My changes can be wrapped in an env var check of ‘SHOTGUN_DISABLE_MISSING_INCLUDE_TANKERROR’. That way there is no affect unless a studio consciously decides to enable this feature.
Win-win… unless I’m missing something.

[Ricardo_Musch]

This looks very cool and exactly what I would like to achieve too!
At the moment I have a “Pipeline Test” config in which I test tools and then link the yaml files from this into the projects but it would be much nicer to keep 1 config up to date (maybe even use distributed configs so I just have to switch out the zip) and then have overrides overriding certain yamls

[jfboismenu]

Hi,

We’re looking into a more easily supportable version of this feature right now. We haven’t scheduled a time to work on it yet, nor can we guarantee it would be implemented, but the idea behind it is this.

Instead of having having a global toggle as an env var to do this, we were wondering if the following syntax in the yaml would work for your workflow?

includes:
- path/to/mandatory/include.yml
- {path: path/to/optional/include.yml, optional: true}

We could also support something like ?path/to/optional/include.yml. The idea is to make it explicit in the config what is optional and what is not. This allows Toolkit to log appropriate messages.

What do you all think?

[Ricardo_Musch]

Let me think on that!

I’m currently working on my own bespoke approach which uses the pipeline_configuration_init core hook.

In that I do the following:

• Check for primary or beta mode (my pipeline config install directory’s have “primary” or “beta” in the name) and it affects some other tools at the moment

• checking for the existence of a “settings” folder just outside the shotgun install folder, if not it creates it and copies example yaml files (the original yamls) into the structure so that they simply have to be renamed from .example to .yml to automatically count as override.

• checks for the existence of each yaml file in that settings folder and if one exists it uses that as override for the one that lives in the config folder.
  At the moment I do that by changing the includes paths to include a env_var for the path.
  The path is normally just: \\path_to_current\shotgun_install\config\env\includes\settings
  but if an override yaml is found the path is changed to my override directory (and the yaml is named the same).

[Patrick]

This is great. Seems to do what we need it to do!

[Ricardo_Musch]

What do you mean by optional?
Would it ignore if the file doesnt exist?

[jfboismenu]

Yes, exactly, this is what Patrick was saying he wanted to do.

When Toolkit does not find an include file, it raises an error, assuming that something is wrong in your setup. With the optional flag, it means that this file can be missing. This would allow anyone to have an include set to $SOMETHING/tk-multi-workfiles2/scene_operations.py that is missing on disk and not raise an error.

---

With that being said @Patrick , I was just looking at the Toolkit source code for dealing with includes and rediscovered a featured I had forgotten existed (I even wrote tests for it when I refactored that code years ago!!!). It allows you disable an include. It’s a bit hacky, but would not require you or us to modify tk-core.

Given the following includes and assuming you are on Windows:

includes:
- "relatve/path/to/file.yml"
- "C:\\absoluote\\path\\to\\file.yaml"
- "\\absolute\\path\\to\\file.yaml"
- "/absolute/path/to/file.yaml"

1. the relative path is will always be evaluated on any platform.
2. The C:\ and \absolute file paths are absolute. Being Windows specific paths, they will always be evaluated on Windows and skipped on macOS/Linux.
3. The /absolute/path/to/file.yaml, being a Linux/macOS absolute path, will always be evaluated on Linux/macOS and skipped on Windows.

Note: It does seem this feature is half-baked, since you can’t have a Linux specific or macOS specific include.

@Patrick , maybe you could leverage this feature instead of modifying Toolkit to treat a file as missing? In your example above, $PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml, $PROJECT_PATH when on Windows could be set to /disable/this/path if the project does not require overrides. I suspect that this is not something you can actually use in your case, since it would disable all overrides using that env var, but you keep in your backpocket for future use.

[Ricardo_Musch]

@jfboismenu

@Patrick , maybe you could leverage this feature instead of modifying Toolkit to treat a file as missing? In your example above, $PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml, $PROJECT_PATH when on Windows could be set to /disable/this/path if the project does not require overrides. I suspect that this is not something you can actually use in your case, since it would disable all overrides using that env var, but you keep in your backpocket for future use.

Pretty sure shotgun desktop will error out if any of these paths point to a non existent file though, env variable or not.

[jfboismenu]

Well, we have unit tests that prove otherwise. Maybe I’ve not explained this the right way.

If you are on Windows and one of your includes evaluates, through env var or being hardcoded this way, to an absolute posix path, then the include is automagically skipped as Toolkit knows this path is not meant to be used on the current platform. The opposite is also true for a Windows path for an include on Linux/macOS.

[Ricardo_Musch]

On Windows is doesn’t seem to ignore the include if the path doesn’t resolve to an exisiting file.

my pipeline_configuration_init has the following line:

os.environ[“tk-nuke-writenode_path”] = “g:/”
(in your proposed setup this would contain an env var that points to a override location)

[jfboismenu]

Only Linux/macOS paths are ignored on Windows. Set the environment variable to /other/os/path.