Dynamic Pipeline Configurations

Pipeline Integrations
1

Hi!

I am currently developing our shotgun site configuration based on an episodic project, where we have episodes, sequences and shots.
But for other smaller projects, this configuration will not work since it would be an overload and we might not need episodes or even sequences.

One solution would be to have different pipeline configurations for those solutions,
but that would require probably a lot of redundant work, right?

I wonder if it is possible to combine, for example,
a structure with and without an episode dynamically within one single pipeline configuration.

I found a very interesting topic here:

But how could I, for example, create a folder schema like this (when episodic):
[episode]/[sequence]/[shot]
and when not episodic like this:
[sequence]/[shot]

It looks like the “pick_environment.py” hook could be a entry point into this,
but from then on I struggle to find a way to dynamically set the config data I want to,
especially in the core of the config with the schema and templates.

It would be nice if someone could point me in the right direction here!

Thanks a lot,

cheers
Sebastian

4 Likes
3

Hi @sebastian.brandhuber!

Not necessarily, you could as you mentioned have a separate PipelineConfiguration entity for each project. But each PipelineConfiguration entity that wants to share the config, could point at the same config bundle. If for example, you were using a path descriptor they could all point to the same config on disk, but the same applies to any of the other descriptors, such as git or shotgun upload, they could point to the same source.

This I think would be more work and becomes messy and confusing, but is doable.
You would have your schema set up so that the Sequence folder in the {Episode}/{Sequence}/{Shot} hierarchy only got created when an episode was provided using the filters.

- { "path": "episode", "relation": "is_not", "values": [ null ] }

And the reverse for the Sequence folder in the {Sequence}/{Shot} structure.

- { "path": "episode", "relation": "is", "values": [ null ] }

That would then mean you would need a set of templates for episodic and a set for non episodic. Note it is possible it define optional fields in a template using the square brackets syntax ([Epsiode] ) but this is only intended to make keys optional in the file name and not in the path, you would likely hit errors if you tried to make the episode field optional, so its best to have two sets of templates, episodic and non episodic.

That would then mean you would need to change your environment depending on whether it was episodic or not. And at that point, you would as you mentioned need to change the pick_environment.py hook to if the sequence had an episode, and then choose the correct environment accordingly.
This means that your environment files need branching depending on whether or not you are working with episodes. Option 2 in the post you linked could help make that easier, but in my opinion, I think what your trying to do with the schema means you are better off having separate configs.
I would be almost like cramming two configs inside one.

I hope that helps?

4 Likes
4

Hi Philip,
thanks for the explanation.

I have two following questions:

  1. As you described, it would require me to branch the environment files based on working with episodes or not. In the linked post, you suggest using env variables and this part is clear, but how could I, for example, have a templatesA.yml and templatesB.yml and chose between those?
    I could not find a possibility to “change” a path or link for that with a custom variable.
    Or thinking bigger - maybe change folders like “schmema” to “schemaA” and “schemaB”.

  2. You suggest that having separate configs would be the better approach.
    Even if they are two completely different config bundles, would I be able to “share”
    files between them? like hooks or .yml files?

thanks a lot!
cheers

Sebastian

2 Likes
5

You can only have a templates.yml. Off the top of my head, I think you can maybe include other yamls from within it, I’m not certain, but what I was saying was you would need episodic and non episodic templates within the templates.yml.

This isn’t possible, at this point the answer is really to have a separate config.

Depends on how your configs are setup. Are they centralized or using path descriptors, or are they uploaded or pulled from git?
If they are always in a hardcoded location, then you could, in theory, include files from locations outside of the config, but if they are distributed (downloaded on demand) then you can’t be sure that the other config will exist locally.

2 Likes
6

Okay, that helps me a lot.
We are using a distributed setup where we develop the config with git, but then
upload the config for distribution.

But I have the feeling that we really should aim for separate configs here.
Thanks,

Sebastian

3 Likes
7

Sebastian,

We’ve got the same setup, and we’re also looking to use that episodic / non-episodic folder structure. I can see that having two configs greatly reduces the amount of templates needed, but it seems that it’ll require extra work to push updates two two separate configs every time any config updates are made. Do you (or anyone reading this) have a painless way to keep two slightly different configs in sync with future updates? We’re also using distributed config with git.

2 Likes
8

It’s been a few years but I just stumbled across this thread in my hamfisted attempts to get a single site config to support both episodic and non-episodic projects. When it came to the templates.yml the optional fields were the appealing route but ended up being wildly unstable as @philip.scadding suggested. In our case, reference to the episode token were limited to our shot_root alias, so the easy solve was to move that part of the paths definition to its own dedicated yaml file to be included into templates.yml like so:

includes:
- ./template_${SG_PROJECT_TYPE}_roots.yml

From that point it was just a matter of defining the $SG_PROJECT_TYPE env var in the pipeline_configuration_init.py core hook (before the templates are loaded) and we were off to the races.

4 Likes
9

As a sanity check for who come across this in the future, I’d like to note explicitly one code example that proves this point from Philip.

Note it is possible it define optional fields in a template using the square brackets syntax ([Epsiode] ) but this is only intended to make keys optional in the file name and not in the path, you would likely hit errors if you tried to make the episode field optional, so its best to have two sets of templates, episodic and non episodic.

When attempting to have an optional portion of the sequence_root template in templates.yml that looks like something like this:

...[/episodes/{Episode}]...

I noticed that the File Open dialog provided by tk-multi-workfiles2 no longer was able to create a new file for a particular Task context (using the + New File button at the bottom left of the window).
Upon clicking this button, I was greeted with an error like this:

tank.errors.TankError: Cannot resolve template fields for context '<task name>, Shot <shot name>' - the following keys could not be resolved: 'Shot, Step'.  Please run the folder creation for '<site shotgun url>/detail/Task/<task ID>' and try again! //

Tracing through the code lead me down this path:

  1. This line in new_file_action.py of tk-multi-workfiles2.
  2. This line in context.py of tk-core.
  3. This next line in context.py of tk-core.
  4. This final line of context.py of tk-core. This line is what is causing at least some of sgtk’s incapability to support the ability to have 1 templates.yml that supports both episodic and non-episodic projects.

The more exact position of incompatibility lies here in template.py of tk-core. It seems that the parent property of a TemplatePath doesn’t function correctly when optional paths are present in the dirname portion of a path.

For example: If I have defined a template called shot_root like:

>>> shot_root_template = tk.templates['shot_root']
>>> print(shot_root_template)
<Sgtk TemplatePath shot_root: production[/episodes/{Episode}]/sequences/{Sequence}/shots/{Shot}>

then run

>>> template.parent
<Sgtk TemplatePath production\episodes\{Episode}\sequences\{Sequence}\shots>

Calling template.parent bakes in the optional keys here, and doesn’t consider the ‘parent’ path of the TemplatePath without applying the optional keys in the path.

_get_template_ancestors in context.py doesn’t account for this shortcoming of template.parent and therefore the templates.yml doesn’t work for both episodic and non-episodic projects.

1 Like