Refactoring templates

Pipeline Integrations
1

We are in a situation where we have over 200 templates. We do not need all of them, but even if I cleaned up, they would still be too many. And the bigger problem is with the ones we do use.

There are several dimensions along which templates are separated:

  • work vs publish - their directory hierarchy is different, work files are usually in user sandboxes, etc.
  • asset vs shot - again different hierarchy, and often assets are not user-specific.
  • dcc app - esp. tag/output/node field - different dcc tk apps have hardcoded different field names for the tag. I thought field aliases were the solution to this, but could not get it to work.
  • file type and roots - for instance we have separate roots for renders, caches. Scene and render templates are quite different in other ways.

We are getting a combinatorial explosion of these, and managing them becomes increasingly difficult.

We have abstracted some pieces away into their own templates that get referenced (e.g. ā€œcache_base_nameā€), but it is not satisfactory at this point.
Of course, one answer would be to simplify our naming conventions, but some decisions are very practical, such as having different roots.

Anyone reached template nirvana and willing to share?

3 Likes
2

Hey Mois ā€“ moved this one over to the Pipeline Integrations category so we can get more Toolkit eyes on it.

One thing you might want to consider is using separate files to organize your templates via includes ā€“ doesnā€™t limit the quantity of templates you have, but at least makes the files shorter and hopefully easier to look at.

Hereā€™s a simple example using includes in templates.yml:

templates.yml (not including all of it here, just the important bits)

include: "./test.yml"


paths:


    sequence_root: sequences/{Sequence}

    foo_bar_in_master: "@sequence_root/@foo_bar"

And the included test.yml file:

keys:
    Foo:
        type: str
    Bar:
        type: str

paths:
    foo_bar: test_path/{Foo}/{Bar}

strings:
    foo_bar_name: "string_{Foo}_{Bar}"

And when I run this in a shell, I get:

>>> tk.templates["foo_bar"]
<Sgtk TemplatePath foo_bar: test_path/{Foo}/{Bar}>
>>> tk.templates["foo_bar_name"]
<Sgtk TemplateString foo_bar_name: string_{Foo}_{Bar}>
>>> tk.templates["foo_bar_in_master"]
<Sgtk TemplatePath foo_bar_in_master: sequences/{Sequence}/test_path/{Foo}/{Bar}>

Beyond that, Iā€™m spitballing, but maybe itā€™d be helpful to employ some sort of bidirectional conversion between YAML and CSV, so you can view/edit your data in Google Sheets or Excel?

And, beyond that, :crossed_fingers: that you get some good ideas from the community. Good luck!

4 Likes
3

A word of advice on what probably not to do. Our templates are fairly consistent across DCCs and shotgun contexts so we found ourselves repeating the same work and publish templates over and over again varying only the relevant key. We thought itā€™d be a good idea to consolidate all of these into a series of ā€œgenericā€ templates which relied heavily on optional keys to describe the context (the rationale being that contexts are largely mutually exclusive, so you wouldnā€™t have a key like ā€œAssetā€ and ā€œShotā€ defined in one context):

generic_publish:
definition: ā€˜[assets/{sg_asset_category}/{sg_asset}/][sequences/{sg_sequence}/][sequences/{sg_shot_sequence}/{sg_shot}/]{sg_step}/publish/{task_name}/{version}[/{width}x{height}][/{name}]/{Project}[{sg_asset}][{sg_shot}]{task_name}[{name}]_v{version}[.{seq_frame}].{ext}ā€™
root_name: ā€˜primaryā€™

This worked well for a while until we started spotting significant increases in startup time in toolkit. Note that the above example contains 9 optional keys, which tk-core internal expands to a list of every conceivable permutation of template definition for a total of 512 (2^9). Iterating through hundreds of templates in python is still manageable and accounted for mere seconds of additional overhead. It wasnā€™t until we introduced new contexts and increased the number of optional keys to 14 (for a grand total of 2^14=16,384 permutations) that startup went from seconds to minutes.

Long story short, weā€™ve rolled back to a lengthy list of individually managed templates. Would likewise love to know if thereā€™s any other solutions that have been implemented with greater success.

6 Likes
4

Thank you both,

@tannaz this would actually help, we could divide templates by dcc or by purpose. Readability is indeed a concern.
Re: the conversion idea, it might work to have an additional layer (e.g. python) which allows you to manage templates at a higher level, while preserving them more or less as they are now.

@nico.vandenbosch ooh, having a lot of optional keys does sound difficult to manage, too. Funny that it impacted performance as well.

One thing I would like to see is tk dcc apps provide a configuration option for what the tag field name should be. This wonā€™t be difficult to pull off, and would reduce the number of templates by a factor of {NUMBEROFDCCS}.

2 Likes
5

Mois, can you say more about

Maybe give an example of what you have in mind? Iā€™m not sure what ā€˜tag fieldā€™ youā€™re referring to.

1 Like
6

Right, I did not qualify it. Usually you can output more than one artifact from a scene.
For renders, you have a {name} field for the scene name. You have one extra token for the artifact:

  • tk-nuke-writenode uses nuke.output
  • tk-houdini-alembicnode uses houdini.node
  • in max we use our own name, so that could be changed easily.

So the name is something like {project}_{sequence}_{shot}_{name}_{render_tag}.exr
but we need a separate template for each dcc.

Is this something that field aliases can help with? I couldnā€™t wrap my head around them.

2 Likes
7

Ah, thatā€™s interesting. Iā€™m not sure itā€™s a direction we want to go in, but Iā€™ll share it with the product team and leave it to them to decide.

As far as the alias setting on keys, youā€™re not the first to be confused by it! Unfortunately itā€™s not helpful here: itā€™s more for the case where an app is expecting a field called, for example version, but you need to define version differently for different situations, so you have a naming clash. In that case, you can define two keys, say, nuke_version and maya_version, but alias them both to version so that the app thatā€™s reading the template gets the key itā€™s expecting. For the record, here is the documentation for the alias setting.

2 Likes
8

Agreed, @mmoshev the one-liner looks pretty heinous but considering that it was designed to not only replace multiple template definitions as well as entire entity context settings blocks in tk-multi-app.yml files per app the reduction seemed like a reasonable trade-off in complexity.

3 Likes