Names for apps/frameworks specified via shotgun descriptors

We’re using app and framework descriptors of type “shotgun” to deploy tools remotely as described here: https://developer.shotgunsoftware.com/tk-core/descriptor.html#pointing-at-a-file-attachment-in-shotgun

This all works fine to a point. The configuration, apps, and frameworks all download themselves into a bundle cache on the client-side. However when the app is run, it crashes and burns when it explicitly attempts to import a framework by name using sgtk.platform.import_framework(‘tk-framework-studio’)

This seems to be due to the way toolkit assigns names to frameworks with shotgun descriptors:

print engine.apps.keys()
[‘tk-multi-vendordelivery’, ‘tk-multi-pythonconsole’]
print engine.apps[‘tk-multi-vendordelivery’].frameworks.keys()
[‘CustomNonProjectEntity02_5’, ‘CustomNonProjectEntity02_4’, ‘tk-framework-qtwidgets’, ‘tk-framework-studiowidgets’, ‘tk-framework-login’, ‘tk-framework-shotgunutils’]

Note the ‘CustomNonProjectEntity02_*’ frameworks are what we want to import, but we’re unable to identify it by name. Interestingly, our ‘tk-multi-vendordelivery’ app is also defined via a shotgun descriptor but gets to keep its name. Also noteworthy is that the ‘name’ field is disallowed on shotgun descriptors so how toolkit is arriving at these names is anyone’s guess. Is this a bug? Working as designed? Something we overlooked in the configuration?

1 Like

Hmm that is odd.

I had a go at replicating, but it seemed to work fine for me.

In my env/includes/frameworks.yml:

tk-framework-randomnumber_v1.x.x:
    location:
      type: shotgun
      entity_type: CustomNonProjectEntity02
      id: 6
      field: sg_payload
      version: 30398

In my Shotgun site I have a CustomNonProjectEntity02

In the custom app’s info.yml:

...
frameworks:
  - {"name": "tk-framework-randomnumber", "version": "v1.x.x" }

In the custom app’s code I have:

...
self._app = sgtk.platform.current_bundle()
print self._app.engine.apps['tk-multi-starterapp'].frameworks.keys()
rn = sgtk.platform.import_framework('tk-framework-randomnumber', 'tkrandom')
...
>> ['tk-framework-randomnumber']

So it imports and lists the framework under the app’s list of frameworks.

Is there anything different I might have done here to you?
Best
Phil

That is odd Phil. Your repro steps are pretty consistent with what we’re doing. Nothing jumps out as different. We might need to take this into the debugger to see what noodly little detail we might’ve missed.

In the meantime, we’re successfully working around this by switching our framework descriptors to ‘path’ and embedding them directly in the pipeline config payload.

Thanks

1 Like

Hi Phil,
Thanks for replying. That does seem to be the same issue.
I also fixed it by using a dev descriptor, but that isn’t really a lasting solution.
See if you can recreate it with an uploaded shotgun descriptor based engine definition (that uses {engine} field in the hook template).
I’ll have a dig around the code to see if I can find the source of the issue.
p.

1 Like

Hah, that didn’t take long.

The shortname is derived from the entity type and version number.
This probably works from the assumtion that an uploaded config follows the convention of the app_store, but that won’t be the case for typical studio usage of the shotgun descriptor.

Either we’ll be forced to use the engine name as the toolkit bundle name for uploaded app/engines/frameworks (which isn’t viable given the website not providing any additional info on dropdowns in cases where you are trying to select a value for an entity field) or another approach to the shotgun descriptor system may be required; eg another field to override this behaviour, eg ‘system_name’ where one can enter the desired value, eg ‘tk-houdini’.

I’m not going to be able to offer a pull-request on a fix for this as the knock-on effects of changing the descriptor class behaviour are far and wide by the looks of it.

2 Likes

I’ve submitted a pull-request with a fix for this that appears to work.
The approach is to add a “system_name” field to the descriptor to allow users to override the default descriptor behaviour. I’m not sure how else this could be achieved as there’s no easy way for the code to know what the correct name of the toolkit bundle should be for the {engine} template field to resolve correctly.

1 Like

Cheerio guys,
is there anything new about this? I’m experiencing something really similiar I think.
I’ve uploaded a custom tk-nuke engine to Shotgun as a new Toolkit Component, referencing it in the Pipeline Configurations with the shotgun descriptor type.
However, when launching Hiero I get the following error:

[ERROR tk-hiero] App configuration Error for tk-hiero-export (configured in environment '[PATH_TO_PIPELINE_CONFIG]\env\project.yml'). It will not be loaded: The app could not be loaded since it only supports the following engines: ['tk-hiero', 'tk-nuke']. Your current engine has been identified as 'CustomNonProjectEntity02_36'

Could you give me some advice on whether or not to include the changes of Patrick into a custom tk-core as it is not implemented into the official tk-core according to GitHub?!

Bumping this topic as the problem has come up with another user on the shotgun-dev google group. Given I’ve submitted a fix that works, it might be worth fast-tracking through testing in to a production release?

2 Likes

As Patrick said, we were talking about this error on the shotgun-dev google group (this post)

I uploaded some apps and engines on CustomNonProjectEntity03 and I uploaded the pipeline configuration with location type as ‘shotgun’ in the same way as above, and I got the error below:

Failed to complete ‘New File’ action:
Cannot execute hook ‘C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\bundle_cache\sg\vetorzero\v12469\hooks\scene_operation_CustomNonProjectEntity03_5.py’ - this file does not exist on disk!

With this Traceback:
Traceback (most recent call last):
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\bundle_cache\sg\vetorzero\v12469\python\tk_multi_workfiles\actions\new_file_action.py”, line 75, in execute
if not reset_current_scene(self._app, NEW_FILE_ACTION, self._environment.context):
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\bundle_cache\sg\vetorzero\v12469\python\tk_multi_workfiles\scene_operation.py”, line 90, in reset_current_scene
return _do_scene_operation(app, action, context, “reset”, result_types=(bool,))
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\bundle_cache\sg\vetorzero\v12469\python\tk_multi_workfiles\scene_operation.py”, line 54, in _do_scene_operation
read_only=read_only)
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\vetorzero\p278c117.basic.desktop\cfg\install\core\python\tank\platform\bundle.py”, line 575, in execute_hook
**kwargs
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\vetorzero\p278c117.basic.desktop\cfg\install\core\python\tank\hook.py”, line 605, in execute_hook_method
hook = create_hook_instance(hook_paths, parent, base_class=base_class)
File “C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\vetorzero\p278c117.basic.desktop\cfg\install\core\python\tank\hook.py”, line 675, in create_hook_instance
“Cannot execute hook ‘%s’ - this file does not exist on disk!” % hook_path
TankFileDoesNotExistError: Cannot execute hook ‘C:\Users\gabriel.valderramos\AppData\Roaming\Shotgun\bundle_cache\sg\vetorzero\v12469\hooks\scene_operation_CustomNonProjectEntity03_5.py’ - this file does not exist on disk! //

I would really appreciate it if someone could help me with this problem too.

Thank you

1 Like

That’s an interesting error, I don’t ‘think’ my updates will help there… you can try though and let us know!

1 Like

Hi Gabriel,

When you’re using Shotgun descriptors and you use the “id” field to refer to your CustomNonProjectEntity, the bundle gets referred to by the name: CustomNonProjectEntity##_{id}. But if you use the “name” field instead, the bundle will be named as expected. This does have its limitations but I think it would work in your case.

So instead of:

apps.tk-multi-workfiles2.location:
  entity_type: CustomNonProjectEntity03
  field: sg_uploaded_apps
  id: '1'
  type: shotgun
  version: '12469'

try using:

apps.tk-multi-workfiles2.location:
  entity_type: CustomNonProjectEntity03
  field: sg_uploaded_apps
  name: tk-multi-workfiles2   #This should match the "code" on your SG entity
  type: shotgun
  version: '12469'

and see if that solves your problem. You will need to do the same for your tk-nuke and tk-maya engine descriptors as well.

3 Likes

Nice. the only problem here is that if you want to use bundles elsewhere in the website config, you would end up with a list of identical named entities in any dropdown where you might want to select a specific release of a bundle. Because of that, I have to have a unique “code” for each bundle version released. I either use semver or the commit hash for this in the bundle code.

I think the bottom line here is that there needs to be a seperation of the “name” used by SG for forming template names, and the bundle name. Or, at least make it configurable… which is what my PR attempts to achieve.

1 Like

Thank you @Ehsan, it seems very nice, I will try it.

Thanks!

1 Like

Hi Patrick,

Could you elaborate a bit on what your workflow is and where in the process you are linking to the custom entity that the toolkit bundle is uploaded to? What is the entity that is referring to the toolkit bundle entity?

Thanks

1 Like

I’ve created a system whereby we distribute custom dcc configs as toolkit bundles. So I might have a nuke config in a repo called rsNuke, when I push an update, my CICD process zips the repo up and uploads it to a new toolkit bundle entity with the code <branch/tag><commit_short_hash>.

To assign this bundle across the studio, we have a specific software entity for this. Software entities have a “default bundles” field which allows you to pick tooklit bundle entities. On app_launch, we cache and run env setup for all the bundles in the studio default bundles field.
We can override the studio software bundles at software, and pipeline config levels, which both also have a field for specifying toolkit bundles.

It started off as a tk-app-lite setup… but in actual fact they are heading towards being fully-fledged tk-apps that are assigned via the SG website rather than via pipeline config ymls.
One other nice feature of this is that each toolkit bundle has its own set of hooks, so we can encapsulate any pre-app-launch dcc code here and then be able to call it quite easily in a deadline preJobLoad hook.

It’s a nice system, with the only major issue being that the lack of paper-trail in terms of how the bundles are assigned at any time for any user. At least with a fully file-based system, you might feasibly have a paper trail if you enforce version control. With it all being on the database, what you gain in convenience, you lose in robustness (it’s very easy to change fields on the website by accident)… One step I’ve taken to reduce this loss of robustness is to log every change of bundle in logfiles on the project… so if something goes horribly wrong, or you’re digging up a very old project, you’ll be able to see exactly which bundles were used at any point in time.

2 Likes