Why do publishes require path cache entries in tk-multi-workfiles2?

Pipeline Integrations
1

Our tools do a lot of publishing by copying files to their locations and then using the TK api register_publish() command. However, when we publish the first iteration for an entity, and a user opens the File Manager in their DCC to open the publish, it doesn’t show up and they’re met with an error message because there’s no path cache entries for the entity yet.

We could run generate_filesystem_locations() in our tools to solve this, however there are multiple problems with this route for us:

  • Our tools use threads and the TK Shotgun API isn’t thread-safe (we have our own thread-safe version).
  • This could be a big performance hit since our tools may publish a lot of files at once.
  • Folders and path cache entries may not need to be created for some of the publishes either because they already exist or the config doesn’t use a schema (our tools are used by several clients with different needs).

I know the file manager takes care of creating folders when you create a new file or open an existing one. So it makes sense that it can’t see work files if there’s no path cache entries to rely on to match things up. However, PublishedFiles are in Shotgun and have all of the info needed to make this association. You click on your Task in the File Manager, and it should be able to look up in Shotgun that there are PublishedFiles for that Task and display them. Then if you choose to work on that file, it would do the folder creation process necessary.

  • Why can’t tk-multi-workfiles see publishes just from their records in Shotgun without a path cache entry?
  • Is there a hook in tk-multi-workfiles where we could check and run generate_filesystem_locations() if needed to work around this issue? I didn’t see anything reasonable for this but am putting it out there in case anyone has insight I missed.
6 Likes
2

Hey @kporangehat!

Yeah, it’s a fair question, and I think the answer is, that it is historically like that.

If we were to rewrite the app today, then it’s possible that we wouldn’t require the folders/templates for PublishedFiles. Arguably it would be useful to be able to match a PublishedFile path to a template so that we can extract field data if, for example, we want to create a workfile from it, but I suspect fallback methods could be implemented in the situation where a path doesn’t match a template.

I don’t think there is a way you could relax this restriction via hooks, I think you would need to take over the app. I also don’t think there is a way to create the folder structure automatically via a workfiles hook.
It is expected that if a file exists then Toolkit has already created the folders for it. It seems like it would be better to try and overcome the issues upstream at file creation.

Perhaps you could add an open action to the loader, which would create the folder structure when you opened the file?

Maybe pop a suggestion into our roadmap.

1 Like
3

Hey @kporangehat , I’m wondering which method you used to deal with this issue, if you have any thoughts on the most efficient solution.

4

Hey @stavernia

I modified the filter_publishes.py hook in tk-multi-workfiles2 to check for any publishes that don’t have folders associated (path cache entries) and then generated them on the fly. It’s not ideal but it has been working very well in production for a while.

Happy to take any improvements or hear thoughts on this workaround approach!

cheers,
kp