Change sgtk file and folder creation methods- overwrite tank core?

sogrday · September 11, 2020, 8:02am

Hi,

we are looking to use our own python module whenever shotgun is creating folders or files.
Think of it as a custom python os module.

There are tk-apps like tk-shotgun-folders that are configurable with hooks.
However we like to make a change that affect all other tk-apps as well.

The best place to do a modification like that seems to be in
<path_to_sg_config>/install/core/python/tank/util/filesystem.py where the function such as ensure_folder_exists and .touch_file could be overwritten.

Essentially that is a modification to the tank module and with every future tank core update this modification would be undone. So I am really expecting that it is not a recommended thing to do.

Which leads to the questions:
1.) Is there a better way to change file and folder creation that is used by through out all of sgtk ?
(similar to filesystem.py)
2.) Is there an easy way to make this change configurable ?
(Not having to change anything in install/core)

DavidMason · September 18, 2020, 10:55am

Hi @sogrday,

I don’t think you can configure without branching but I’ll check with the TK team to know more.

-David

philip.scadding · September 18, 2020, 11:08am

Hey!
The process folder creation core hook will be called when processing the project folder creation and any files that that might entail:

github.com

shotgunsoftware/tk-core/blob/v0.19.16/hooks/process_folder_creation.py

# Copyright (c) 2018 Shotgun Software Inc.
#
# CONFIDENTIAL AND PROPRIETARY
#
# This work is provided "AS IS" and subject to the Shotgun Pipeline Toolkit
# Source Code License included in this distribution package. See LICENSE.
# By accessing, using, copying or modifying this work you indicate your
# agreement to the Shotgun Pipeline Toolkit Source Code License. All rights
# not expressly granted therein are reserved by Shotgun Software Inc.

"""
This hook is invoked during folder creation when :meth:`sgtk.Sgtk.create_filesystem_structure` is
called.
"""

from tank import Hook
import os
import sys
import shutil
from tank_vendor import six

This file has been truncated. show original

However, the method you have called out, certainly doesn’t call this, I think we use that for any time we need to create say configuration folders, such as the bundle cache or things like that.

If you only need control over the project folder and file creation then the core hook will be what you need, but if you need to control literally everything that we ever create on disk, then you are going to have to start making changes. At that point, I wonder if there isn’t a better approach, that can be taken, maybe even patching things at a lower level like os, but that could also lead to unexpected problems.

sogrday · September 21, 2020, 11:10am

Hey @philip.scadding

thanks for the prompt reply.

We definitely do not want to modify the python os module, or use it’s namespace.

sgtk.Sgtk.create_filesystem_structure seems like a good example and kind of what I was looking for.

1.) First, however compared to filesystem.ensure_folder_exists that hook is not used much throughout the tk-apps. Instead filesystem.ensure_folder_exists seems to be the favorite call in a lot of tk-apps. Meaning that replacing these call is quite a bit of work and code to replace in your well working number of tk-apps right ?

2.) Secondly, I am wondering where and how you would configure such a core hook.?
For example there is https://github.com/shotgunsoftware/tk-config-default2/blob/master/core/hooks/pick_environment.py but I am looking for the .yml file which contains something like this

...
setting.core.hooks:
   pick_environment: {self}/pick_cnvironment.py:{config}/pick_environment.py
...

Thanks !
Enno

philip.scadding · September 21, 2020, 12:23pm

Whilst the apps generally require that the folder structure was created with a call like sgtk.Sgtk.create_filesystem_structure, sometimes templates could include folders that aren’t actually in the schema, and so the apps try to ensure that the folders exist.

I would recommend putting a suggestion in on our roadmap page for that logic to be moved over into a hook. But for now if you need to manage all instances of of folder and file creation then you will have to take things over unfortunatly.

Core hooks unlike app hooks don’t require any YAML setup, you just copy them from tk-core over to your config/core/hooks folder. This is because core hook will run regardless of the environment so it doesn’t get configured in the env YAML files…

sogrday · September 21, 2020, 1:47pm

Great thank you,

In that case, could you point me to the file that’s contain the logic of inheriting and running from install/core/hooks to config/core/hooks ?

Thanks @philip.scadding

philip.scadding · September 22, 2020, 7:12am

Sure! Here is the place where it decides whether to run the hook from tk-core install or from your config:

github.com

shotgunsoftware/tk-core/blob/358beceb14f50ce58f97fc69d1e3e22f87619ae3/python/tank/pipelineconfig.py#L1208-L1218


if not os.path.exists(hook_path):
    # no custom hook detected in the pipeline configuration
    # fall back on the hooks that come with the currently running version
    # of the core API.
    hooks_path = os.path.abspath(
        os.path.join(os.path.dirname(__file__), "..", "..", "hooks")
    )
    hook_path = os.path.join(hooks_path, file_name)

try:
    return_value = hook.execute_hook(hook_path, parent, **kwargs)

There isn’t any inheritance here though, when you take over the hook it just runs your version.