Folder creation optimization

I’m starting a long overdue investigation of our shotgun configuration to optimize the folder creation time which I feel is quite slow.

FYI In our shotgun setup we have two root locations, primary and secondary

I’m testing with these commands :

from pprint import pprint
import time
import sgtk
tk = sgtk.sgtk_from_entity("Shot", 99999)
start = time.perf_counter()
folders_list = tk.preview_filesystem_structure("Shot", [99999])
stop = time.perf_counter()
print("Duration in sec:{}".format(stop-start))

I get a duration of about 20sec for the preview_filesystem_structure (btw when using the create_filesystem_structure, I also have a duration of about 20s)

722 folders/files are listed

I read the docs : and consequently made yml files for the static folders containg :

# the type of dynamic content
type: "static"

# defer creation
defer_creation: ["tk-shell", "tk-shotgun"]

I suppose with these modifications, those static folders will still be created upon project creation (using the Advanced Project setup) and/or when doing a ‘Create Folders’ on the website, right ?

Running the script again, this reduced the number of items to 378. I almost halved the number of folders, but unfortunately this only reduced the duration from 20s to 18.5sec. Is it normal that the duration doesn’t drop more ?

I would also like to defer creation of some static files (luts that I want to get copied on project creation, but ignored afterwards for example). I did not see how to defer the creation of individual files in the docs. I can’t defer the creation of the folder containing those static files because that folder also contains folders that references shotgun entities.

Most puzzling to me : I observe that a lot of folders in the ‘folders_list’ are repeated multiple times. For example some of those luts files are repeated 8 times. If I do a python ‘set’ on the list, it shows me that there are only 109 individual folders/files. Why do I have so many repeating items ?

(Note : I have not a single create_with_parent: true in the whole schema)
Any tips appreciated



I do not have any information about the issue here. Just posting to follow this thread, since I am very interested in the results as well.
The one thing I know that there seems to be a big gap between operating systems. For some reason Windows always seemed way slower for me then Linux. On the other hand that could have been server/network setup related. Hope you’ll get some answers!

1 Like


@Fabian if you (or any others following this thread, the more the merrier) have some time, could you run my little bit of code ? I would particularly be interested to know if you too have lots of duplicates in the list of folders



1 Like

Hi @donat !

I tried your code on Linux and Windows and in both cases it takes 2-3 secs and I do not have duplicates in the returned folders. This is manageable for us, but I have definitely seen your behaviour as well a couple of years ago.


Hi @Fabian
Thanks for testing this. Lots of room for improvement on my side then! I submitted a ticket to support.

1 Like

Hello folks!

I did some profiling to preview_filesystem_structure function of tk-core.

Unfortunately, I wasn’t able to decrease the processing time and the main reason is that by design, the logic performs several HTTP requests (using python-api) which creates latency in the whole operation.

There are 3 requests that I’ve inspected they’re being made several times with the exact same set of input arguments:

And of course, IO operations are expensive causing an impact on the overall performance when having a large number of directories to be created. Caching is not straightforward at the moment because the input arguments are not immutable (not hashable). But this opens an exciting opportunity for improvement.


Nice one, thanks Carlos, its great to see some official involvement in this :slight_smile:

1 Like