Distributed config with path descriptor still used from network drive

Pipeline Integrations
1

Hello there!

We’re usign a distributed SG pipeline configuration. The idea was, to store this configuration on a network drive, and on the website, add a path descriptor that points to this path.
So in theory the workstations’ SG launchers download the pipeline config, because it is distributed, and they run it locally. When this config on the network drive changes, on the next config loading the SG launcher recognizes the change, and updates the local copy of the configuration.

This is not how this works though. It seems like the path descriptor makes the SG launcher load things from the network, causing significantly longer loading times.

2023-07-14 14:08:23,165 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\asset.yml
2023-07-14 14:08:26,027 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\asset_step.yml
2023-07-14 14:08:27,940 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\playlist.yml
2023-07-14 14:08:28,203 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\project.yml
2023-07-14 14:08:29,977 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\publishedfile.yml
2023-07-14 14:08:30,097 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\sequence.yml
2023-07-14 14:08:31,139 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\sequence_step.yml
2023-07-14 14:08:31,294 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\shot.yml
2023-07-14 14:08:32,770 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\shot_step.yml
2023-07-14 14:08:34,117 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\site.yml
2023-07-14 14:08:34,238 [ DEBUG] [PROXY] Loading environment data from path: \network_path\env\version.yml

Also it gets stuck for a solid 15-20 sec on Invoking pre engine start callback '>'. Even avoiding this would be a huge boost.

Did I misinterpret the Distributed configuration? I thought it creates a copy of the config locally, so it doesn’t have to reach for a central path.

I want to have a central source, but that should be downloaded only upon changes, and loaded from the local copy.
Any ideas how could I achieve this?

(I know if I would upload the config as a zip, it would work, however I want to avoid doing this to every project upon Pipeline configuration changes.)

best regards!

2

I think with a path descriptor, it only caches the pointer to where the package lives, eg on your network drive.
You could implement a simple github action or similar ci/cd process to automate the process of zipping the config and uploading it when you merge a PR.

2 Likes
3

+1

Zip the config with a github action and upload the zip as update.

1 Like
4

@Patrick @Ricardo_Musch
Thank you for your answers!
I thought I could get around it with the path descriptor.
I’ll have to develop an automated solution for this then.

Thanks again!

1 Like
5

This is the script you’ll want to use in your automatiom.

It can package up the toolkit config, download all required git packages and zip them up for you.

1 Like
6

Thank you, this is a good start.

1 Like