Shotgun Desktop - Loading project with multiple git descriptor defined configurations

[koaleksa]

Hi,

I’ve noticed a few things when using git based descriptors, and you click on the project in Shotgun Desktop:

• If the version is not specified (tag or commit) pipeline configuration is cloned twice.
• When loading the project, configuration is cloned for each defined configuration (even though, only one is used).

I guess my question is why does it need to clone it twice and for each config if its only loading one?

–Aleksandar

[tommy.kiser]

Good question Aleks! Will flag this for our Toolkit experts to follow up on next week. Have a great weekend!

[jing.liu]

Hi @koaleksa,

For one project, it just loads one configure. Usually, a cloned configure is for test and developer. Do you read our document which make you feel confused?

Thanks

Loney

[koaleksa]

Hi @jing.liu,

Let me explain in a little bit more detail:

Firstly, I’m strictly talking about pipeline configuration entities. And the cloning I’m referring to is the one that gets the git repository from the remote location to temp folder (on windows it would be something like: c:\users\<username>\appdata\local\temp\sgtk_clone_a1f3fd3a9fe54935913c263f5046cb55)

We use descriptors to set desired config source for each project. The project will usually have one “Dev” pipeline config and “Primary” config. Both will have their descriptor fields set as needed. We may have more than that though (depends on multiple factors really).

For now, i think we can ignore the Dev (sandbox) config, and focus on Primary.

This config will have a descriptor field set in shotgun as follows:

sgtk:descriptor:git_branch?path=</path/to/repo>&branch=<branch_name>
&version=<commit_hash>

For the whole picture’ sake, repository is not on the internet but rather on the network location. Don’t think it makes any difference, but worth noting.

Ok, back to the problem:

Situation number 1

We define descriptor exactly as mentioned above and the specified repo/branch/commit was already cached:

> no cloning captured in the log

Situation number 2

Define descriptor as above but the commit was never cached locally:

> git clone --no-hardlinks... captured once

Situation number 3

Omit the version from the descriptor (this forces it to use the latest commit):

> the log now shows git clone --no-hardlinks... is executed 4 times!

For each pipeline configuration added on shotgun website for the project, this number grows by 2.

It does that every time the project is loaded, no mater if it was previously cached - which i can understand since it doesn’t know whats “latest” at that point - but it makes little sense to me that it needs to clone the same repository so many times.

Note

I can send the logs privately if needed.

I’ve only caught this because I was investigating (once again) slow startup times and realized that this action takes very long and it is executed multiple times for some projects.

The time it takes per single clone is nothing to do with shotgun - the repo sits on filesystem and was never garbage collected so it got quite dirty over the course of 2 years. This can be improved but that’s on our end. I was just wandering if the optimization can be taken even further.

Of course, we could always provide the exact version to the descriptor but sometimes its useful to have the option of “latest”.

The reason it does this repeated cloning could easily be a result of something I introduced unwittingly. I just can’t figure out what.

Cheers,
Aleksandar

[tannaz]

Hi Aleksandar –

Thanks for laying all that out so clearly! It looks like this is a known issue; there’s a bit more explanation in this ticket about what’s going on:

Distributed git configs - why does it clone the repo multiple times?

So my theory was correct, that not providing a commit id, means that it will always attempt to download it. As for why it is happening twice, it’s not clear without investigating, but at a guess, we think it was probably written in a way where at any point where it requires the bundle, it will ensure it is local. Since usually, it would only download if it wasn’t found locally so it would typically be a very quick check, so not a problem. I appreciate that doesn’t fully answer the question, an…

At any rate, I’ve put in an internal ticket to track the bug (or at least… unoptimization). I’ve linked this post to the internal ticket, so I’ll be notified when it’s resolved, and will notify you here in turn.

[koaleksa]

Thanks Tannaz,

That’s the exact same problem and it seems I was too lazy to search the forum before creating a new thread.

The added caveat is that it’s not only twice but twice per linked configuration. I’ve got the idea why this could be happening and it probably has to do with the browser integration and right click menus (of all things…)

Unlike the experience in Shotgun Desktop where when you click on the project you are presented with the actions for a single configuration and the dropdown menu to change the config, on the browser side it needs to load actions for all configs the user has access to (for the given project) because they are all presented to the user.

As far as I understand, the procedure to initiate project is the same for both so that could very well be the reason it is caching them all.

Combined with the fact that it needs to do it as many times for the repo that’s quite heavy to clone, our load time was massively under stress.

Glad this was cleared out and I hope the team does something to optimize the number of clone actions needed when possible.

Cheers,
Aleksandar