A bit of context to the above (a bit click-baity, I’m sorry!). We are having severe performance issues with Maya 2019.2 that are largely intermittent and proving very hard to pin down. However one of the things that seems to be emerging in our testing is a relationship between the poor performance and sessions of Maya launched via Shotgun desktop.
This is something that we aren’t seeing on all machines, so it seems unlikely to be something related to shotgun. However we were wondering if it’s possible that during normal Maya operation (like setting a keyframe, scrubbing the timeline, or rotating the camera) that SGTK is “doing stuff” in the background. While we’re not expecting that SGTK would be triggered by these actions, does SGTK periodically check a connection to the database or perform some authentication check that might cause Maya to pause briefly and thus interrupt basic operations in Maya? If this is the case then it might be worth us investigating further any kind of firewall or network settings on that machine to see if they are somehow stifling SGTK’s ability to perform these checks
For further context we have tried a number of things that we’ve been able to rule out as the cause of the performance issues including:
Hey @tom-fw !
The times that our Maya integration will do stuff in the background are as follows:
If you launch an app like workfiles or loader, then it will spawn off a thread to go and fetch the data from Shotgun in the background. This shouldn’t stay active after completion though I don’t think.
The Shotgun panel app also uses threads to retrieve data in the background, but I don’t think it polls shotgun for data and if it did it should again be in the background so shouldn’t impact the main thread performance.
We create a callback for automatic context switching to detect new, opening, and saving of files. You can try disabling the automatic_context_switch setting in the app settings to disable this.
As far as I’m aware we don’t poll Shotgun for updates, and we pretty much react to user driven events such as launching or interacting with an app. I’ll mark this to run by the rest of the team though in case there is something I’ve missed.
We have seen background threading issues in the past in Maya on linux where they caused an impact on the main thread, but they were solved. I’ve not heard of any other issues recently.
To be honest this is pretty much what I thought would be the case and all the examples you gave are things that I’d expect. Our artists don’t really see that any of the sgtk apps are really slow, it’s more about the performance of the software once files are open.
It would be great if you could do some test or checks (and maybe help us do the same kind of thing at our end?) just in case maybe some thread or event isn’t been closed or triggered properly that might cause some kind of interference.
We’ll continue investigating things at our end, but we look forward to hearing about the results of your tests
We’re going to keep talking about this one internally, but we had a few questions to consider as you carry out your investigation on your end:
Just to set a baseline, what is the behavior/performance like without Shotgun? Is it consistently faster or do you ever see these slowdowns when you launch from Desktop?
It might be worth checking the environment variables in both conditions – maybe something ending up different around Qt? Does your studio use the before_app_launch to set any environment variables?
One other test you can try out, to rule out background processes, but still use Shotgun:
Hi @tannaz I’m also trying to investigate this issue we’re having with Tom. Sorry for the late reply we got swamped with a lot of other stuff that was going on but we were able to do some tests.
Some numbers for a comparison:
Normal experience:
Scene open time - 2 and a half minutes
Maya fps - roughly 20fps
Slowdown experience:
Scene open time - over 10 minutes
Maya fps - less than 5 fps
This is with the same Maya Preferences and Shotgun Config.
Just to set a baseline, what is the behavior/performance like without Shotgun? Is it consistently faster or do you ever see these slowdowns when you launch from Desktop?
We see the above speeds when experiencing a slowdown within Shotgun. We can have artists work off pipe without Shotgun and we consistently see the improved speeds. Interestingly after working off pipe for 30 minutes to an hour we’re able to have artists return to working on pipe with shotgun and see the faster speeds.
Currently our fix for this issue is to have artists work off pipe for around an hour and then go back to working on pipe, obviously this isn’t ideal, especially if they have deadlines and need to publish.
It might be worth checking the environment variables in both conditions – maybe something ending up different around Qt? Does your studio use the before_app_launch to set any environment variables?
We do use before_app_launch quite extensively to setup things like our 3rd party plugins, arnold variables, colour space, etc. But I don’t believe we set anything that would influence QT.
We still have some further tests to do around the automatic_context_switch and trying a maya file open but from within a shotgun desktop launched maya.
Frustratingly we’ve recently seen a handful of artists where going “off pipe” is not a fix. The only thing that seems to work for those artists is time. Though there are more artists where off pipe is the fix vs those where it isn’t.
We’ve also tried a test using Maya’s File Open but from a shotgun desktop launched Maya and the results were the same. Admittedly, this was with a user where going off pipe did not resolve their issue so I’ll try testing it with a few more artists when I can.
What does your storage situation look like?
If you have multiple storage nodes, could there be an issue with one of them therefore only impacting a certain amount of users?
I remember on our Pixtor one of the storage nodes was running low on swap and only a few artists had issues while the rest of the traffic was being fine on the other node.
If not, is there anything further to deduce that is similar?
Could artists for example have Nuke open in the background to see if at the moment of slowdown Nuke also freezes?
