Install python packages for use in tk-config-default2

shapeare · January 21, 2022, 9:19am

I am developing a Shotgrid pipeling configuration based on tk-config-default2. How can I use any python packges that are normally install via pip, e.g. the Pillow library?

pw_simon · August 3, 2022, 2:15pm

I’m also interested. What’s the best method to use third party python packages in the config?

Ricardo_Musch · August 3, 2022, 2:41pm

There are various ways.

You could hardcode and append the PYTHONPATH to look at a location on your network or inside the config.

I usually ship my configs with the dependencies and create a resources/python folder inside the config.

Then use the bootstrap.py core hook to append the PYTHONPATH there.

But this is very much up to how you want to manage things.
You could even run a command on each installation of SG desktop and have it’s shipped python executable use pip to install the libs.

pw_simon · August 3, 2022, 3:07pm

@Ricardo_Musch Thank you very much for the quick response. All sounds good.
However since I have multiple projects with different, but overlapping and constantly changing dependency packages simultaneously, putting all the packages in the config wouldn’t be practical.
I was thinking about handling each package separately and only include them kind of like the apps or frameworks. So in a yaml I’d declare the package names, versions and descriptors etc.
And when I want to upgrade those packages, I would just change the package’s code, and change the version in the yaml, then SG would handle the config update on the client machines.
However the SG apps and frameworks doesn’t seem to fit this idea, or I’m just not familiar enough with them yet. I can’t seem to find how to solve this properly.
What do you think?

Ricardo_Musch · August 3, 2022, 8:36pm

In that case I would highly recommend using CPENV which is a Package Manager with excellent ShotGrid Integration.

It’s also easily supported on Deadline Render Management Software.

tk-cpenv has a nice UI where you can assign packages to Projects (and specific software).

pw_simon · August 4, 2022, 7:19am

@Ricardo_Musch Wow, thanks! This looks very promising! I see you’re one of the contributors, great job!
Currenty I use REZ for package management. If I’d decide to use CPENV would it mean I have to manually transform and add all my packages to it?

Update: I’ve read the readme-s, and I’ve got a few question:

Does tk-cpenv remove unused modules from the local machine?
Does it download new modules on program start?
What ‘activate’ does exactly?
You’ve mentioned Deadline but I haven’t found anything about that in the readme-s. Where can I find out more about that?

Ricardo_Musch · August 4, 2022, 9:37am

Hey @pw_simon ,

Well all of the heavy lifting was done by @Dan_Bradham, I use CPENV quite a bit and ocassionally help out to add features

Yes, you would have to transform your REZ packages manually however the syntax for a CPENV package is pretty easy so perhaps there is a way to build a script to do it.

I’m also pretty sure (but havent tested this) that CPENV could be used together with REZ, altough this may be slow and messy.

Reply to your questions:

1. Does tk-cpenv remove unused modules from the local machine?
Not currently however this would be a good feature to add.

2. Does it download new modules on program start?
Yes, it checks the module versions needed for the environment and downloads any that are not already locally cached. Modules can live in a Custom Entity on ShotGrid but also in filesystem folders, CPENV searches all available/set-up locations.

3. What ‘activate’ does exactly?
Activate will resolve all the requested packages, find them and then combine their environment setups to “activate” the environment and then launch a subshell with the resolved environment. (Its exactly what REZ does)

4. You’ve mentioned Deadline but I haven’t found anything about that in the readme-s. Where can I find out more about that?
Here you go… the whole CPENV github has a few more repo’s.

Dan_Bradham · August 5, 2022, 3:44am

Hey!

Thanks for promoting and explaining cpenv @Ricardo_Musch!

I wanted to add, depending on how many rez packages you have @pw_simon , it might be worth looking into the example hook below to integrate rez with your ShotGrid config. That might be preferable to recreating all of your packages as cpenv modules.

Also check out this related thread for more details on Rez support in SG and some background info on cpenv.

ali-jafargholi · October 9, 2022, 7:23am

@Dan_Bradham and @Ricardo_Musch thanks putting this tool together, it’s pretty awesome and it’s going to be extremely useful, I’ve been testing in past few days and I think it’s going to be make it very easy to deploy modules.

One question, it looks like the environments are DCC based, meaning if I have a standalone app that I launch from tk-shell or tk-desktop, it doesn’t seem I can attach an environment to it. Is that the correct understanding or did I not get it right.

Cheers

Ricardo_Musch · October 9, 2022, 6:55pm

Hi Ali,

indeed that is because tk-shell/tk-desktop apps don’t run via tk-multi-launchapp to launch so they don’t run the before_app_launch hook.

What you could do is add a before_app_launch hook to your custom app and add the snippet of code to launch tk-cpenv before you finish loading up your app.
I think that should work.

ali-jafargholi · October 9, 2022, 9:36pm

Thanks for the relay @Ricardo_Musch, but I’m not sure if launching the cpenv is the issue. The cpenv app is running during the app launch but it doesn’t seem it’s activating any environment because of no engine. At least that’s what I’m gathering from my debugging, unless I misunderstood your comment above.

I’ll continue digging into it and I might be able to hack it, but if anyone has a working example that could point me to, I would be really appreciate it.

Cheers

Ricardo_Musch · October 9, 2022, 11:28pm

So you could run this bit of code when initialising your app.

        # Get the tk-cpenv app from the current engine.
        self.logger.info('Loading tk-cpenv...')
        tk_cpenv = self.parent.engine.apps.get('tk-cpenv')

        # Call the tk-cpenv's before_app_launch method.
        self.logger.info('Running tk-cpenv before_app_launch...')
        tk_cpenv.before_app_launch(
            app_path,
            app_args,
            version,
            engine_name,
            software_entity,
            **kwargs
        )

I believe with the latest update @Dan_Bradham added support for software that doesn’t have an engine.

See this for more info:

Dan_Bradham · October 10, 2022, 2:00pm

The latest release does support the Software.code field as @Ricardo_Musch mentioned. If we’re talking about launching software with a before_app_launch hook that should be working. If not, you may need to update the version in your app_locations.yml file? The latest release is v0.5.11 so your descriptor should look like the below.

apps.tk-cpenv.location:
  type: github_release
  organization: cpenv
  repository: tk-cpenv
  version: v0.5.11

Currently there is no mechanism in tk-cpenv of activating an Environment for tk-shell or tk-desktop engines. You could probably add that functionality within one of the core hooks of your configuration, maybe engine_init hook, but I don’t know how safe that actually is. For example: what happens when your config changes in tk-desktop, what would you expect to happen?

Ricardo_Musch · October 10, 2022, 3:37pm

What about a initialize hook for the custom app.

I would imagine usually the things you want to bundle are not needed just to open/register the app so could we run tk-cpenv in a custom before_app_launch/app_init hook?

literally right in the app_init?

I.e. right here?

github.com

shotgunsoftware/tk-multi-starterapp/blob/4868f16290474ad32bbada6ba0f750e27e95f13c/app.py#L25


      
          class StgkStarterApp(Application):
              """
              The app entry point. This class is responsible for initializing and tearing down
              the application, handle menu registration etc.
              """
          
          
    def init_app(self):
                  """
                  Called as the application is being initialized
                  """
          
          
        # first, we use the special import_module command to access the app module
                  # that resides inside the python folder in the app. This is where the actual UI
                  # and business logic of the app is kept. By using the import_module command,
                  # toolkit's code reload mechanism will work properly.
                  app_payload = self.import_module("app")
          
          
        # now register a *command*, which is normally a menu entry of some kind on a Shotgun
                  # menu (but it depends on the engine). The engine will manage this command and
                  # whenever the user requests the command, it will call out to the callback.

Dan_Bradham · October 10, 2022, 4:02pm

You could do that, but it sort of breaks the toolkit configuration’s sandboxing. The reason SG has that import_module method is so that it can import python packages without polluting the global sys.modules object and this allows different versions of Apps to load in different configurations within the same Python session. That’s sort of an oversimplification of the import_module method but it’s an okay summary.

If you were to use cpenv to load an Environment there and then imported packages, they would end up in sys.modules. That ends up being a can of worms because who is responsible for cleaning up those modules and environment variables when the SG context changes and all apps are loaded again?

It’s complicated territory and one of the reasons that cpenv and rez are used to provision an Environment for an application prior to launch and not during. That’s not to say it’s impossible to use these tools during runtime, just in practice it can make things very complicated if you’re not careful. Personally, I do have a set of python packages that I include in a python folder in our SG config. Those are special dependencies that I know aren’t changing much over time or at all and I add the config’s python folder to sys.path so I can import those modules in SG config hooks and other places they are needed.

For individual SGTK apps that I’ve written, I tend to vendor any python dependencies they require directly in the app’s python folder and use the import_module method. That makes my apps atomic, and I know they are self-contained. tk-cpenv itself vendors cpenv in this way if you’d like to take a look at a concrete example.

Topic		Replies	Views
Setting Python 3 and distributed TK configs	3	329	November 6, 2021
What's the best way to include pip dependencies in a SGTK app/framework? Pipeline Integrations fptr-tk	6	809	April 20, 2021
Importing third party modules in toolkit bundle Pipeline Integrations	5	816	May 29, 2023
Studio backend Pipeline Integrations python-api , fptr-tk	6	1488	April 3, 2020
Path Generator Pipeline Integrations python-api , fptr-tk	2	729	June 30, 2021

Install python packages for use in tk-config-default2

Related topics