Install python packages for use in tk-config-default2

I am developing a Shotgrid pipeling configuration based on tk-config-default2. How can I use any python packges that are normally install via pip, e.g. the Pillow library?

1 Like

Iā€™m also interested. Whatā€™s the best method to use third party python packages in the config?

There are various ways.

You could hardcode and append the PYTHONPATH to look at a location on your network or inside the config.

I usually ship my configs with the dependencies and create a resources/python folder inside the config.

Then use the bootstrap.py core hook to append the PYTHONPATH there.

But this is very much up to how you want to manage things.
You could even run a command on each installation of SG desktop and have itā€™s shipped python executable use pip to install the libs.

2 Likes

@Ricardo_Musch Thank you very much for the quick response. All sounds good.
However since I have multiple projects with different, but overlapping and constantly changing dependency packages simultaneously, putting all the packages in the config wouldnā€™t be practical.
I was thinking about handling each package separately and only include them kind of like the apps or frameworks. So in a yaml Iā€™d declare the package names, versions and descriptors etc.
And when I want to upgrade those packages, I would just change the packageā€™s code, and change the version in the yaml, then SG would handle the config update on the client machines.
However the SG apps and frameworks doesnā€™t seem to fit this idea, or Iā€™m just not familiar enough with them yet. I canā€™t seem to find how to solve this properly.
What do you think?

In that case I would highly recommend using CPENV which is a Package Manager with excellent ShotGrid Integration.

Itā€™s also easily supported on Deadline Render Management Software.

tk-cpenv has a nice UI where you can assign packages to Projects (and specific software).

1 Like

@Ricardo_Musch Wow, thanks! This looks very promising! I see youā€™re one of the contributors, great job!
Currenty I use REZ for package management. If Iā€™d decide to use CPENV would it mean I have to manually transform and add all my packages to it?

Update: Iā€™ve read the readme-s, and Iā€™ve got a few question:

  1. Does tk-cpenv remove unused modules from the local machine?
  2. Does it download new modules on program start?
  3. What ā€˜activateā€™ does exactly?
  4. Youā€™ve mentioned Deadline but I havenā€™t found anything about that in the readme-s. Where can I find out more about that?

Hey @pw_simon ,

Well all of the heavy lifting was done by @Dan_Bradham, I use CPENV quite a bit and ocassionally help out to add features :slight_smile:

Yes, you would have to transform your REZ packages manually however the syntax for a CPENV package is pretty easy so perhaps there is a way to build a script to do it.

Iā€™m also pretty sure (but havent tested this) that CPENV could be used together with REZ, altough this may be slow and messy.

Reply to your questions:

1. Does tk-cpenv remove unused modules from the local machine?
Not currently however this would be a good feature to add.

2. Does it download new modules on program start?
Yes, it checks the module versions needed for the environment and downloads any that are not already locally cached. Modules can live in a Custom Entity on ShotGrid but also in filesystem folders, CPENV searches all available/set-up locations.

3. What ā€˜activateā€™ does exactly?
Activate will resolve all the requested packages, find them and then combine their environment setups to ā€œactivateā€ the environment and then launch a subshell with the resolved environment. (Its exactly what REZ does)

4. Youā€™ve mentioned Deadline but I havenā€™t found anything about that in the readme-s. Where can I find out more about that?
Here you goā€¦ the whole CPENV github has a few more repoā€™s.

2 Likes

Hey!

Thanks for promoting and explaining cpenv @Ricardo_Musch!

I wanted to add, depending on how many rez packages you have @pw_simon , it might be worth looking into the example hook below to integrate rez with your ShotGrid config. That might be preferable to recreating all of your packages as cpenv modules.

Also check out this related thread for more details on Rez support in SG and some background info on cpenv.

2 Likes

@Dan_Bradham and @Ricardo_Musch thanks putting this tool together, itā€™s pretty awesome and itā€™s going to be extremely useful, Iā€™ve been testing in past few days and I think itā€™s going to be make it very easy to deploy modules.

One question, it looks like the environments are DCC based, meaning if I have a standalone app that I launch from tk-shell or tk-desktop, it doesnā€™t seem I can attach an environment to it. Is that the correct understanding or did I not get it right.

Cheers

1 Like

Hi Ali,

indeed that is because tk-shell/tk-desktop apps donā€™t run via tk-multi-launchapp to launch so they donā€™t run the before_app_launch hook.

What you could do is add a before_app_launch hook to your custom app and add the snippet of code to launch tk-cpenv before you finish loading up your app.
I think that should work.

Thanks for the relay @Ricardo_Musch, but Iā€™m not sure if launching the cpenv is the issue. The cpenv app is running during the app launch but it doesnā€™t seem itā€™s activating any environment because of no engine. At least thatā€™s what Iā€™m gathering from my debugging, unless I misunderstood your comment above.

Iā€™ll continue digging into it and I might be able to hack it, but if anyone has a working example that could point me to, I would be really appreciate it.

Cheers

So you could run this bit of code when initialising your app.

        # Get the tk-cpenv app from the current engine.
        self.logger.info('Loading tk-cpenv...')
        tk_cpenv = self.parent.engine.apps.get('tk-cpenv')

        # Call the tk-cpenv's before_app_launch method.
        self.logger.info('Running tk-cpenv before_app_launch...')
        tk_cpenv.before_app_launch(
            app_path,
            app_args,
            version,
            engine_name,
            software_entity,
            **kwargs
        )

I believe with the latest update @Dan_Bradham added support for software that doesnā€™t have an engine.

See this for more info:

The latest release does support the Software.code field as @Ricardo_Musch mentioned. If weā€™re talking about launching software with a before_app_launch hook that should be working. If not, you may need to update the version in your app_locations.yml file? The latest release is v0.5.11 so your descriptor should look like the below.

apps.tk-cpenv.location:
  type: github_release
  organization: cpenv
  repository: tk-cpenv
  version: v0.5.11

Currently there is no mechanism in tk-cpenv of activating an Environment for tk-shell or tk-desktop engines. You could probably add that functionality within one of the core hooks of your configuration, maybe engine_init hook, but I donā€™t know how safe that actually is. For example: what happens when your config changes in tk-desktop, what would you expect to happen?

1 Like

What about a initialize hook for the custom app.

I would imagine usually the things you want to bundle are not needed just to open/register the app so could we run tk-cpenv in a custom before_app_launch/app_init hook?

literally right in the app_init?

I.e. right here?

You could do that, but it sort of breaks the toolkit configurationā€™s sandboxing. The reason SG has that import_module method is so that it can import python packages without polluting the global sys.modules object and this allows different versions of Apps to load in different configurations within the same Python session. Thatā€™s sort of an oversimplification of the import_module method but itā€™s an okay summary.

If you were to use cpenv to load an Environment there and then imported packages, they would end up in sys.modules. That ends up being a can of worms because who is responsible for cleaning up those modules and environment variables when the SG context changes and all apps are loaded again?

Itā€™s complicated territory and one of the reasons that cpenv and rez are used to provision an Environment for an application prior to launch and not during. Thatā€™s not to say itā€™s impossible to use these tools during runtime, just in practice it can make things very complicated if youā€™re not careful. Personally, I do have a set of python packages that I include in a python folder in our SG config. Those are special dependencies that I know arenā€™t changing much over time or at all and I add the configā€™s python folder to sys.path so I can import those modules in SG config hooks and other places they are needed.

For individual SGTK apps that Iā€™ve written, I tend to vendor any python dependencies they require directly in the appā€™s python folder and use the import_module method. That makes my apps atomic, and I know they are self-contained. tk-cpenv itself vendors cpenv in this way if youā€™d like to take a look at a concrete example.

1 Like