Install python packages for use in tk-config-default2

I am developing a Shotgrid pipeling configuration based on tk-config-default2. How can I use any python packges that are normally install via pip, e.g. the Pillow library?

1 Like

I’m also interested. What’s the best method to use third party python packages in the config?

There are various ways.

You could hardcode and append the PYTHONPATH to look at a location on your network or inside the config.

I usually ship my configs with the dependencies and create a resources/python folder inside the config.

Then use the bootstrap.py core hook to append the PYTHONPATH there.

But this is very much up to how you want to manage things.
You could even run a command on each installation of SG desktop and have it’s shipped python executable use pip to install the libs.

1 Like

@Ricardo_Musch Thank you very much for the quick response. All sounds good.
However since I have multiple projects with different, but overlapping and constantly changing dependency packages simultaneously, putting all the packages in the config wouldn’t be practical.
I was thinking about handling each package separately and only include them kind of like the apps or frameworks. So in a yaml I’d declare the package names, versions and descriptors etc.
And when I want to upgrade those packages, I would just change the package’s code, and change the version in the yaml, then SG would handle the config update on the client machines.
However the SG apps and frameworks doesn’t seem to fit this idea, or I’m just not familiar enough with them yet. I can’t seem to find how to solve this properly.
What do you think?

In that case I would highly recommend using CPENV which is a Package Manager with excellent ShotGrid Integration.

It’s also easily supported on Deadline Render Management Software.

tk-cpenv has a nice UI where you can assign packages to Projects (and specific software).

@Ricardo_Musch Wow, thanks! This looks very promising! I see you’re one of the contributors, great job!
Currenty I use REZ for package management. If I’d decide to use CPENV would it mean I have to manually transform and add all my packages to it?

Update: I’ve read the readme-s, and I’ve got a few question:

  1. Does tk-cpenv remove unused modules from the local machine?
  2. Does it download new modules on program start?
  3. What ‘activate’ does exactly?
  4. You’ve mentioned Deadline but I haven’t found anything about that in the readme-s. Where can I find out more about that?

Hey @pw_simon ,

Well all of the heavy lifting was done by @Dan_Bradham, I use CPENV quite a bit and ocassionally help out to add features :slight_smile:

Yes, you would have to transform your REZ packages manually however the syntax for a CPENV package is pretty easy so perhaps there is a way to build a script to do it.

I’m also pretty sure (but havent tested this) that CPENV could be used together with REZ, altough this may be slow and messy.

Reply to your questions:

1. Does tk-cpenv remove unused modules from the local machine?
Not currently however this would be a good feature to add.

2. Does it download new modules on program start?
Yes, it checks the module versions needed for the environment and downloads any that are not already locally cached. Modules can live in a Custom Entity on ShotGrid but also in filesystem folders, CPENV searches all available/set-up locations.

3. What ‘activate’ does exactly?
Activate will resolve all the requested packages, find them and then combine their environment setups to “activate” the environment and then launch a subshell with the resolved environment. (Its exactly what REZ does)

4. You’ve mentioned Deadline but I haven’t found anything about that in the readme-s. Where can I find out more about that?
Here you go… the whole CPENV github has a few more repo’s.

2 Likes

Hey!

Thanks for promoting and explaining cpenv @Ricardo_Musch!

I wanted to add, depending on how many rez packages you have @pw_simon , it might be worth looking into the example hook below to integrate rez with your ShotGrid config. That might be preferable to recreating all of your packages as cpenv modules.

Also check out this related thread for more details on Rez support in SG and some background info on cpenv.

2 Likes