[tech question] Write shrunken file vs converting file to shrink it

Slightly offtopic but it’s a question in relation to publishing large amounts of exr files (or any file really).

I was thinking of doing the following:

  1. Copy a sequence of exr files received from client to a new location (Let’s say these are 3GB total, 70mb per exr)
  2. Compress the EXR files to DWAA-15, (let’s say this reduced the total space to 1GB, 12mb per exr)

I am now wondering if this creates heaps of fragmentation in the sectors (we use a Pixtor) on the raid.
As I copied the file it allocates 70mb for each exr, but when I run the compression it reduces the filesize to 12mb. That means the sectors that where just recently allocated to store the file from 13mb-70mb are now free (but ads fragmentation).

I wonder if it’s better to do this:

  1. Original sequence
  2. Load original exr, compress and save in new location

Does anyone have any knowledge of how this sort of stuff works in regards to raids and SAN’s (striped data)?

2 Likes

Hey @Ricardo_Musch,

I don’t know much about SAN’s and Raids striping effects, but just a quick question, are you overwriting the original .exr with the DWAA compressed version?
Just asking because it’s a lossy compression method and would go agains the non-destructive workflow, with my producer hat on I would say create the compressed .exr in a new location and treat it like a proxy until final render.

Hoping someone else has info about fragmentation :slight_smile:

1 Like

Hiya,

no and yes.
We keep the original source package we receive from the client, and we ingest that media (copy) it to the shot/asset to work from.
(So we keep original and also retranscode it to work from)

A lot of VFX studios have switched to a DWAA workflow.
If you set the compression to 15 the difference is so negligible that it can be ignored as rounding error. If it’s good enoiugh for Dreamworks and ILM it’s good enough for us :wink:

Believe me, with my Comp hat on, the change is very acceptable to me and we haven’t had issues with DI on various shows yet :slight_smile:

The nice thing about dwaa is that it’s not compression like h264 or any video compression.
Even the grain stays intact! :slight_smile: