That is correct, and after quick look at the code, I think it is hardcoded that way, so associations to entities on other fields would not be considered.
Looking at the code I also don’t think that is possible to extract. It sounds like your use case is not something that was probably considered when this was written. If you were passed some PublishedFiles then I suppose you could try and figure out the context from their entity field, but it sounds like in your case there wouldn’t be a PublishedFile linked in the traditional way.
I think it is fair to say that the loader expects publishes to be linked in the normal way and isn’t flexible in its approach.