You can modify the publish plugin UI, here is an example, but for anything else you would need to take over the app. However I would still make use of the item properties to store your settings as this will mean you then don’t need to modify the publish plugin at all, as it already looks for the publish_fields property.
From the error you are getting is looks like you are providing incorrect field names. They look like custom fields, so I would expect them to start with sg_ at least. I would check that you are providing the correct PublishedFile field names.