Thank you so much for your input. It is very appreciated!
Sorry for disappearing for so long. A new project landed that kept me super busy for the last few weeks.
I am giving this pipeline a lot of thought, and want to make sure I have everything well planned before putting my hands on the code.
However I had to slow down a bit on this idea because unfortunately I have only so much time I can spend on it.
For the time being, I am testing the approach that you have suggested and doing some more in-depth research on the topic.
I will be back as soon as possible with updates 