Lately, I’ve been experimenting with different ways to access live SG/FPT’s data and make it available almost real-time to other platforms like Google Sheets, Grafana, PowerBI, and others without requiring user input.
I came up with this approach to generate a live CSV feed, using cron to update it every hour. Then, serve it up via a super simple nginx server. It’s pretty basic, but it works!
Now, I’m no developer, so I’m sure there’s tons of room for improvement. That’s why I’ve put the project up on GitHub so anyone in the community can check it out and contribute!
I’d really love to see your feedback on how this can be improved, thanks all!
Hey Alfonso, great initiative! Pretty simple but efficient.
In my past company we had a solution for this with Django.
It could be done with any backend service.
It’s pretty easy in python to have an endpoint that streams CSV content. so basically you end up with a URL like https://my-domain.com/get_csv_data
You could simply add what you’re looking for as GET or POST parameters, like the entity type, the filters, or whatever you’re looking for.
my new library, fpt-api · PyPI, could allow to get data for all the Entities in a page, or to get query fields data.
What you do in that endpoint is simple: fetch the data, and stream it as a CSV.
Which means that in Google Sheets you can use IMPORTDATA for example.
One caveat for this approach and any live CSV agnostic approach: google sheets for example has a cell limit in IMPORTDATA, and some other tools might have the same limitations, which means that sometimes you have to have the business logic directly in the tool.
Our system was more complex, using AMIs to provide a URL for each page you started from, authentication token, etc. But the basic functionality I described above is sufficient.
If you want me to elaborate on any of these aspects, DM me.
So glad to see you!
Thank you for sharing your insights! This tool grew out of a simple necessity: I wanted a quick way to pipe SG data into PowerBI, rather than manually exporting CSVs every time. I’ll admit, it was a bit of a “brute-force” solution from a producer who cosplays as coder from time to time. You know how budgets work
With that said, I’m well aware there’s plenty of room for improvement and optimization. If you have ideas on how to make it stronger, I’d love to explore them together and share something really valuable with the wider community!
