I’ve some incoming data files in S3 which are available via Glue data catalog / Athena or even plain text. I’d like to load them to neondb postgresql db but there seems to be no simple serverless way.
So far I found 3 options:
- Create an EC2 instance ; download files ; run postgresql copy
- Run Glue ETL job to load data from s3 to neondb
- Create custom python job and run it in either lambda or glue etc…
First two options have minimum cost associated with them to run all the time even when there is no data to load and third option might be slower in general.
Any other third party or more efficient options available ? Is neon considering any native integrations to S3 ?