Hey folks! We’re looking to use Postgres as an “event sourcing lite” source of truth, and I was wondering how others are handling this in Neon + Serverless, or what’s available. Basically when ever a row gets appended to the events table, we want to execute arbitrary logic (run Inngest handler, produce Kafka message or call a webhook for example). We’re in early stages so very much exploring, and as such really hoping to learn as much as we can. Thanks!
Generally, I’d recommend Debezium CDC for a regular postgres instance. However with neon, adding Debezium is not an option.
Below is a very hacky, very bad, and not-so-real-time workaround - which only works, if your events table is updated less frequently than a minute, and your compute node is only spun up if your events table is updated. If so, Set your RW endpoint to 1 minute suspend.
If all above applies, when your application inserts a row to the event table, there will be a start_compute operation for your compute node.
You can get a list of recent operations using below API endpoint.
Get a list of operations
You can poll this endpoint, let’s say every 5 minutes (again, this is a very very bad design!!), and if there’s a new start_compute operation compared your last detected one, your script would log in to you neon db, check the events table for new records & if there are, execute your logic.
Using this method, you can avoid proactively (&unnecessarily) connecting to your db instance - sparing yourself from active endpoint costs - however, you’ll heavily utilise the Neon API, which would probably not the best thing to do.
One cool thing would be, if there would be an option in Neon, which would could call a webhook, when a specific operation (start_compute) happens.
Hey @Steven,
It might be worth exploring Postgres LISTEN/NOTIFY (note however that you will need to use the direct connection and have to disable the default auto-suspend behavior.
We also plan on adding support for logical replication pretty soon, which could be helpful for your use case.
run Inngest handler, produce Kafka message or call a webhook for example)
Anything that requires network access is currently not supported in Neon.
LISTEN / NOTIFY has major limitations like character limits and will cut off messages unlike FULL replication.
Looking forward to Replication support!
1 Like
Any updates on this, simple triggers to a standard webhook are “table stakes” these days.
Separately, has anyone had luck connecting zapier to neon (source)?
Hey @weisisheng , Neon logical replication is available
see postgres docs: PostgreSQL: Documentation: 15: Chapter 31. Logical Replication