pg_durable: Microsoft open sources in-database durable execution (github.com)

Article summary

Microsoft has open-sourced pg_durable, a PostgreSQL extension that allows for in-database durable execution of long-running SQL functions. This enables teams to define workflows in SQL and have them executed with checkpointing and resume capabilities in case of crashes or failures. The extension is designed for teams that already store their state in Postgres and want to simplify their background work. pg_durable provides a SQL-native way to define functions and workflows, eliminating the need for external services or infrastructure.

Main themes

• PostgreSQL extension
• Durable execution
• In-database workflows
• SQL-native functions
• Background work simplification
• Database reliability

What commenters say

• Some developers prefer to keep their queue logic in code, rather than in the database, for better version control and flexibility.
• The project's DSL ergonomics can be improved, but it has the potential to substantially reduce code and increase reliability.
• Storing orchestration logic in the database can provide benefits such as snapshotting and point-in-time recovery, but may not be suitable for all use cases.
• The use of pg_durable can simplify ETL pipelines and state machine-type jobs by keeping the job execution and data in the same server.
• Some developers are hesitant to adopt pg_durable due to concerns about debugging, testing, and releasing workflows defined in SQL.
• The project's approach may not be comparable to external orchestration tools like Temporal or Airflow, which have their own strengths and use cases.
• The ability to define workflows in SQL can be beneficial for teams that already store their state in Postgres and want to minimize external dependencies.
• The trade-offs between storing workflow logic in the database versus in code need to be carefully considered, depending on the specific use case and requirements.