news.volyx.in

SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue (prequel.co)

628 points by ctc24 · 1237 days ago · 361 comments on HN

Article summary

The company Prequel replaced RabbitMQ with a queue built on top of their Postgres database due to issues with RabbitMQ's prefetch behavior, which caused delays in job processing. The new queue implementation is simpler and more reliable, and has improved the overall resiliency of their system. The change took half a day to implement and test, and reduced the codebase by 580 lines. The company attributes the success of the new queue to their familiarity with SQL and Postgres.

Main themes

  • queue implementation
  • RabbitMQ vs Postgres
  • scalability and performance
  • operational complexity
  • database-as-queue
  • message brokers
  • system design and architecture

What commenters say

  • The company's issues with RabbitMQ may have been due to misconfiguration or misunderstanding of its prefetch behavior.
  • Using a database as a queue can be a good solution for small to medium-scale applications, but may not be suitable for large-scale applications.
  • Postgres can handle a high volume of jobs per day as a queue, but its performance may degrade as the queue size increases.
  • The decision to replace RabbitMQ with a Postgres queue may have been motivated by a desire to simplify the system and reduce operational complexity.
  • Some commenters believe that the company's approach is an example of 'Not Invented Here' syndrome, where they preferred to build their own solution rather than using an existing one.
  • Others argue that using a database as a queue can provide benefits such as atomicity and reliability, but requires careful consideration of issues like tombstoning and vacuuming.
  • The company's experience highlights the importance of carefully evaluating the trade-offs between different solutions and considering factors like scalability, reliability, and operational complexity.
  • Some commenters suggest that the company's solution may not be suitable for all use cases, and that RabbitMQ or other message brokers may be more appropriate for certain applications.