news.volyx.in

Kafka is Fast – I'll use Postgres (topicpartition.io)

561 points by enether · 269 days ago · 401 comments on HN

Article summary

The article discusses the idea of using Postgres as a replacement for Kafka in certain use cases, presenting benchmark results for pub/sub messaging and queueing. It argues that Postgres can handle a significant portion of the use cases that Kafka is typically used for, with less complexity and cost. The author suggests that the choice between Postgres and Kafka should be based on practical considerations rather than a desire to use the latest technology. The article also touches on the trend of over-engineering and the importance of simplicity in software design.

Main themes

  • Postgres vs Kafka
  • Over-engineering
  • Simple technology
  • Scalability
  • Database performance
  • Software design

What commenters say

  • Using Postgres for everything can be a good default approach, as it can handle most use cases with less complexity and cost.
  • Kafka is a useful technology, but it may not be necessary for many use cases, and its adoption can be driven by resume-driven design rather than actual need.
  • The choice between Postgres and Kafka should be based on practical considerations, such as the specific requirements of the project, rather than a desire to use the latest technology.
  • Over-engineering and premature optimization can be major problems in software design, and simplicity should be preferred whenever possible.
  • Postgres can be used for queuing and pub/sub messaging, but it may not be the best choice for very high-traffic or high-performance applications.
  • The decision to use Postgres or Kafka should be based on the actual needs of the project, rather than fear of scaling or performance issues.
  • Using the same Postgres instance for both data and queueing can be a good approach, as long as it can handle the load, and separating them can add unnecessary complexity.
  • There are cases where Kafka's features, such as handling traffic bursts, make it a better choice than Postgres, and the decision should be based on the specific requirements of the project.