news.volyx.in

Big data is dead (2023) (motherduck.com)

585 points by armanke13 · 809 days ago · 457 comments on HN

Article summary

The article argues that the era of Big Data is over, as most organizations do not have massive amounts of data and the technology to handle large data sets has become more accessible and affordable. The author, a former engineer at Google BigQuery, claims that the focus on Big Data was misguided and that most companies can manage their data with traditional systems. The article presents data and examples to support this claim, including the fact that most customers using BigQuery have less than a terabyte of data. The author concludes that the industry should shift its focus from handling large data sets to using data to make better decisions.

Main themes

  • Big Data
  • Data Management
  • Scalability
  • Cloud Computing
  • Artificial Intelligence
  • Database Systems

What commenters say

  • The concept of Big Data has been overhyped and is no longer a significant concern for most organizations.
  • Planning for scalability and using complex systems can be a waste of resources and hinder productivity.
  • Most companies do not need to use specialized Big Data solutions and can manage their data with traditional database systems like PostgreSQL or SQLite.
  • The focus on Big Data has been replaced by a focus on Artificial Intelligence, but AI is not a practical tool for working with large data sets due to its tendency to hallucinate results.
  • The goal of becoming a unicorn is not a realistic or desirable goal for most businesses, and prioritizing scalability and growth over execution and iteration can be counterproductive.
  • The idea that companies need to plan for massive growth and scalability from the outset is a myth, and it is more effective to focus on solving current problems and iterating quickly.
  • The use of Big Data solutions is often driven by a desire to appear innovative and cutting-edge, rather than a genuine need for the technology.
  • The distinction between Big Data and traditional data management is not as clear-cut as often claimed, and many companies can effectively manage their data with a combination of traditional and specialized systems.