news.volyx.in

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5 (jeffgeerling.com)

617 points by rbanffy · 217 days ago · 228 comments on HN

Article summary

The article discusses the author's experience with a Mac Studio cluster using RDMA over Thunderbolt 5, which allows the Macs to act like they have a giant pool of RAM. The author tested the cluster with Exo 1.0, an open-source private AI clustering tool, and achieved significant performance improvements. The Mac Studio cluster is compared to other systems like the Nvidia DGX Spark and AMD AI Max+ 395, and its performance and efficiency are evaluated. The author also highlights the challenges of managing a Mac cluster and the limitations of Thunderbolt 5.

Main themes

  • Mac Studio cluster performance
  • RDMA over Thunderbolt 5
  • AI clustering
  • High-performance computing
  • Mac management challenges
  • Thunderbolt 5 limitations

What commenters say

  • The current RDMA setup over Thunderbolt 5 has limitations, including the need for each Mac to be connected to every other Mac, which can lead to a cabling mess.
  • Using QSFP instead of Thunderbolt 5 could provide better performance and scalability, but it may not be suitable for consumer devices.
  • The latency hit from using a switch with 200GbE is a significant concern, and RDMA over Thunderbolt 5 may have an advantage in this regard.
  • The Marvell switch chip used in the Mikrotik CRS812 switch is highly configurable and could potentially support combining multiple 50G ports into a single 200G port.
  • The cost and power usage of a 200GbE switch may be prohibitively expensive for some use cases, making the current RDMA setup more appealing.
  • The ability to use RDMA over Thunderbolt 5 is a significant advantage for Mac users, but it may not be enough to overcome the limitations of the Mac ecosystem for high-performance computing.
  • The author's experience with the Mac Studio cluster highlights the challenges of managing a cluster of Macs, including the need for additional tooling and the limitations of macOS.
  • The use of RDMA over Thunderbolt 5 is a promising development for AI clustering on Macs, but it requires further testing and evaluation to determine its suitability for different use cases.