SQLite: 35% Faster Than the Filesystem (sqlite.org)

Article summary

The article compares the performance of SQLite and filesystem I/O, finding that SQLite reads and writes small blobs 35% faster than the same blobs stored in individual files on disk. This performance difference arises from the reduced overhead of open() and close() system calls when using SQLite. The article also notes that SQLite uses about 20% less disk space than storing blobs in individual files. The measurements were made using a version of SQLite between 3.19.2 and 3.20.0.

Main themes

• SQLite performance
• Filesystem I/O comparison
• Database storage efficiency
• Operating system dependencies
• Filesystem limitations

What commenters say

• Embedding SQLite in the kernel could potentially lead to significant performance gains, but it would require careful consideration of the added complexity and potential issues.
• Using SQLite as a filesystem, such as through FUSE, may introduce additional overhead and negate the performance benefits of using SQLite.
• The performance difference between SQLite and filesystem I/O is largely due to the reduced overhead of system calls and the ability of SQLite to optimize data access patterns.
• Some commenters argue that the performance benefits of SQLite are not unique to the database itself, but rather a result of the inefficiencies of certain filesystems, such as NTFS, in handling many small files.
• Others suggest that the idea of using a database as a filesystem is not new and has been explored in the past, with mixed results.
• The article's findings are not universally applicable and may depend on specific use cases, hardware, and operating system configurations.
• Some commenters question the relevance of the article's findings, given that the data is from 2017 and may not reflect current performance characteristics.