S3-Backed Databases and Table Formats
Open table formats add database consistency and transactions to S3's cheap, durable storage.
Section
11 stories in S3 Architecture.
Open table formats add database consistency and transactions to S3's cheap, durable storage.
They work best as layers, not rivals.
Faster checkpointing patterns preserve GPU utilization when distributed training clusters fail.
Four tools mount S3 as a filesystem, each with different tradeoffs on speed and POSIX compatibility.
Versioning creates hidden storage costs that compound silently until the bill arrives.
Retrieval fees and minimum durations, not storage rates, decide your cold archive bill.
Design S3 data lake zones and partitions around query patterns, not data arrival.
Lifecycle rules expire data on schedule; Object Lock prevents deletion before its time.
Misunderstanding day-counting and filters turns cost-saving rules into budget disasters.
Traditional archiving thresholds don't match AI's dormancy cycles and retrieval needs.
Assess which workloads truly need block storage before redesigning applications for object storage.