It is funny how when websites reference S3 (Simple Storage Service) storage, they typically reference a couple of the Hyperscalers (usually from Azure, GCP and AWS). While I get the reference to AWS, as they developed the service that has become the foundation for the API specification. But the association with just those hyperscalers is deeply misleading (and, to be honest, it bugs me). Any infrastructure as a service (IaaS) provider provides an S3 solution, and would probably not be taken seriously if they didn’t.

But S3 is not just a cloud-only capability today; enterprise-class SANs have S3 as part of their offering. There are open-source offerings that will allow you to overlay S3 onto commodity hardware Ceph, MinIO, to name two). Ceph even has an associated foundation that is a child of the Linux Foundation.

Why is this important?

Aside from the distributed storage capabilities. Perhaps a key factor is that the portability opens up several solution options that can be deployed both to the cloud and on-premises (yes, a POSIX filesystem is portable, but doesn’t offer versioning, and distribution isn’t transparent). One of these is that development around the adoption of HDFS has slowed.

Datalakes and cold-stored data

But S3 isn’t just helping data lakes; it is gaining adoption with OLAP structured storage because it is easy to use with Parquet files and indexes to create partitioning tables. Parquet’s columnar characteristics mean it’s easy to retrieve just the attributes you want (with focus on the rows through the use of indexing and segmentation). This helps power retained data that is cold stored (e.g. historic monthly finance figures kept in a database for analytical and compliance needs).

The open source ColdFront extension for Postgres (led by pgedge) leverages Parquet and S3 to store cold data. Still, the clever thing is that the feature transparently combines the relational data with the cold-stored data in Parquet files stored in S3.

S3 Tables

This confluence of Parquet files to support columnar table data storage has led to the development of a specialization of S3 in the form of S3 Tables. Today, S3 Tables are only an AWS feature, and it will be interesting to see if there is a wider adoption of the specification.

The Oracle database while not explicitly supporting the S3Tables API, can effect the same type of behavior by being able to treat S3 storage as an external table, and handle the metadata that will define a Parquet file within the S3 store as a while or segment of a table (for more read go here).