Anatomy of a file

Every geospatial format exists to solve the same fundamental problem: how do we persist spatial (and sometimes temporal) data so it can be efficiently stored, shared, and queried? These concerns apply to all data types: raster, vector, point cloud, etc. Yet as a community, we often treat formats as islands, each with its own ecosystem, tooling, and mental model.

In this talk, we start from data. We have geospatial data and we need to persist it. How do we linearize multidimensional data into bytes? How do we chunk it for partial reads? Are we going to support writing in parallel? What encoding and compression strategies will we use and what tradeoffs do they carry? As we work through these questions, the need for metadata arises—coordinate systems, data types, spatial indices, encoding indicators—because without it, the data is useless. Piece by piece, we build up the anatomy of a file.

We then map what we’ve built to real formats. The same structures appear in each: chunking strategies, index structures, encoding schemes, metadata organization. The specific tradeoffs differ, but the underlying anatomy is consistent across raster, vector, and beyond.

This consistency raises a question worth confronting: if formats share this much of their architecture, why does our tooling treat them as fundamentally different things? Perhaps we should be recognizing our ecosystems have built walls around format tradeoffs that the underlying anatomy doesn’t justify.