How Tiled Fits into the Ecosystem

What Tiled is Not

  • Tiled is a not a storage format like TileDB or Zarr. It is expected to work well with those formats because they enable efficient chunk-based access, but it is designed to work with less modern formats as well, to accept the data “as is” and convert it on the fly upon request.

  • Tiled is not a data-browsing or visualization front-end like Plotly Chart Studio, Vega Voyager, or JupyterLab Data Explorer but it can be used to serve data into them. (Plotly and Vega both work today; the JupyterLab Data Explorer is being investigated.)

  • Tiled is not an application server like Vaex. Tiled offers no compute; it is focused only on I/O.

  • Tiled is a not a server for specific data format like Xpublish or JupyterLab HDF5, though it may be possible to build similar things with Tiled easily these formats and others.

Lineage

The originating authors of Tiled have worked on several previous, related projects.

  • PIMS (2013, JHU Leheny Lab), or Python Image Sequence, abstracts over a variety of image time series formats with a lazy-loading, numpy-slicable Python object. It has some goals in common with Dask Array but it is tightly focused on image time series, which lends it ceratin advantages for that use case. PIMS is still maintained and widely used, including as a dependency of dask-image.

  • Databroker (2015, Brookhaven National Laboratory) provides searchable dict-like “catalogs” of data with entries that ultimately provide N-dimensional or tablular data as PIMS objects or pandas DataFrames respectively. It also provides streaming access to data and is tightly coupled to the Bluesky “Document Model” for streaming data.

  • Intake (2017, Anaconda Inc.) is similar to Databroker in its goals and feature set—searchable catalogs of data that ultimately provide standard SciPy data structures—but its intended scope is broader than Databroker’s, not being tied to the Bluesky “Document Model” in particular. It also supports nested catalogs and chainable search queries. In 2018–2020, Databroker was refactored to become a set of Intake drivers.

Intake also has a prototype of an HTTP server, which has not yet been fully developed into a robust tool. Attempts to rework that server led to the conclusion that Intake’s architecture and API privilege ergonomic interactive use and direct access from Python. The changes neccessary to use it as library code effectively within a performant service or larger application would be largely subtractive, in tension with its use as a user-facing interactive exploratory tool.

Which brings us to Tiled (2021, Brookhaven National Laboratory with the Bluesky Collaboration). Tiled is designed with HTTP-based access as the driving use case, targeting Python clients, browser-based clients, and clients in other languages. Tiled is compatible with PIMS for reading image series. Tiled has search capability and some other features drawn from Databroker. Tiled has a Tree abstraction similar to Intake’s Catalog, with small but important differences to suit the server–client interaction. (It is hoped that Tiled Trees and Intake Catalogs can be made interoperable in the near future.) Finally, Tiled has an Adapter abstraction that is something like an amalgam of PIMS Readers and Intake’s DataSources.

In summary:

  • Tiled can use PIMS internally to efficiently slice and read image series.

  • Tiled can probably be made interoperate with Intake objects.

  • Databroker will evolve to be a thin wrapper around Tiled, adding concepts and capabilities specific to the Bluesky “Document Model”.