Draft Specification

There are four user-facing objects in the system:

  • Reader – extracts the metadata, structure, and (tiled) data from some source of store or generated data

  • Tree – nestable collection of other Trees or Readers

  • Query – high-level description of a search query over entries in a Tree

  • AccessPolicy – class with methods for enforcing access control on which entries in a Tree a given identity (user) can see

This specification proposes the Python API required to duck-type as a Tree or Reader as well as a sample HTTP API based on JSON API.

Python API

Readers

  • Readers MUST implement a metadata attribute or property which returns a dict-like. This metadata is treated as user space, and no part of the server or client will rely on its contents.

  • Readers MUST implement a structure_family attribute or property which returns a string of the general type that will be returned by read(), as in intake. These will be generic terms like "array", not the __qualname__ of the class. It is meant to encompass the range of concrete types (cupy array, sparse array, dask array, numpy ndarray) that duck-type as a given generic structure.

  • Readers MUST implement a method structure() with no arguments which returns a description of the structure of this data. For each structure family (array, dataframe, etc.) there will be a specific schema for this description (TBD). For example, “array” reports machine data type, shape, and chunks. Richer structure (e.g. xarray) will include high-level structure like columns, dimensions, indexes.

  • Readers MUST implement a method read() returns the data structure. It MAY return a lazy object, such as a PIMS sequence, dask array, dask dataframe, or other structure with delayed I/O.

  • Readers MUST implement a method read_block(...) that efficiently reads and return a single block of data. It MUST perform all necessary I/O and return a self-contained and complete structure; that is, unlike read() it must not be delayed. The signature SHOULD reuse existing terminology from the corresponding data strucutre. Examples:

    array_reader.read_block(block=(0, 0))
dataframe_reader.read_block(partition=0)
dataset_reader.read_block(variable="image", coord="x", block=(0,))

    Additionally, the method SHOULD accept a slice parameter which takes a tuple of Python slice objects into the block to specify a partial block.

  • Reader MAY allocate system resources (file handles, network connections, a cache in memory) but MUST implement a method close() which releases those resources. It MUST also implement the context manager API and call close() on exiting the context.

  • Readers MAY implement other methods beyond these for application-specific needs or usability.

Trees

  • Trees MUST implement the collections.abc.Mapping interface. That is:

    tree.__getitem__
tree.__iter__

    Trees may omit __len___ as long as they provide __length_hint__, an estimated length that may be less expensive for Trees backed by databases. That is, implement one or both of these:

    tree.__len__
tree.__length_hint__

  • Trees MUST implement an attributes which support efficient positional lookup and slicing.

    tree.keys_indexer[i]             # -> str
tree.values_indexer[i]           # -> Union[Tree, Reader]
tree.items_indexer[i]            # -> Tuple[str, Union[Tree, Reader]]
tree.keys_indexer[start:stop]    # -> List[str]
tree.items_indexer[start:stop]   # -> List[Union[Tree, Reader]]
tree.values_indexer[start:stop]  # -> List[Tuple[str, Union[Tree, Reader]]]

  • It is NOT necessary to implemented strided slicing, as in [start:stop:5]. It is sufficient to assume the stride is always 1.

  • The values in a Tree MUST be other Trees or Readers.

  • The keys in a Tree MUST be non-empty strings adhering to the JSON API spec for allowed characters in resource ids.

  • Trees MUST implement a search method which accepts a Query as its argument and returns another Tree with a subset of the items. specified.

  • Trees MUST implement a metadata attribute or property which returns a dict-like. This metadata is treated as user space, and no part of the server or client will rely on its contents.

  • Trees MAY implement other methods beyond these for application-specific needs or usability.

  • The method for initializing this object is intentionally unspecified. There will be variety.

  • [This may need revisiting. Should it be qualified? Eliminated?] The items in a Tree MUST have an explicit and stable order.

  • [This may need revisiting. Should it be qualified? Eliminated?] The data underlying the Tree may be updated to add items, even though the Tree itself is a read-only view on that data. Any items added MUST be added to the end. Items may not be removed.

Queries

  • Queries MUST be dataclasses.

  • They MAY have any attributes. There are no required attributes.

Access Policy

An AcessPolicy is tightly coupled to:

  1. How the metadata related to data management are stored — “Which type of database? Which collection / table in the database? Which field / column in that table?”

  2. The particulars of a given access management system that encodes who can see what — “Given an authenticated identity, what API do I use to check what it is allowed to see?”

We envision a custom AccessPolicy will be needed for every combination of (1) and (2). It is injected into a given Tree at __init__ time.

An AccessPolicy gets two opportunities to restrict the entries that a caller can see. First, a Tree gives it the opportunity to modify a query before it is processed in order to restrict the set of results marshalled from storage. Second, it may filter the query results before they are returned to the user. Depending on the storage and the complexity of the data access rules, one or both of these modes may be used by a given AccessPolicy.

  • An AccessPolicy MUST implement a method modify_query(query: Query, authenticated_identity: str) -> Query. This MAY filter the result at the database level so that less data is marshalled from storage.

  • An AccessPolicy MUST implement a method filter_results(tree, authenticated_identity: str) -> Tree

  • An AccessPolicy MUST implement a method check_compatibility(tree) -> bool which can be used by the Tree to verify at its __init___ time that the AccessPolicy it has been given understands how to modify its queries.