Structures¶
Tiled Adapters provide data in one of a fixed group of standard structure families. These are not Python-specific structures. They can be encoded in standard, language-agnostic formats and transferred from the service to a client in potentially any language.
Supported structure families¶
The structure families are:
array — a strided array, like a numpy array
sparse — a sparse array (i.e. an array which is mostly zeros)
dataframe — tabular data, as in Apache Arrow or pandas
node — a grouping of other structures, akin to a dictionary or a directory
Support for sparse arrays and Awkward Array are planned.
How structure is encoded¶
Tiled can describe a structure—its shape, chunking, labels, and so on—for the client so that the client can intelligently request the pieces that it wants.
The structures encodings are designed to be as unoriginal as possible, using established standards and, where some invention is required, using established names from numpy, pandas/Arrow, and dask.
Some structures are encoded in two parts:
Macrostructure — This is the high-level structure including things like shape, chunk shape, number of partitions, and column names. This structure has meaning to the server and shows up in the HTTP API.
Microstructure — This is low-level structure including things like machine data type(s) and partition boundary locations. It enables the service-side Adapter to communicate to the client how to interpret the bytes that represent a given “tile” of data.
Examples¶
These examples were generated by serving the demo tree
tiled serve pyobject --public tiled.examples.generated:tree
making an HTTP request with httpie and then extracting the portion of interest with jq, as shown below.
Array (single chunk)¶
An array is described with a shape, chunk sizes, and a data type.
The parameterization and spelling of the data type follows the
numpy __array_interface__ protocol.
Both built-in data types and
strucuted data types are supported.
An optional field, dims (“dimensions”) may contain a list with
a string label for each dimension.
This (10, 10)-shaped array fits in a single (10, 10)-shaped chunk.
$ http :8000/node/metadata/small_image | jq .data.attributes.structure
{
"macro": {
"chunks": [
[
100
],
[
100
]
],
"shape": [
100,
100
],
"dims": null,
"resizable": false
},
"micro": {
"endianness": "little",
"kind": "f",
"itemsize": 8
}
}
Array (multiple chunks)¶
This (10000, 10000)-shaped array is subdivided into 4 × 4 = 16 chunks,
(2500, 2500). Chunks do not in general have to be equally-sized,
which is why the size of each chunk is given explicitly.
$ http :8000/node/metadata/big_image | jq .data.attributes.structure
{
"macro": {
"chunks": [
[
2500,
2500,
2500,
2500
],
[
2500,
2500,
2500,
2500
]
],
"shape": [
10000,
10000
],
"dims": null,
"resizable": false
},
"micro": {
"endianness": "little",
"kind": "f",
"itemsize": 8
}
}
Array (with a structured data type)¶
This is a 1D array where each item has internal structure, as in numpy’s strucuted data types
$ http :8000/node/metadata/structured_data/pets | jq .data.attributes.structure
{
"macro": {
"chunks": [
[
2
]
],
"shape": [
2
],
"dims": null,
"resizable": false
},
"micro": {
"itemsize": 48,
"fields": [
{
"name": "name",
"dtype": {
"endianness": "little",
"kind": "U",
"itemsize": 40
},
"shape": null
},
{
"name": "age",
"dtype": {
"endianness": "little",
"kind": "i",
"itemsize": 4
},
"shape": null
},
{
"name": "weight",
"dtype": {
"endianness": "little",
"kind": "f",
"itemsize": 4
},
"shape": null
}
]
}
}
Sparse Array¶
There are a variety of ways to represent sparse arrays. The Coordinate list (COO) layout consists of writing the coordinate (e.g. row, column, or N-dimensional position) and value of each nonzero element. A N-dimensional COO array with M nonzero elements is described by an NxM array of coordinates and a length-M array of values.
Tiled describes this as a chunked array where each chunk contains a table of coordinates and their values. The data types within each table are not described at this level; they are self-described by the individual chunk payloads.
The key layout is always set to COO currently. In the future if other sparse
representations are supported, this key will be used to indicate which is used.
{
"shape": [
100,
100
],
"chunks": [
[
100
],
[
100
]
],
"dims": null,
"resizable": false
}
DataFrame¶
With dataframes, we speak of “partitions” instead of “chunks”. There are a couple important distinctions. We always know the size of chunk before we ask for it, but we will not know the number of rows in a partition until we actually read it and enumerate them. Therefore, we cannot slice into dataframes the same way that we can slice in to arrays. We can ask for a subset of the columns, and we can fetch partitions one at a time in any order, but we cannot make requests like “rows 100-200”. (Dask has the same limitation, for the same reason.)
$ http :8000/node/metadata/long_table | jq .data.attributes.structure
{
"macro": {
"npartitions": 5,
"columns": [
"A",
"B",
"C"
],
"resizable": false
},
"micro": {
"meta": "data:application/vnd.apache.arrow.file;base64,...",
"divisions": "data:application/vnd.apache.arrow.file;base64,...",
}
}
Notice that the microstructure contains base64-encoded data. The correct way to encode dataframes and their data types in a cross-language way is with Apache Arrow. Apache Arrow is a binary format. It explicitly does not support JSON. (There is a JSON implementation, but the documentation states that it is intended only for integration testing and should not be used by external code.) Therefore, when JSON is requested, we base64-encode it. When binary msgpack is requested instead of JSON, we pack the binary data directly.
The microstructure has two parts:
meta— This contains the names and data types of the columns and index. To generate this we build a dataframe with zero rows in it but the same columns and indexes as the original, and then serialize that with Arrow.divisions— This contains the index values that delineate each partition. We generate this in a similar way.
Both of the concepts (and their names) are borrowed directly from dask.dataframe. They should enable any client, including in languages other than Python, to perform the same function.
Node¶
The node structure is a container for other structures. It may be compared to a directory, a JSON object, a Python dictionary, or an HDF5 Group. Nodes may contain other nodes, any other structure, or a mixture.
Some nodes contain a small number of children, easy to list in a single request, while others may contain many listed via multiple paginated requests. Some Tiled deployments currently in use contain nodes with with up to hundreds of thousands of children.
Typically, a node’s structures tell us only how many children it has (count).
The contents key is typically set to null, which indicates that we will need
a separate request to fetch information of this node’s children.
{
"contents": null,
"count": 2
}
In certain cases, it is efficient to in-line all the information about the node’s children (their metadata, structure, and more) in a single response.
{
"contents": {
"lat": {
"attributes": {
"ancestors": [
"structured_data",
"xarray_dataset"
],
"metadata": {},
"sorting": null,
"specs": [
"xarray_coord"
],
"structure": {
"macro": {
"chunks": [
[
2
],
[
2
]
],
"dims": [
"x",
"y"
],
"resizable": false,
"shape": [
2,
2
]
},
"micro": {
"endianness": "little",
"itemsize": 8,
"kind": "f"
}
},
"structure_family": "array"
},
"id": "lat",
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/lat?block={index_0},{index_1}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/lat",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/lat"
},
"meta": null
},
"lon": {
"attributes": {
"ancestors": [
"structured_data",
"xarray_dataset"
],
"metadata": {},
"sorting": null,
"specs": [
"xarray_coord"
],
"structure": {
"macro": {
"chunks": [
[
2
],
[
2
]
],
"dims": [
"x",
"y"
],
"resizable": false,
"shape": [
2,
2
]
},
"micro": {
"endianness": "little",
"itemsize": 8,
"kind": "f"
}
},
"structure_family": "array"
},
"id": "lon",
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/lon?block={index_0},{index_1}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/lon",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/lon"
},
"meta": null
},
"precipitation": {
"attributes": {
"ancestors": [
"structured_data",
"xarray_dataset"
],
"metadata": {},
"sorting": null,
"specs": [
"xarray_data_var"
],
"structure": {
"macro": {
"chunks": [
[
2
],
[
2
],
[
3
]
],
"dims": [
"x",
"y",
"time"
],
"resizable": false,
"shape": [
2,
2,
3
]
},
"micro": {
"endianness": "little",
"itemsize": 8,
"kind": "f"
}
},
"structure_family": "array"
},
"id": "precipitation",
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/precipitation?block={index_0},{index_1},{index_2}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/precipitation",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/precipitation"
},
"meta": null
},
"temperature": {
"attributes": {
"ancestors": [
"structured_data",
"xarray_dataset"
],
"metadata": {},
"sorting": null,
"specs": [
"xarray_data_var"
],
"structure": {
"macro": {
"chunks": [
[
2
],
[
2
],
[
3
]
],
"dims": [
"x",
"y",
"time"
],
"resizable": false,
"shape": [
2,
2,
3
]
},
"micro": {
"endianness": "little",
"itemsize": 8,
"kind": "f"
}
},
"structure_family": "array"
},
"id": "temperature",
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/temperature?block={index_0},{index_1},{index_2}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/temperature",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/temperature"
},
"meta": null
},
"time": {
"attributes": {
"ancestors": [
"structured_data",
"xarray_dataset"
],
"metadata": {},
"sorting": null,
"specs": [
"xarray_coord"
],
"structure": {
"macro": {
"chunks": [
[
3
]
],
"dims": [
"time"
],
"resizable": false,
"shape": [
3
]
},
"micro": {
"endianness": "little",
"itemsize": 8,
"kind": "M"
}
},
"structure_family": "array"
},
"id": "time",
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/time?block={index_0}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/time",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/time"
},
"meta": null
}
},
"count": 5
}