Serve Files with Custom Formats¶

Out of the box, Tiled can serve a directory of files that have common formats with recognizable file names like *.csv, *.tiff, or *.h5. In this guide, we will configure it to recognize files that have nonstandard (e.g. custom) names and/or custom formats.

Note

Tiled is not limited to serving data from files.

Large deployements typically involve a database, supporting fast search on metadata, and perhaps external files or “blob stores” with large data.

But starting with files is a good way to get rolling with Tiled.

Formats are named using “MIME types”¶

Tiled refers to formats using a web standard called MIME types. MIME types look like:

text/csv
image/png
application/x-hdf5

There is an official list of registered MIME types, and if an official one exists we use it. If a format is not registered, then the standard tells us to use text/x-SOMETHING if the format is textual or application/x-SOMETHING if it is binary. For example, we use text/x-xdi for XDI and applicaiton/x-hdf5 for HDF5, formats which are not registered.

Case 1: Unfamiliar File Extension¶

Suppose you have data files that are formatted in a supported format like CSVs. If they were named *.csv then Tiled could handle them natively without any additional configuration:

tiled serve directory path/to/directory

But if they use some unusual extension like *.stuff Tiled needs to be told that it should read *.stuff files like CSVs.

Map the unfamiliar file extension to a MIME type¶

We use a configuration file like this:

# config.yml
trees:
- tree: files
  args:
    directory: path/to/directory
    mimetypes_by_file_ext:
      .stuff: text/csv

We are mapping the file extension, .stuff (including the leading .) to the MIME type text/csv.

Multiple file extensions can be mapped to the same MIME type. For example, Tiled’s default configuration maps both .tif and .tiff to image/tiff.

We then use the configuration file like this:

tiled serve config config.yml

The configuration file config.yml can be named anything you like.

Case 2: No File Extension¶

Not all files have a name like <name>.<extension>. Some have no dot, like:

data0001
data0002
data0003

Others do have a dot, but the part after the dot is not really a file extension; it does not signify the format. Instead, it’s scientific metadata of some kind, as in:

polymer_10_new_Ck150V.2050
polymer_10_new_Ck150V.3050
polymer_10_new_Ck150V.4050

Write a custom function for detecting the MIME type¶

The best solution is to avoid naming files like this, but we cannot always control how our files are named. To cope with this, we need to write a Python function.

# custom.py

def detect_mimetype(filepath, mimetype):
    if mimetype is None:
        # If we are here, detection based on file extension came up empty.
        ...
        mimetype = "text/csv"
    return mimetype

The function detect_mimetype will be passed the full filepath (e.g. path/to/filename) not just the filename. It can use this to examine the filename or even open the file to, for example, look for a file signature. The function will also be passed the mimetype, if any, that was detected based on its file extension. Therefore, this function can be used to catch files that have no file extension or to override the determination based file extension if it is wrong.

If the Python script custom.py is placed in the same directory as config.yml, Tiled will find it. (Tiled temporarily adds the directory containing the configuration file(s) to the Python import path while it parses the configuration.)

# config.yml
trees:
- tree: files
  args:
    directory: path/to/directory
    mimetype_detection_hook: custom:detect_mimetype

Alternatively, if the function can be defined in some external Python package like my_package.my_module.func and configured like

mimetype_detection_hook: my_package.my_module:func

Note that the packages are separated by . but the final object (func) is preceded by a :. If you forget this, Tiled will raise a clear error to remind you.

The names custom.py and detect_mimetype are arbitrary. The mimetype_detection_hook may be used in combination with mimetypes_by_file_ext.

As in Case 1, we use the configuration file like this:

tiled serve config config.yml

Case 3: Custom Format¶

In this case we format that Tiled cannot read. It’s not just a familiar format with an unfamiliar name; it’s a new format that Tiled needs to be taught how to read.

Choose a MIME type¶

Referring back to the top of this guide, we need to choose a MIME type to refer to this format by. As an example, we’ll call our format

application/x-stuff

The is, of course, some risk of name collisions when we invent names outside of the official list of MIME types, so be specific.

Write a custom adapter¶

Tiled must represent the content of your file as:

An array + a dictionary of metadata
A table (dataframe) + dictionary of metadata
A nested structure (i.e. directory-like hierarchy) of the above

You must choose which is appropriate for this data format. Examples for each structure follow.

Simple Array example¶

# custom.py
from tiled.adapters.array import ArrayAdapter

def read_custom_format(filepath):
    # Extract an array and an optional dictionary of metadata
    # from your file.
    array = ...  # a numpy array
    metadata = ...  # a dictionary or None
    return ArrayAdapter.from_array(array, metadata=metadata)

Simple Tabular (DataFrame) example¶

# custom.py
from tiled.adapters.dataframe import DataFrameAdapter

def read_custom_format(filepath):
    # Extract a DataFrame and an optional dictionary of metadata
    # from your file.
    df = ...  # a pandas DataFrame
    metadata = ...  # a dictionary or None
    return DataFrameAdapter.from_pandas(df, npartitions=1, metadata=metadata)

Simple Nested Structure example¶

# custom.py
from tiled.adapters.array import ArrayAdapter
from tiled.adapters.dataframe import DataFrameAdapter
from tiled.adapters.mapping import MapAdapter

def read_custom_format(filepath):

    # Build a dictionary (potentially nested) of arrays and/or dataframes.
    # See examples above for ArrayAdapter and DataFrameAdapter usage.

    return MapAdapter(
        {
            "stuff": ArrayAdapter.from_array(...),
            "things": DataFrameAdapter.from_pandas(...),
        }
        metadata={...},
    )

Advanced: Delay I/O¶

See the implementations in the pacakage tiled.adapters for more advanced examples, especially ways to refer reading the entire file up front if the user only wants to read part of it.

Advanced: Mark up Structure with optional “Specs”¶

If the array, table, or nested structure follows some convention or standard for its internal layout or naming scheme, it can be useful to notate that. Some Tiled clients may be able to use that information to provide additional functionality or performance.

See :doc:../explanations/metadata for more information on Specs.

Specify them as an argument to the Adapter, as in:

DataFrameAdapter(..., specs=["xdi"])

Configure Tiled to use this Adapter¶

Our configuration file should use mimetypes_by_file_ext (Case 1) or mimetype_detection_hook (Case 2) to recognize this custom file. Additionally, it should add a section readers_by_mimetype to map our MIME type application/x-stuff to our custom function.

Again, Tiled will find custom.py if it is placed in the same directory as config.yml. The name is arbitrary, and you can have multiple such files if needed.

# config.yml
trees:
- tree: files
  args:
    directory: path/to/directory
    mimetype_detection_hook: custom:detect_mimetype
    readers_by_mimetype:
      application/x-stuff: custom:read_custom_format

We then use the configuration file like this:

tiled serve config config.yml