Add Custom Export Formats

The Tiled server can provide data—in whole or in part—in a variety of formats. See Deliberate Export for a list of the formats supported out of the box for each structure family ("array", "table", "container", …).

This set of formats can easily be extended. A complete working example is included in the tiled source tree at example_configs/custom_export_formats. We will build it up from scratch.

We’ll start with a text-based format and then address a binary one.

Text Format Example

As our first example, we will invent a variation on CSV (comma-separated variables) that uses a 🙂 instead of a comma, as in

1🙂2🙂3
4🙂5🙂6

We will apply this format to arrays. Tiled expects a function with the interface:

def f(
    array: numpy.ndarray,
    metadata: Optional[dict]
): -> str | bytes
    ...

Here is an implementation that exports an array as smiley-separated variables.

# custom_exporters.py

def smiley_separated_variables(array, metadata):
    return "\n".join("🙂".join(str(number) for number in row) for row in array)

In real-world cases, there is often already a library that writes the format of interest. Then, our goal isn’t to write an exporter from “scratch”; it’s to integrate some existing exporter with Tiled. For example, numpy can be made to write smiley-separated variables. The trick is to make the library write to a buffer in memory rather than to a file on disk, and then return a string. Most libraries support the following approach.

import io
import numpy

def smiley_separated_variables(array, metadata):
    # This StringIO presents a file-like interface that numpy can write to.
    file = io.StringIO()
    numpy.savetxt(buffer, array, delimiter="🙂", fmt="%s")
    return file.getvalue()

Either approach—from scracth or using numpy—-will work in our case. Notice that we also get a dictionary of metadata. Some formats give us nowhere to put this extra information, and we can just drop it in that case.

To integrate this with Tiled, we invoke it in a configuration file.

# config.yml

# Register a custom format for the "array" structure family.
media_types:
  array:
    application/x-smileys: custom_exporters:smiley_separated_variables
# And provide some example data to try it with....
trees:
  - path: /
    tree: tiled.examples.generated_minimal:tree

The term application/x-smileys is a “media type”, also known as “MIME type”. In our case, there is no registered IANA Media Type for our exotic format. Therefore, the standard tells us to invent one of the form application/x-*. There is, of course, some risk of name collisions when we invent names outside of the official list, so be specific.

With custom_exporters.py and config.yml placed side by side in some directory, we can start the server.

tiled serve config --public config.yml

Note

If custom_exporters.py is placed in the same directory as config.yml, the Tiled server will be able to find and import the custom_exporters module even if it isn’t installed in the normal Python module search path or placed in the current working directory.

When it loads the configuration, Tiled temporarily adds the directory containing config.yml to the Python module search path (sys.path). This makes it easy to prototype and integrate custom code. Of course, the configuration can also load modules that are installed in the normal fashion.

We can request data as smiley-separated variables from the command line using HTTPie:

$ http :8000/array/full/A?slice=:5,:5 Accept:application/x-smileys
HTTP/1.1 200 OK
content-length: 159
content-type: application/x-smileys; charset=utf-8
date: Wed, 12 Jan 2022 21:38:24 GMT
etag: 8b6ec7a60f30c181762a4c73a6b433b0
server: uvicorn
server-timing: read;dur=3.3, tok;dur=0.1, pack;dur=0.2, app;dur=8.1
set-cookie: tiled_csrf=JDHYkMUIBWECLqIJJvTaEcinv_Vd3kTxS08XCw3N4Yg; HttpOnly; Path=/; SameSite=lax

1.0🙂1.0🙂1.0🙂1.0🙂1.0
1.0🙂1.0🙂1.0🙂1.0🙂1.0
1.0🙂1.0🙂1.0🙂1.0🙂1.0
1.0🙂1.0🙂1.0🙂1.0🙂1.0
1.0🙂1.0🙂1.0🙂1.0🙂1.0

or using the Tiled Python client.

from tiled.client import from_uri
c = from_uri("http://localhost:8000/api")
c['A'][:5, :5].export("test.txt", format="application/x-smileys")

Binary Format Example

Let’s add support for JPEG images. Tiled doesn’t build in support for JPEG; for scientific uses, PNG is better because it is lossless.

We’ll use the library PIL to write the JPEG data. As with the numpy example above, we need to intercept its output in a buffer. In this case, it will be a binary buffer, BytesIO, instead of StringIO.

# custom_exporters.py

import io
from PIL import Image
from tiled.structures.image_serializer_helpers import img_as_ubyte

def to_jpeg(array, metadata):
    # PIL detail: ensure array has compatible data type before handing to PIL.
    prepared_array = img_as_ubyte(array)
    image = Image.fromarray(prepared_array)
    file = io.BytesIO()
    image.save(file, format="jpeg")
    return file.getbuffer()

This covers the basic functionality. See the built-in exporters in tiled/strucures/array.py for details that add polish, like scaling the image’s dynamic range and failing gracefully when given arrays that have the wrong dimensionality to be exported as an image.

We’ll add it to our configuration.

# config.yml

media_types:
  array:
    application/x-smileys: custom_exporters:smiley_separated_variables
    image/jpeg: custom_exporters:to_jpeg
trees:
  - path: /
    tree: tiled.examples.generated_minimal:tree

Start the server again

tiled serve config --public config.yml

and navigate a web browser to http://localhost:8000/api/v1/array/full/A?format=image/jpeg. Since the example data is just an array of ones, this will appear as a white square image.

File extensions as convenience aliases

Now, image/jpeg is unwieldy for users unfamiliar with MIME types. Adding

file_extensions:
  jpg: image/jpeg
  jpeg: image/jpeg

to the configuration enables

http://localhost:8000/api/v1/array/full/A?format=jpeg
http://localhost:8000/api/v1/array/full/A?format=jpg

as equivalent to

http://localhost:8000/api/v1/array/full/A?format=image/jpeg

Note

The format can also be specified as an HTTP Accept Header. In that case, it must be given as a MIME type, in accordance with the standard. The file extension alias is not accepted.

Advanced: Streaming export

HTTP supports chunked responses, where data is streamed incrementally. This is a good fit for streaming-oriented formats such as newline-delimited JSON.

To create a chunked exporter, implement your exporter as a Python generator that yields bytes.

def export(array, metadata):
    for ... in ...:
        yield b"..."

Further examples

At the bottom of each of the modules in tiled/structures, you will find the code for the built-in exporters (a.k.a “serializers”).