Data Model

A primary design goal of bluesky is to enable better research by recording rich metadata alongside measured data for use in later analysis. Documents are how we do this.

A document is our term for a Python dictionary with a schema. The bluesky RunEngine emits documents during plan execution. All of the metadata and data generated by executing the plan is organized into documents. Bluesky’s document-based data model supports complex, asynchronous data collection and enables sophisticated live, prompt, streaming, and post-facto data analysis.

The bluesky documentation describes how outside functions can “subscribe” to a stream of these documents, visualizing, processing, or saving them. This section provides an outline of documents themselves, aiming to give a sense of the structure and familiarity with useful components.

Overview

The data model is composed of six types of Documents, which in Python are represented as dictionaries but could be represented as nested mappings (e.g. JSON) in any language. Each document class has a defined, but flexible, schema.

  • Run Start Document — Everything we know about an experiment or simulation before any data acquisition begins: the who / why / what and metadata such as sample information.

  • Event — A “row” of measurements with associated timestamps.

  • Event Descriptor — Metadata about a series of Events. Envision richly-detail column headings in a table, encompassing physical units, hardware configuration information, etc.

  • Resource — A pointer to an external file (or resource in general).

  • Datum — A pointer to a specific slice of data within a Resource.

  • Run Stop Document — Everything that we can only know at the very end, such as the time it ended and the exit status (succeeded, aborted, failed due to error).

Every document contains a unique identifier. The Event documents also have a descriptor field linking them to the Event Descriptor with their metadata. And the Event Descriptor and Run Stop documents have a run_start field linking them to their Run Start. Thus, all the documents in a run are linked back to the Run Start.

Event documents may contain the literal values or pointers to values that are stored in some external file or networked resource, yet to be loaded. The Resource and Datum document types manage references to externally-stored data.

Example Runs

_images/document-generation-timeline.svg

Finally, Event and Datum can be represented in “paged” form, where multiple rows are contained in one structure for efficient transport and vectorized computation. The representations contain equivalent information: an EventPage can always be losslessly transformed into an Event and vice versa.

The scope of a “run” is left to the instrument team or individual scientist. It is quite generic: a set of Documents generated by following a given sequence of instructions. This might encompass a single short measurement (say, “a count”) or a multi-step procedure such as a raster scan. A run should represent a set of individual measurements that will be collected or processed together.

Document Types in Detail

For each type, we will show:

  • the minimal nontrivial example that satisfies the schema

  • one or more “typical” examples generated by bluesky’s RunEngine during an experiment

  • the formal schema, encoded using JSON schema

Run Start Document

Again, a ‘run start’ document marks the beginning of the run. It comprises everything we know before we start taking data, including all metadata provided by the user and the plan.

Minimal nontrivial valid example:

Documentation note: It might seem more natural to use json code-blocks here than python ones, but using python allows us to include comments in line.

# 'run start' document
{'time': 1550069716.5092213,  # UNIX epoch (seconds since 1 Jan 1970)
 'uid': '10bf6945-4afd-43ca-af36-6ad8f3540bcd'}  # globally unique ID

A typical example

# 'run start' document
{'detectors': ['random_walk:x'],
 'hints': {'dimensions': [(['random_walk:dt'], 'primary')]},
 'motors': ('random_walk:dt',),
 'num_intervals': 2,
 'num_points': 3,
 'plan_args': {'args': ["EpicsSignal(read_pv='random_walk:dt', " "name='random_walk:dt', " 'value=1.0, ' 'timestamp=1550070001.828528, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:dt', " 'limits=False, ' 'put_complete=False)', -1, 1],
               'detectors': ["EpicsSignal(read_pv='random_walk:x', " "name='random_walk:x', " 'value=1.61472277847348, ' 'timestamp=1550070000.807677, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:x', " 'limits=False, ' 'put_complete=False)'],
               'num': 3,
               'per_step': 'None'},
 'plan_name': 'scan',
 'plan_pattern': 'inner_product',
 'plan_pattern_args': {'args': ["EpicsSignal(read_pv='random_walk:dt', " "name='random_walk:dt', " 'value=1.0, ' 'timestamp=1550070001.828528, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:dt', " 'limits=False, ' 'put_complete=False)', -1, 1],
                       'num': 3},
 'plan_pattern_module': 'bluesky.plan_patterns',
 'plan_type': 'generator',
 'scan_id': 2,
 'time': 1550070004.9850419,
 'uid': 'ba1f9076-7925-4af8-916e-0e1eaa1b3c47'}

Note

Time is given in UNIX time (seconds since 1970). Software for looking at the data would, of course, translate that into a more human-readable form.

The run start document formal schema:

{
    "definitions": {
        "data_type": {
            "title" : "data_type",
            "patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
            "additionalProperties": false
        }
    },
    "properties": {
        "project": {
            "type": "string",
            "description": "Name of project that this run is part of"
        },
        "sample": {
            "type": ["object", "string"],
            "description": "Information about the sample, may be a UID to another collection"
        },
        "scan_id": {
            "type": "integer",
            "description": "Scan ID number, not globally unique"
        },
        "time": {
            "type": "number",
            "description": "Time the run started.  Unix epoch time"
        },
        "uid": {
            "type": "string",
            "description": "Globally unique ID for this run"
        },
        "group": {
            "type": "string",
            "description": "Unix group to associate this data with"
        },
        "owner": {
            "type": "string",
            "description": "Unix owner to associate this data with"
        },
        "hints": {
            "type": "object",
            "description": "Start-level hints",
            "properties": {
                "dimensions": {
                    "type": "array",
                    "description": "The independent axes of the experiment.  Ordered slow to fast",
                    "items": {
                        "type": "array",
                        "description": "Each entry in this list is of the from ([<FIELD>, ...], <STREAM>).  A 1d scan will have 1 such entry, a scan with 3 independent entries would have 3",
                        "items": [
                            {
                                "type": "array",
                                "description": "The data key(s) for the given dimension.",
                                "items":
                                {
                                    "type": "string"
                                }
                            },
                            {
                                "type": "string",
                                "description": "The stream to find the datakeys in."

                            }
                        ],
                        "additionalItems": false,
                        "minItems": 2

                    }
                }
            },
            "patternProperties": {
                "^([^.]+)$": {"$ref": "#/definitions/data_type"}},
            "additionalProperties": false
        }
    },
    "patternProperties": {
        "^([^./]+)$": {"$ref": "#/definitions/data_type"}},
    "additionalProperties": false,
    "required": [
        "uid",
        "time"
     ],
    "type": "object",
    "description": "Document created at the start of run.  Provides a seach target and later documents link to it"
}

Event Descriptor

As stated above, an ‘event descriptor’ document provides a schema for the data in the Event documents. It provides useful information about each key in the data and about the configuration of the hardware. The layout of a descriptor is detailed and takes some time to cover, so we defer it to a later section.

Minimal nontrivial valid example:

# 'event descriptor' document
{'configuration': {},
 'data_keys': {'camera_image': {'dtype': 'number',
                                'shape': [512, 512],
                                'source': 'PV:...'}},
 'hints': {},
 'name': 'primary',
 'object_keys': {},
 'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd',  # foreign key
 'time': 1550070954.276659,
 'uid': 'd08d2ada-5f4e-495b-8e73-ff36186e7183'}

Typical example:

# 'event descriptor' document
{'configuration': {'random_walk:dt': {'data': {'random_walk:dt': -1.0},
                                      'data_keys': {'random_walk:dt': {'dtype': 'number',
                                                                       'lower_ctrl_limit': 0.0,
                                                                       'precision': 0,
                                                                       'shape': [],
                                                                       'source': 'PV:random_walk:dt',
                                                                       'units': '',
                                                                       'upper_ctrl_limit': 0.0}},
                                      'timestamps': {'random_walk:dt': 1550070004.994477}},
                   'random_walk:x': {'data': {'random_walk:x': 1.9221013521832928},
                                     'data_keys': {'random_walk:x': {'dtype': 'number',
                                                                     'lower_ctrl_limit': 0.0,
                                                                     'precision': 0,
                                                                     'shape': [],
                                                                     'source': 'PV:random_walk:x',
                                                                     'units': '',
                                                                     'upper_ctrl_limit': 0.0}},
                                     'timestamps': {'random_walk:x': 1550070004.812525}}},
 'data_keys': {'random_walk:dt': {'dtype': 'number',
                                  'lower_ctrl_limit': 0.0,
                                  'object_name': 'random_walk:dt',
                                  'precision': 0,
                                  'shape': [],
                                  'source': 'PV:random_walk:dt',
                                  'units': '',
                                  'upper_ctrl_limit': 0.0},
               'random_walk:x': {'dtype': 'number',
                                 'lower_ctrl_limit': 0.0,
                                 'object_name': 'random_walk:x',
                                 'precision': 0,
                                 'shape': [],
                                 'source': 'PV:random_walk:x',
                                 'units': '',
                                 'upper_ctrl_limit': 0.0}},
 'hints': {'random_walk:dt': {'fields': ['random_walk:dt']},
           'random_walk:x': {'fields': ['random_walk:x']}},
 'name': 'primary',
 'object_keys': {'random_walk:dt': ['random_walk:dt'],
                 'random_walk:x': ['random_walk:x']},
 'run_start': 'ba1f9076-7925-4af8-916e-0e1eaa1b3c47',
 'time': 1550070005.0109222,
 'uid': '0ad55d9e-1b31-4af2-865c-7ab7c8171303'}

Formal schema:

{
    "definitions": {
        "data_type": {
            "title" : "data_type",
            "patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
            "additionalProperties": false
        }
    },
    "properties": {
        "project": {
            "type": "string",
            "description": "Name of project that this run is part of"
        },
        "sample": {
            "type": ["object", "string"],
            "description": "Information about the sample, may be a UID to another collection"
        },
        "scan_id": {
            "type": "integer",
            "description": "Scan ID number, not globally unique"
        },
        "time": {
            "type": "number",
            "description": "Time the run started.  Unix epoch time"
        },
        "uid": {
            "type": "string",
            "description": "Globally unique ID for this run"
        },
        "group": {
            "type": "string",
            "description": "Unix group to associate this data with"
        },
        "owner": {
            "type": "string",
            "description": "Unix owner to associate this data with"
        },
        "hints": {
            "type": "object",
            "description": "Start-level hints",
            "properties": {
                "dimensions": {
                    "type": "array",
                    "description": "The independent axes of the experiment.  Ordered slow to fast",
                    "items": {
                        "type": "array",
                        "description": "Each entry in this list is of the from ([<FIELD>, ...], <STREAM>).  A 1d scan will have 1 such entry, a scan with 3 independent entries would have 3",
                        "items": [
                            {
                                "type": "array",
                                "description": "The data key(s) for the given dimension.",
                                "items":
                                {
                                    "type": "string"
                                }
                            },
                            {
                                "type": "string",
                                "description": "The stream to find the datakeys in."

                            }
                        ],
                        "additionalItems": false,
                        "minItems": 2

                    }
                }
            },
            "patternProperties": {
                "^([^.]+)$": {"$ref": "#/definitions/data_type"}},
            "additionalProperties": false
        }
    },
    "patternProperties": {
        "^([^./]+)$": {"$ref": "#/definitions/data_type"}},
    "additionalProperties": false,
    "required": [
        "uid",
        "time"
     ],
    "type": "object",
    "description": "Document created at the start of run.  Provides a seach target and later documents link to it"
}

Event Document

The Event document may contain data directly:

# 'event' document
{'data': {'camera_image': [[...512x512 array...]]},
 'descriptor': 'd08d2ada-5f4e-495b-8e73-ff36186e7183',  # foreign key
 'filled': {},
 'seq_num': 1,
 'time': 1550072091.2793343,
 'timestamps': {'camera_image': 1550072091.2793014},
 'uid': '8eac2f83-2b3e-4d67-ae2c-1d3aaff29ff5'}

or it may reference it via a datum_id from a Datum document.

# 'event' document
{'data': {'camera_image': '272132cf-564f-428f-bf6b-149ee4287024/1'},  # foreign key
 'descriptor': 'd08d2ada-5f4e-495b-8e73-ff36186e7183',  # foreign key
 'filled': {'camera_image': False},
 'seq_num': 1,
 'time': 1550072091.2793343,
 'timestamps': {'camera_image': 1550072091.2793014},
 'uid': '8eac2f83-2b3e-4d67-ae2c-1d3aaff29ff5'}

Note the appearance of {'filled': {'camera_image': False}}, indicating that the value of camera image is a foreign key; the data is not “filled in”.

Typical example:

# 'event' document
{'data': {'random_walk:dt': -1.0,
          'random_walk:x': 1.9221013521832928},
 'descriptor': '0ad55d9e-1b31-4af2-865c-7ab7c8171303',
 'filled': {},
 'seq_num': 1,
 'time': 1550070005.0189056,
 'timestamps': {'random_walk:dt': 1550070004.994477,
                'random_walk:x': 1550070004.812525},
 'uid': '7b5343fe-dfd7-4884-bc18-a0b571ff60b7'}

From a data analysis perspective, these readings were simultaneous, but in actuality the occurred at separate times. The separate times of the individual readings are not thrown away (they are recorded in ‘timestamps’) but the overall event ‘time’ is often more useful.

Formal schema:

{
    "properties": {
        "data": {
            "type": "object",
            "description": "The actual measurement data"
        },
        "timestamps": {
            "type": "object",
            "description": "The timestamps of the individual measurement data"
        },
        "filled": {
            "type": "object",
            "additionalProperties": {"type": ["boolean", "string"]},
            "description": "Mapping each of the keys of externally-stored data to the boolean False, indicating that the data has not been loaded, or to foreign keys (moved here from 'data' when the data was loaded)"
        },
        "descriptor": {
            "type": "string",
            "description": "UID of the EventDescriptor to which this Event belongs"
        },
        "seq_num": {
            "type": "integer",
            "description": "Sequence number to identify the location of this Event in the Event stream"
        },
        "time": {
            "type": "number",
            "description": "The event time. This maybe different than the timestamps on each of the data entries."
        },
        "uid": {
            "type": "string",
            "description": "Globally unique identifier for this Event"
        }
    },
    "required": [
        "uid",
        "data",
        "timestamps",
        "time",
        "descriptor",
        "seq_num"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "event",
    "description": "Document to record a quanta of collected data"
}

Event contents can also be represented in “paged” form, where multiple rows are contained in one structure for efficient transport and vectorized computation. The representations contain equivalent information: an EventPage can always be losslessly transformed into an Event and vice versa.

{
    "definitions": {
        "dataframe": {
            "type": "object",
            "title": "dataframe",
            "description": "A DataFrame-like object",
            "additionalProperties": {
                "type": "array"
            }
        },
        "dataframe_for_filled": {
            "title": "dataframe_for_filled",
            "type": "object",
            "description": "A DataFrame-like object with boolean or string entries",
            "additionalProperties": {
                "type": "array",
                "items": {
                    "type": [
                        "boolean",
                        "string"
                    ]
                }
            }
        }
    },
    "properties": {
        "descriptor": {
            "type": "string",
            "description": "The UID of the EventDescriptor to which all of the Events in this page belong"
        },
        "data": {
            "$ref": "#/definitions/dataframe",
            "description": "The actual measurement data"
        },
        "timestamps": {
            "$ref": "#/definitions/dataframe",
            "description": "The timestamps of the individual measurement data"
        },
        "filled": {
            "$ref": "#/definitions/dataframe_for_filled",
            "description": "Mapping each of the keys of externally-stored data to an array containing the boolean False, indicating that the data has not been loaded, or to foreign keys (moved here from 'data' when the data was loaded)"
        },
        "seq_num": {
            "type": "array",
            "items": {
                "type": "integer"
            },
            "description": "Array of sequence numbers to identify the location of each Event in the Event stream"
        },
        "time": {
            "type": "array",
            "items": {
                "type": "number"
            },
            "description": "Array of Event times. This maybe different than the timestamps on each of the data entries"
        },
        "uid": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "Array of globally unique identifiers for each Event"
        }
    },
    "required": [
        "descriptor",
        "uid",
        "data",
        "timestamps",
        "time",
        "seq_num"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "event_page",
    "description": "Page of documents to record a quanta of collected data"
}

Run Stop Document

A ‘run stop’ document marks the end of the run. It contains metadata that is not known until the run completes.

The most commonly useful fields here are ‘time’ and ‘exit_status’.

Minimal nontrivial valid example:

# 'run stop' document
{'uid': '546cc556-5f69-46b5-bf36-587d8cfe67a9',
 'time': 1550072737.175858,
 'run_start': '61bb1db8-c95c-4144-845b-e248c06d80e1',
 'exit_status': 'success',
 'reason': '',
 'num_events': {}}

Formal schema:

{
    "definitions": {
        "data_type": {
            "title" : "data_type",
            "patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
            "additionalProperties": false
        }
    },
    "properties": {
        "run_start": {
            "type": "string",
            "description": "Reference back to the run_start document that this document is paired with."
        },
        "reason": {
            "type": "string",
            "description": "Long-form description of why the run ended"
        },
        "time": {
            "type": "number",
            "description": "The time the run ended. Unix epoch"
        },
        "exit_status": {
            "type": "string",
            "enum": ["success", "abort", "fail"],
            "description": "State of the run when it ended"
        },
        "uid": {
            "type": "string",
            "description": "Globally unique ID for tihs run"
        },
        "num_events": {
            "type": "object",
            "properties": {},
            "additionalProperties": { "type": "integer" },
            "description": "Number of Events per named stream"
        }
    },
    "patternProperties": {
        "^([^./]+)$": {"$ref": "#/definitions/data_type"}},
    "additionalProperties": false,
    "required": [
        "uid",
        "run_start",
        "time",
        "exit_status"
    ],
    "type": "object",
    "description": "Document for the end of a run indicating the success/fail state of the run and the end time"

}

Resource Document

Minimal nontrivial valid example:

# 'resource' document
{'path_semantics': 'posix',
 'resource_kwargs': {},
 'resource_path': '/local/path/subdirectory/data_file',
 'root': '/local/path/',
 'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd',
 'spec': 'SOME_SPEC',
 'uid': '272132cf-564f-428f-bf6b-149ee4287024'}

Formal schema:

{
    "properties": {
        "spec": {
            "type": "string",
            "description": "String identifying the format/type of this Resource, used to identify a compatible Handler"
        },
        "resource_path": {
            "type": "string",
            "description": "Filepath or URI for locating this resource"
        },
        "resource_kwargs": {
            "type": "object",
            "description": "Additional argument to pass to the Handler to read a Resource"
        },
        "root": {
            "type": "string",
            "description": "Subset of resource_path that is a local detail, not semantic."
        },
        "path_semantics": {
            "type": "string",
            "description": "Rules for joining paths",
            "enum": ["posix", "windows"]
        },
        "uid": {
            "type": "string",
            "description": "Globally unique identifier for this Resource"
        },
        "run_start": {
            "type": "string",
            "description": "Globally unique ID to the run_start document this resource is associated with."
        }
    },
    "required": [
        "spec",
        "resource_path",
        "resource_kwargs",
        "root",
        "uid"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "resource",
    "description": "Document to reference a collection (e.g. file or group of files) of externally-stored data"
}

Datum Document

Minimal nontrivial valid example:

# 'datum' document
{'resource': '272132cf-564f-428f-bf6b-149ee4287024',  # foreign key
 'datum_kwargs': {'index': 0},  # format-specific parameters
 'datum_id': '272132cf-564f-428f-bf6b-149ee4287024/1'}

Formal schema:

{
    "properties": {
        "datum_kwargs": {
            "type": "object",
            "description": "Arguments to pass to the Handler to retrieve one quanta of data"
        },
        "resource": {
            "type": "string",
            "description": "The UID of the Resource to which this Datum belongs"
        },
        "datum_id": {
            "type": "string",
            "description": "Globally unique identifier for this Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
        }
    },
    "required": [
        "datum_kwargs",
        "resource",
        "datum_id"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "datum",
    "description": "Document to reference a quanta of externally-stored data"
}

Like Events, Datum contents can also be represented in “paged” form, and the representations contain equivalent information.

{
    "definitions": {
        "dataframe": {
            "type": "object",
            "title": "dataframe",
            "description": "A DataFrame-like object",
            "additionalProperties": {
                "type": "array"
            }
        }
    },
    "properties": {
        "resource": {
            "type": "string",
            "description": "The UID of the Resource to which all Datums in the page belong"
        },
        "datum_kwargs": {
            "$ref": "#/definitions/dataframe",
            "description": "Array of arguments to pass to the Handler to retrieve one quanta of data"
        },
        "datum_id": {
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "Array unique identifiers for each Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
        }
    },
    "required": [
        "resource",
        "datum_kwargs",
        "datum_id"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "datum",
    "description": "Page of documents to reference a quanta of externally-stored data"
}

“Bulk Events” Document (DEPRECATED)

This is another representation of Events. This representation is deprecated. Use EventPage instead.

{
    "patternProperties": {
        "^.*$": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "description": "The actual measument data"
                    },
                    "timestamps": {
                        "type": "object",
                        "description": "The timestamps of the individual measument data"
                    },
                    "filled": {
                        "type": "object",
                        "description": "Mapping the keys of externally-stored data to a boolean indicating whether that data has yet been loaded"
                    },
                    "descriptor": {
                        "type": "string",
                        "description": "UID to point back to Descriptor for this event stream"
                    },
                    "seq_num": {
                        "type": "integer",
                        "description": "Sequence number to identify the location of this Event in the Event stream"
                    },
                    "time": {
                        "type": "number",
                        "description": "The event time.  This maybe different than the timestamps on each of the data entries"
                    },
                    "uid": {
                        "type": "string",
                        "description": "Globally unique identifier for this Event"
                    }
                },
                "required": [
                    "uid",
                    "data",
                    "timestamps",
                    "time",
                    "descriptor",
                    "seq_num"
                ],
                "additionalProperties": false,
                "type": "object",
                "title": "bulk_events",
                "description": "Document to record a quanta of collected data"
            }
        }
    }
}

“Bulk Datum” Document (DEPRECATED)

This is another representation of Datum. This representation is deprecated. Use DatumPage instead.

{
    "properties": {
        "datum_kwargs_list": {
            "type": "array",
            "items": {"type": "object"},
            "description": "Array of arguments to pass to the Handler to retrieve one quanta of data"
        },
        "resource": {
            "type": "string",
            "description": "UID of the Resource to which all these Datum documents belong"
        },
        "datum_ids": {
            "type": "array",
            "items": {"type": "string"},
            "description": "Globally unique identifiers for each Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
        }
    },
    "required": [
        "datum_kwargs",
        "resource",
        "datum_id"
    ],
    "additionalProperties": false,
    "type": "object",
    "title": "bulk_datum",
    "description": "Document to reference a quanta of externally-stored data"
}