databroker
1.2.5.post15+g5fbc5eb1

Tutorials

  • Installation Tutorial
  • Find Runs in a Catalog
  • Get Data from a Run
  • Navigate Metadata in a Run
    • Set up for Tutorial
    • (Hardware) Configuration
    • How It Started
    • How It Ended
    • Low-level Hardware Timestamps
  • Export Data

How-to Guides

  • How to download some Catalogs with data samples
  • How to create a new Catalog backed by files
  • How to store data from the Run Engine
  • How to store analysis results
  • How to create a new Catalog backed by MongoDB

Explanations

  • What are the API versions v0, v1, v2?
  • How can it be used locally and remotely?
  • What is Databroker’s relationship to Intake?

Reference

  • New (v2) API
  • Original (v1 / v0) API
  • Configuration Reference
  • Changelog

Bluesky Project

  • Homepage
  • GitHub

Getting Help

  • Gitter
databroker
  • »
  • Navigate Metadata in a Run
  • View page source

Navigate Metadata in a Run¶

In this tutorial we will access secondary measurements and metadata including:

  • Hardware configuration readings (e.g. exposure time)

  • User-provided context like sample information

  • Whether the Run completed with an error (and if so what error)

  • Hardware-level timestamps for each measurement

Set up for Tutorial¶

Before you begin, install databroker and databroker-pack, following the Installation Tutorial.

Start your favorite interactive Python environment, such as ipython or jupyter lab.

For this tutorial, we’ll use a catalog of publicly available, openly licensed sample data. Specifically, it is high-quality transmission XAS data from all over the periodical table.

This utility downloads it and makes it discoverable to Databroker.

In [1]: import databroker.tutorial_utils

In [2]: databroker.tutorial_utils.fetch_BMM_example()
Out[2]: bluesky-tutorial-BMM:
  args:
    name: bluesky-tutorial-BMM
    paths:
    - /home/runner/.local/share/bluesky_tutorial_data/bluesky-tutorial-BMM/documents/*.msgpack
    root_map: {}
  description: ''
  driver: databroker._drivers.msgpack.BlueskyMsgpackCatalog
  metadata:
    catalog_dir: /home/runner/.local/share/intake/
    generated_by:
      library: databroker_pack
      version: 0.3.0
    relative_paths:
    - ./documents/*.msgpack

Access the catalog and assign it to a variable for convenience.

In [3]: import databroker

In [4]: catalog = databroker.catalog['bluesky-tutorial-BMM']

Let’s take a Run from this Catalog.

In [5]: run = catalog[23463]

(Hardware) Configuration¶

The Run may include configurational readings necessary for interpreting the data. These are typically things that change slowly or not at all during the Run, like detector exposure time, detector gain settings, or the configured maximum motor velocity.

First, let’s look at the I0 readings in the primary stream. What are the configuration readings that might be necessary to interpret this data or compare it with other data?

In [6]: da = run.primary.read()["I0"]

In [7]: da.head()
Out[7]: 
<xarray.DataArray 'I0' (time: 5)>
array([135.16644077, 134.97916279, 134.78557224, 134.63573771,
       134.70294218])
Coordinates:
  * time     (time) float64 1.584e+09 1.584e+09 1.584e+09 1.584e+09 1.584e+09
Attributes:
    object:   quadem1

This section at the bottom of that summary

Attributes:
    object:   quadem1

is showing us that I0 was measured by the device quadem1. We can also access that programmatically like

In [8]: da.attrs.get("object")
Out[8]: 'quadem1'

We can then look up all the configuration readings associated with quadem1 in this stream.

In [9]: run.primary.config["quadem1"].read()
Out[9]: 
<xarray.Dataset>
Dimensions:                   (time: 411)
Coordinates:
  * time                      (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
Data variables:
    quadem1_integration_time  (time) float64 0.0004 0.0004 ... 0.0004 0.0004
    quadem1_averaging_time    (time) float64 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0
    quadem1_em_range          (time) <U6 '350 pC' '350 pC' ... '350 pC' '350 pC'
    quadem1_values_per_read   (time) int64 5 5 5 5 5 5 5 5 5 ... 5 5 5 5 5 5 5 5
    quadem1_num_averaged      (time) int64 250 250 250 250 ... 250 250 250 250

If another Run ran the quadem1 detector with a different integration time, we could use this information to normalize the readings and compare them accurately.

TO DO: Get an example of that.

Let’s look at some other readings in the dataset. The It also comes from quadem1, so those same configuration readings apply.

In [10]: ds["It"].attrs
Out[10]: {'object': 'quadem1'}

The dcm_energy``readings, on the other hand, comes from a different device, which happens to also be named ``dcm_energy.

In [11]: ds["dcm_energy"].attrs
Out[11]: {'object': 'dcm_energy'}

We can see that no configuration was recorded for that device.

In [12]: run.primary.config["dcm_energy"].read()
Out[12]: 
<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*

How It Started¶

There are many useful pieces of metadata that we know at the start, before we begin acquiring data or running data processing/analysis. This includes what we intend to do (i.e. which scan type or which data processing routine), who is doing it, and any additional context like sample information.

The only fields guaranteed by Databroker to be present are uid (a globally unique identifier for the Run) and time (when it started) but there is often a great deal more.

>>> run.metadata["start"]
Start({
'XDI': {'Beamline': {'collimation': 'paraboloid mirror, 5 nm Rh on 30 nm Pt',
                   'focusing': 'not in use',
                   'harmonic_rejection': 'flat mirror, Pt stripe, pitch = '
                                         '7.0 mrad relative to beam',
                   'name': 'BMM (06BM) -- Beamline for Materials '
                           'Measurement',
                   'xray_source': 'NSLS-II three-pole wiggler'},
      'Column': {},
      'Detector': {'I0': '10 cm N2', 'Ir': '25 cm N2', 'It': '25 cm N2'},
      'Element': {'edge': 'K', 'symbol': 'Ni'},
      'Facility': {'GUP': 305832,
                   'SAF': 305669,
                   'current': '399.6',
                   'cycle': '2020-1',
                   'energy': '3.0',
                   'mode': 'top-off',
                   'name': 'NSLS-II'},
      'Mono': {'angle_offset': 16.058109,
               'd_spacing': '3.1353241',
               'direction': 'forward',
               'encoder_resolution': 5e-06,
               'name': 'Si(111)',
               'scan_mode': 'fixed exit',
               'scan_type': 'step'},
      'Sample': {'name': 'Ni', 'prep': 'Ni foil in ref'},
      'Scan': {'edge_energy': 8332.800000000001,
               'experimenters': 'Neil Hyatt, Martin Stennett, Dan Austin, '
                                'Seb Lawson'},

 ...  # snipped for brevity
 }

How It Ended¶

There are other things we can only know at the stop (end) of an experiment, including when and how it finished and how many events (rows) of data were collected in each stream.

In [13]: run.metadata["stop"]
Out[13]: 
Stop({'exit_status': 'success',
 'num_events': {'baseline': 2, 'primary': 411},
 'run_start': '4393404b-8986-4c75-9a64-d7f6949a9344',
 'time': 1583577680.5466747,
 'uid': '1548ec1e-1a01-4df2-9a2b-e17927971c58'})

We can use this to print the unique IDs of any experiments that failed

In [14]: for uid, run in catalog.items():
   ....:     if run.metadata["stop"]["exit_status"] != "success":
   ....:         print(f"Run {uid} failed!")
   ....: 

or, getting a bit fancier, to tally the number of failures.

In [15]: from collections import Counter

In [16]: counter = Counter()

In [17]: for _, run in catalog.items():
   ....:     counter.update({run.metadata["stop"]["exit_status"]: 1})
   ....: 

In [18]: counter
Out[18]: Counter({'success': 123})

TO DO: Obtain an example catalog that has some failures in it so that this example is not so trivial.

Low-level Hardware Timestamps¶

Note

Any preicse timing measurements should be in the data itself, not in this supplemental hardware timestamp metadata. This should generally be considered good for ~0.1 second precision alignment.

Control systems provide us with individually timestamps for every reading. These should generally not be used for data analysis. Any timing readings necessary for analysis should be recorded as data, as a column in some stream. These are intended to be used for debugging and troubleshooting.

The timestamps associated with the readings in run.primary.read() are available as

In [19]: run.primary.timestamps.read()
Out[19]: 
<xarray.Dataset>
Dimensions:                   (time: 411)
Coordinates:
  * time                      (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
Data variables:
    I0                        (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    It                        (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    Ir                        (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    dwti_dwell_time           (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    dwti_dwell_time_setpoint  (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    dcm_energy                (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    dcm_energy_setpoint       (time) float64 1.584e+09 1.584e+09 ... 1.584e+09

Configuration readings also come with timestamps. The timestamps associated with the configuration readings in run.primary.config["quadem1"].read() are available as

In [20]: run.primary.config_timestamps["quadem1"].read()
Out[20]: 
<xarray.Dataset>
Dimensions:                   (time: 411)
Coordinates:
  * time                      (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
Data variables:
    quadem1_integration_time  (time) float64 1.578e+09 1.578e+09 ... 1.578e+09
    quadem1_averaging_time    (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
    quadem1_em_range          (time) float64 1.578e+09 1.578e+09 ... 1.578e+09
    quadem1_values_per_read   (time) float64 1.578e+09 1.578e+09 ... 1.578e+09
    quadem1_num_averaged      (time) float64 1.584e+09 1.584e+09 ... 1.584e+09
Previous Next

© Copyright 2015 Brookhaven National Lab.

Built with Sphinx using a theme provided by Read the Docs.