Important

You can run this notebook in a live session Binder or view it on nbviewer or GitHub.

Export data to files with Suitcase

In this notebook you will:

  • Export saved data to CSV and TIFF.

  • See a preview of how exporting data live during a scan works.

Configuration

Below, we will connect to EPICS IOC(s) controlling simulated hardware in lieu of actual motors, detectors. The IOCs should already be running in the background. Run this command to verify that they are running: it should produce output with RUNNING on each line. In the event of a problem, edit this command to replace status with restart all and run again.

[1]:
!supervisorctl -c supervisor/supervisord.conf status
decay                            RUNNING   pid 4977, uptime 0:00:34
mini_beamline                    RUNNING   pid 4978, uptime 0:00:34
random_walk                      RUNNING   pid 4979, uptime 0:00:34
random_walk_horiz                RUNNING   pid 4980, uptime 0:00:34
random_walk_vert                 RUNNING   pid 4981, uptime 0:00:34
simple                           RUNNING   pid 4982, uptime 0:00:34
thermo_sim                       RUNNING   pid 4983, uptime 0:00:34
trigger_with_pc                  RUNNING   pid 4984, uptime 0:00:34
[2]:
%run scripts/beamline_configuration.py
/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pims/image_reader.py:26: RuntimeWarning: PIMS image_reader.py could not find scikit-image. Falling back to matplotlib's imread(), which uses floats instead of integers. This may break your scripts.
(To ignore this warning, include the line "warnings.simplefilter("ignore", RuntimeWarning)" in your script.)
  warnings.warn(RuntimeWarning(ski_preferred))

Scalar data to CSV

[3]:
RE(scan([ph], motor_ph, -1, 1, 5), sample_comp='Ni', barcode='12345')


Transient Scan ID: 3     Time: 2020-07-15 20:25:41
Persistent Unique Scan ID: '6de6abef-a4fc-4551-85bf-f417d2629f52'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |   motor_ph |     ph_det |
+-----------+------------+------------+------------+
|         1 | 20:25:41.7 |     -1.000 |      98259 |
|         2 | 20:25:41.7 |     -0.500 |     101411 |
|         3 | 20:25:41.8 |      0.000 |     101700 |
|         4 | 20:25:41.8 |      0.500 |     102199 |
|         5 | 20:25:41.8 |      1.000 |     101337 |
+-----------+------------+------------+------------+
generator scan ['6de6abef'] (scan num: 3)



[3]:
('6de6abef-a4fc-4551-85bf-f417d2629f52',)
[4]:
import suitcase.csv
[5]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/')
[5]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/6de6abef-a4fc-4551-85bf-f417d2629f52-primary.csv')]}
[6]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}-{start[barcode]}-')
[6]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni-12345-primary.csv')]}
[7]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-')
[7]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345-primary.csv')]}

Image data to TIFF

[8]:
RE(scan([spot], mtr_spotx, -1, 1, 5), sample_comp='Ni', barcode='12345')


Transient Scan ID: 4     Time: 2020-07-15 20:25:42
Persistent Unique Scan ID: 'c4f18a56-0ddf-46d8-9d0c-7e45243452f8'
New stream: 'primary'
+-----------+------------+-------------+------------+
|   seq_num |       time | motor_spotx |   spot_roi |
+-----------+------------+-------------+------------+
|         1 | 20:25:42.3 |          -1 | 1125959588 |
|         2 | 20:25:42.4 |          -0 | 1128811593 |
|         3 | 20:25:42.5 |           0 | 1131563040 |
|         4 | 20:25:42.7 |           0 | 1131637043 |
|         5 | 20:25:42.7 |           1 | 1130232330 |
+-----------+------------+-------------+------------+
generator scan ['c4f18a56'] (scan num: 4)



[8]:
('c4f18a56-0ddf-46d8-9d0c-7e45243452f8',)

TIFF stack, a 3D block of images in one file

[9]:
import suitcase.tiff_stack
[10]:
suitcase.tiff_stack.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-')
[10]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345-primary-spot_img.tiff')]}

TIFF series

[11]:
import suitcase.tiff_series

Unlike CSV or TIFF stack, this exporter creates separate file per row (Event document). We can include information from a specific Event in the filename. Here, we include the motor position.

[12]:
suitcase.tiff_series.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}/{event[data][motor_spotx]}-')
[12]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/-1.0-primary-spot_img-0.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/-0.5-primary-spot_img-1.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/0.0-primary-spot_img-2.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/0.5-primary-spot_img-3.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/1.0-primary-spot_img-4.tiff')]}

“Beta” feature: Set up live exporting

We plan to streamline this in the future, but here is a demonstration that it is possible.

[13]:
# Execute this once to configure RE to send data to an exporter.
# At the beamline, this would normally be done in a script automatically at startup.

from event_model import RunRouter
from databroker.core import discover_handlers

def factory(name, start_doc):

    serializer = suitcase.tiff_series.Serializer(
        'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-{start[uid]:.8}-')

    def cb(name, doc):
        serializer(name, doc)

    return [cb], []


rr = RunRouter([factory], discover_handlers())
RE.subscribe(rr)
[13]:
2
[14]:
RE(count([spot], num=30, delay=1), sample_comp='Lab6', barcode='12345')


Transient Scan ID: 5     Time: 2020-07-15 20:25:43
Persistent Unique Scan ID: 'bef9355f-99b4-440d-95f6-88666ae33e71'
New stream: 'primary'
+-----------+------------+------------+
|   seq_num |       time |   spot_roi |
+-----------+------------+------------+
|         1 | 20:25:43.3 | 1115666264 |
|         2 | 20:25:44.2 | 1059859068 |
|         3 | 20:25:45.2 | 1071035239 |
|         4 | 20:25:46.2 | 1125503138 |
|         5 | 20:25:47.2 | 1113456003 |
|         6 | 20:25:48.2 | 1059150825 |
|         7 | 20:25:49.2 | 1072062125 |
|         8 | 20:25:50.2 | 1126220599 |
|         9 | 20:25:51.2 | 1112400277 |
|        10 | 20:25:52.2 | 1062010708 |
|        11 | 20:25:53.2 | 1071401585 |
|        12 | 20:25:54.2 | 1125918936 |
|        13 | 20:25:55.2 | 1113041835 |
|        14 | 20:25:56.2 | 1058843240 |
|        15 | 20:25:57.2 | 1072599720 |
|        16 | 20:25:58.2 | 1126500827 |
|        17 | 20:25:59.2 | 1111881834 |
|        18 | 20:26:00.2 | 1061603456 |
|        19 | 20:26:01.2 | 1071944449 |
|        20 | 20:26:02.2 | 1126125128 |
|        21 | 20:26:03.2 | 1112424290 |
|        22 | 20:26:04.2 | 1058490516 |
|        23 | 20:26:05.2 | 1073122933 |
|        24 | 20:26:06.2 | 1126850183 |
|        25 | 20:26:07.2 | 1111395110 |
|        26 | 20:26:08.2 | 1057912967 |
|        27 | 20:26:09.2 | 1069029781 |
|        28 | 20:26:10.2 | 1126484112 |
|        29 | 20:26:11.2 | 1111975828 |
|        30 | 20:26:12.2 | 1058184692 |
+-----------+------------+------------+
generator count ['bef9355f'] (scan num: 5)



[14]:
('bef9355f-99b4-440d-95f6-88666ae33e71',)
[ ]: