Important

You can run this notebook in a live session Binder or view it on nbviewer or GitHub.

Export data to files with Suitcase

In this notebook you will:

  • Export saved data to CSV and TIFF.

  • See a preview of how exporting data live during a scan works.

Configuration

Below, we will connect to EPICS IOC(s) controlling simulated hardware in lieu of actual motors, detectors. The IOCs should already be running in the background. Run this command to verify that they are running: it should produce output with RUNNING on each line. In the event of a problem, edit this command to replace status with restart all and run again.

[1]:
!supervisorctl -c supervisor/supervisord.conf status
decay                            RUNNING   pid 4872, uptime 0:00:33
mini_beamline                    RUNNING   pid 4873, uptime 0:00:33
random_walk                      RUNNING   pid 4874, uptime 0:00:33
random_walk_horiz                RUNNING   pid 4875, uptime 0:00:33
random_walk_vert                 RUNNING   pid 4876, uptime 0:00:33
simple                           RUNNING   pid 4877, uptime 0:00:33
thermo_sim                       RUNNING   pid 4878, uptime 0:00:33
trigger_with_pc                  RUNNING   pid 4879, uptime 0:00:33
[2]:
%run scripts/beamline_configuration.py

Scalar data to CSV

[3]:
RE(scan([ph], motor_ph, -1, 1, 5), sample_comp='Ni', barcode='12345')


Transient Scan ID: 3     Time: 2020-06-05 15:43:50
Persistent Unique Scan ID: 'd8ae0bbb-7eff-4c9c-b69d-eaae176bcc46'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |   motor_ph |     ph_det |
+-----------+------------+------------+------------+
|         1 | 15:43:50.4 |     -1.000 |     100483 |
|         2 | 15:43:50.5 |     -0.500 |     101003 |
|         3 | 15:43:50.5 |      0.000 |     102432 |
|         4 | 15:43:50.5 |      0.500 |     101018 |
|         5 | 15:43:50.5 |      1.000 |      99833 |
+-----------+------------+------------+------------+
generator scan ['d8ae0bbb'] (scan num: 3)



[3]:
('d8ae0bbb-7eff-4c9c-b69d-eaae176bcc46',)
[4]:
import suitcase.csv
[5]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/')
[5]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/d8ae0bbb-7eff-4c9c-b69d-eaae176bcc46-primary.csv')]}
[6]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}-{start[barcode]}-')
[6]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni-12345-primary.csv')]}
[7]:
suitcase.csv.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-')
[7]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345-primary.csv')]}

Image data to TIFF

[8]:
RE(scan([spot], mtr_spotx, -1, 1, 5), sample_comp='Ni', barcode='12345')


Transient Scan ID: 4     Time: 2020-06-05 15:43:50
Persistent Unique Scan ID: '1c6c14ee-c535-48dd-a439-71b6ce3f7412'
New stream: 'primary'
+-----------+------------+-------------+------------+
|   seq_num |       time | motor_spotx |   spot_roi |
+-----------+------------+-------------+------------+
|         1 | 15:43:50.9 |          -1 | 1087286498 |
|         2 | 15:43:51.1 |          -0 | 1079961456 |
|         3 | 15:43:51.2 |           0 | 1068855855 |
|         4 | 15:43:51.3 |           0 | 1064117220 |
|         5 | 15:43:51.4 |           1 | 1059395309 |
+-----------+------------+-------------+------------+
generator scan ['1c6c14ee'] (scan num: 4)



[8]:
('1c6c14ee-c535-48dd-a439-71b6ce3f7412',)

TIFF stack, a 3D block of images in one file

[9]:
import suitcase.tiff_stack
[10]:
suitcase.tiff_stack.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-')
[10]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345-primary-spot_img.tiff')]}

TIFF series

[11]:
import suitcase.tiff_series

Unlike CSV or TIFF stack, this exporter creates separate file per row (Event document). We can include information from a specific Event in the filename. Here, we include the motor position.

[12]:
suitcase.tiff_series.export(db[-1].documents(fill=True), 'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}/{event[data][motor_spotx]}-')
[12]:
{'stream_data': [PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/-1.0-primary-spot_img-0.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/-0.5-primary-spot_img-1.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/0.0-primary-spot_img-2.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/0.5-primary-spot_img-3.tiff'),
  PosixPath('/home/travis/build/bluesky/tutorials/docs/source/my_exported_files/Ni/12345/1.0-primary-spot_img-4.tiff')]}

“Beta” feature: Set up live exporting

We plan to streamline this in the future, but here is a demonstration that it is possible.

[13]:
# Execute this once to configure RE to send data to an exporter.
# At the beamline, this would normally be done in a script automatically at startup.

from event_model import RunRouter
from databroker.core import discover_handlers

def factory(name, start_doc):

    serializer = suitcase.tiff_series.Serializer(
        'my_exported_files/', file_prefix='{start[sample_comp]}/{start[barcode]}-{start[uid]:.8}-')

    def cb(name, doc):
        serializer(name, doc)

    return [cb], []


rr = RunRouter([factory], discover_handlers())
RE.subscribe(rr)
[13]:
2
[14]:
RE(count([spot], num=30, delay=1), sample_comp='Lab6', barcode='12345')


Transient Scan ID: 5     Time: 2020-06-05 15:43:51
Persistent Unique Scan ID: '8c16b03b-8a0d-4dc6-ae52-936e6386497f'
New stream: 'primary'
+-----------+------------+------------+
|   seq_num |       time |   spot_roi |
+-----------+------------+------------+
|         1 | 15:43:52.0 | 1053310862 |
|         2 | 15:43:52.9 | 1099380007 |
|         3 | 15:43:53.9 | 1131152533 |
|         4 | 15:43:54.9 | 1085020740 |
|         5 | 15:43:55.9 | 1054050147 |
|         6 | 15:43:56.9 | 1100639105 |
|         7 | 15:43:57.9 | 1130875774 |
|         8 | 15:43:58.9 | 1083805497 |
|         9 | 15:43:59.9 | 1054355536 |
|        10 | 15:44:00.9 | 1095618138 |
|        11 | 15:44:01.9 | 1131050054 |
|        12 | 15:44:02.9 | 1084472856 |
|        13 | 15:44:03.9 | 1054103712 |
|        14 | 15:44:04.9 | 1101083209 |
|        15 | 15:44:05.9 | 1130749463 |
|        16 | 15:44:06.9 | 1083411167 |
|        17 | 15:44:07.9 | 1054372446 |
|        18 | 15:44:08.9 | 1102275463 |
|        19 | 15:44:09.9 | 1131626463 |
|        20 | 15:44:10.9 | 1083861636 |
|        21 | 15:44:11.9 | 1054298434 |
|        22 | 15:44:12.9 | 1101849920 |
|        23 | 15:44:13.9 | 1130582181 |
|        24 | 15:44:14.9 | 1082676780 |
|        25 | 15:44:15.9 | 1054644140 |
|        26 | 15:44:16.9 | 1103031846 |
|        27 | 15:44:17.9 | 1131511118 |
|        28 | 15:44:18.9 | 1083187138 |
|        29 | 15:44:19.9 | 1054504915 |
|        30 | 15:44:20.9 | 1102493192 |
+-----------+------------+------------+
generator count ['8c16b03b'] (scan num: 5)



[14]:
('8c16b03b-8a0d-4dc6-ae52-936e6386497f',)
[ ]: