Important

You can run this notebook in a live session Binder or view it on nbviewer or GitHub.

Let Us Do the Bookkeeping For You

In this notebook you will:

  • Run some simulated experiments and then access the metadata about them.

  • Use that metadata to generate a summary report.

  • Use it filter search results.

  • Explore some of Python’s string formatting features in detail.

Configuration

Below, we will connect to EPICS IOC(s) controlling simulated hardware in lieu of actual motors, detectors. The IOCs should already be running in the background. Run this command to verify that they are running: it should produce output with RUNNING on each line. In the event of a problem, edit this command to replace status with restart all and run again.

[1]:
!supervisorctl -c supervisor/supervisord.conf status
decay                            RUNNING   pid 4977, uptime 0:02:13
mini_beamline                    RUNNING   pid 4978, uptime 0:02:13
random_walk                      RUNNING   pid 4979, uptime 0:02:13
random_walk_horiz                RUNNING   pid 4980, uptime 0:02:13
random_walk_vert                 RUNNING   pid 4981, uptime 0:02:13
simple                           RUNNING   pid 4982, uptime 0:02:13
thermo_sim                       RUNNING   pid 4983, uptime 0:02:13
trigger_with_pc                  RUNNING   pid 4984, uptime 0:02:13
[2]:
%run scripts/beamline_configuration.py
/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pims/image_reader.py:26: RuntimeWarning: PIMS image_reader.py could not find scikit-image. Falling back to matplotlib's imread(), which uses floats instead of integers. This may break your scripts.
(To ignore this warning, include the line "warnings.simplefilter("ignore", RuntimeWarning)" in your script.)
  warnings.warn(RuntimeWarning(ski_preferred))

Generate a Summary Report

Acquire some data like so. The details of what we are doing here are not important for what follows. If you want to know more about data acquisition, start with Hello Bluesky.

[3]:
RE(count([ph]))
RE(count([ph, edge, slit], 3))
RE(scan([edge], motor_edge, -10, 10, 15))
RE(scan([edge], motor_edge, -1, 3, 5))
RE(scan([ph], motor_ph, -1, 3, 5))
RE(scan([slit], motor_slit, -10, 10, 15))


Transient Scan ID: 13     Time: 2020-07-15 20:27:21
Persistent Unique Scan ID: '9f7453a7-d054-40e0-b5de-e052b31acee5'
New stream: 'primary'
+-----------+------------+------------+
|   seq_num |       time |     ph_det |
+-----------+------------+------------+
|         1 | 20:27:21.0 |      13087 |
+-----------+------------+------------+
generator count ['9f7453a7'] (scan num: 13)





Transient Scan ID: 14     Time: 2020-07-15 20:27:21
Persistent Unique Scan ID: '2a6c2513-c9a2-43c4-8d73-0741b68bfea9'
New stream: 'primary'
+-----------+------------+------------+------------+------------+
|   seq_num |       time |     ph_det |   edge_det |   slit_det |
+-----------+------------+------------+------------+------------+
|         1 | 20:27:21.5 |      12974 |        239 |      96344 |
|         2 | 20:27:21.5 |      13457 |        238 |      99828 |
|         3 | 20:27:21.5 |      13570 |        244 |     100842 |
+-----------+------------+------------+------------+------------+
generator count ['2a6c2513'] (scan num: 14)





Transient Scan ID: 15     Time: 2020-07-15 20:27:21
Persistent Unique Scan ID: '82489fe7-1528-41d5-a16a-f2493014adcd'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_edge |   edge_det |
+-----------+------------+------------+------------+
|         1 | 20:27:22.0 |    -10.000 |          0 |
|         2 | 20:27:22.1 |     -8.571 |          0 |
|         3 | 20:27:22.1 |     -7.143 |          0 |
|         4 | 20:27:22.1 |     -5.714 |          0 |
|         5 | 20:27:22.2 |     -4.286 |          0 |
|         6 | 20:27:22.2 |     -2.857 |          0 |
|         7 | 20:27:22.2 |     -1.429 |         16 |
|         8 | 20:27:22.2 |      0.000 |        249 |
|         9 | 20:27:22.2 |      1.429 |       2221 |
|        10 | 20:27:22.2 |      2.857 |      11863 |
|        11 | 20:27:22.2 |      4.286 |      35890 |
|        12 | 20:27:22.2 |      5.714 |      68941 |
|        13 | 20:27:22.2 |      7.143 |      93284 |
|        14 | 20:27:22.2 |      8.571 |     102068 |
|        15 | 20:27:22.3 |     10.000 |     103849 |
+-----------+------------+------------+------------+
generator scan ['82489fe7'] (scan num: 15)





Transient Scan ID: 16     Time: 2020-07-15 20:27:22
Persistent Unique Scan ID: '95c67ea7-1897-42f2-9662-9c313c0b0cd9'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_edge |   edge_det |
+-----------+------------+------------+------------+
|         1 | 20:27:22.7 |     -1.000 |         29 |
|         2 | 20:27:22.7 |      0.000 |        231 |
|         3 | 20:27:22.7 |      1.000 |       1268 |
|         4 | 20:27:22.7 |      2.000 |       4833 |
|         5 | 20:27:22.7 |      3.000 |      13713 |
+-----------+------------+------------+------------+
generator scan ['95c67ea7'] (scan num: 16)





Transient Scan ID: 17     Time: 2020-07-15 20:27:22
Persistent Unique Scan ID: '491e173c-1633-4bdd-89ca-e8b644181ddb'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |   motor_ph |     ph_det |
+-----------+------------+------------+------------+
|         1 | 20:27:23.2 |     -1.000 |     100591 |
|         2 | 20:27:23.2 |      0.000 |     102576 |
|         3 | 20:27:23.2 |      1.000 |     100098 |
|         4 | 20:27:23.3 |      2.000 |      93672 |
|         5 | 20:27:23.3 |      3.000 |      84828 |
+-----------+------------+------------+------------+
generator scan ['491e173c'] (scan num: 17)



/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/utils.py:1502: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all axes decorations.
  f_mgr.canvas.draw()


Transient Scan ID: 18     Time: 2020-07-15 20:27:23
Persistent Unique Scan ID: '45f4b5c2-6d95-4e99-884d-6449d560c8fd'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_slit |   slit_det |
+-----------+------------+------------+------------+
|         1 | 20:27:23.8 |    -10.000 |       7685 |
|         2 | 20:27:23.8 |     -8.571 |      26588 |
|         3 | 20:27:23.8 |     -7.143 |      56452 |
|         4 | 20:27:23.9 |     -5.714 |      82147 |
|         5 | 20:27:23.9 |     -4.286 |      94114 |
|         6 | 20:27:23.9 |     -2.857 |      96557 |
|         7 | 20:27:23.9 |     -1.429 |      96108 |
|         8 | 20:27:23.9 |      0.000 |      96300 |
|         9 | 20:27:23.9 |      1.429 |      96553 |
|        10 | 20:27:23.9 |      2.857 |      96712 |
|        11 | 20:27:24.0 |      4.286 |      93052 |
|        12 | 20:27:24.0 |      5.714 |      81619 |
|        13 | 20:27:24.0 |      7.143 |      55872 |
|        14 | 20:27:24.0 |      8.571 |      26224 |
|        15 | 20:27:24.0 |     10.000 |       7527 |
+-----------+------------+------------+------------+
generator scan ['45f4b5c2'] (scan num: 18)



[3]:
('45f4b5c2-6d95-4e99-884d-6449d560c8fd',)

Here a some code that prints a summary with some of the metadata automatically captured by Bluesky. Note the time filter added to the databroker object (db) - see Filtering for more about this feature.

[4]:
import time
from datetime import datetime

now = time.time()
an_hour_ago = now - 60 * 60 *24
print("HH:MM  plan_name  detectors      motors")
for h in db(since=an_hour_ago):
    md = h.start
    print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
          f"{md['plan_name']:11}"
          f"{','.join(md.get('detectors', [])):15}"
          f"{','.join(md.get('motors', [])):15}")
HH:MM  plan_name  detectors      motors
20:27  scan       ph             motor_ph
20:27  count      ph
20:27  scan       slit           motor_slit
20:27  count      ph,edge,slit
20:27  scan       edge           motor_edge
20:27  scan       edge           motor_edge

Let’s add one more example data, a run that failed because of a user error.

[5]:
# THIS IS EXPECTED TO CREATE AN ERROR.

RE(scan([motor_ph], ph, -1, 1, 3))  # oops I tried to use a detector as a motor
Run aborted
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py", line 1365, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1307, in __call__
    return (yield from plan)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 1160, in baseline_wrapper
    return (yield from plan)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 803, in monitor_during_wrapper
    return (yield from plan2)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 861, in fly_during_wrapper
    return (yield from plan2)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 170, in plan_mutator
    raise ex
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 123, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py", line 1101, in scan
    per_step=per_step, md=_md))
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py", line 995, in scan_nd
    return (yield from inner_scan_nd())
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/utils.py", line 1046, in dec_inner
    return (yield from plan)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 952, in stage_wrapper
    return (yield from finalize_wrapper(inner(), unstage_devices()))
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 509, in finalize_wrapper
    ret = yield from plan
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 950, in inner
    return (yield from plan)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/utils.py", line 1046, in dec_inner
    return (yield from plan)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 327, in run_wrapper
    else_plan=close_run)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 575, in contingency_wrapper
    ret = yield from plan
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py", line 993, in inner_scan_nd
    yield from per_step(detectors, step, pos_cache)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 1114, in one_nd_step
    yield from move_per_step(step, pos_cache)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plan_stubs.py", line 1084, in move_per_step
    yield Msg('set', motor, pos, group=grp)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py", line 195, in plan_mutator
    inner_ret = yield msg
  [Previous line repeated 1 more time]
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py", line 1425, in _run
    new_response = await coro(msg)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py", line 1921, in _set
    ret = msg.obj.set(*msg.args, **kwargs)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/ophyd/device.py", line 1197, in __getattr__
    raise AttributeError(name)
AttributeError: set


Transient Scan ID: 19     Time: 2020-07-15 20:27:24
Persistent Unique Scan ID: 'b677110c-28d4-4a8f-b2dc-72155b9083e9'



---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-2412d29f338e> in <module>
      1 # THIS IS EXPECTED TO CREATE AN ERROR.
      2
----> 3 RE(scan([motor_ph], ph, -1, 1, 3))  # oops I tried to use a detector as a motor

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806
--> 807         self._resume_task(init_func=_build_task)
    808
    809         if self._interrupted:

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932
    933     def install_suspender(self, suspender):

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308
   1309

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1158     if not devices:
   1159         # no-op
-> 1160         return (yield from plan)
   1161     else:
   1162         return (yield from plan_mutator(plan, insert_baseline))

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804
    805

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862
    863

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py in scan(detectors, num, per_step, md, *args)
   1099
   1100     return (yield from scan_nd(detectors, full_cycler,
-> 1101                                per_step=per_step, md=_md))
   1102
   1103

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py in scan_nd(detectors, cycler, per_step, md)
    993             yield from per_step(detectors, step, pos_cache)
    994
--> 995     return (yield from inner_scan_nd())
    996
    997

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1044                 plan = gen_func(*inner_args, **inner_kwargs)
   1045                 plan = wrapper(plan, *args, **kwargs)
-> 1046                 return (yield from plan)
   1047             return dec_inner
   1048         return dec

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953
    954

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1044                 plan = gen_func(*inner_args, **inner_kwargs)
   1045                 plan = wrapper(plan, *args, **kwargs)
-> 1046                 return (yield from plan)
   1047             return dec_inner
   1048         return dec

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in run_wrapper(plan, md)
    325     yield from contingency_wrapper(plan,
    326                                    except_plan=except_plan,
--> 327                                    else_plan=close_run)
    328     return rs_uid
    329

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in contingency_wrapper(plan, except_plan, else_plan, final_plan, pause_for_debug)
    573     cleanup = True
    574     try:
--> 575         ret = yield from plan
    576     except GeneratorExit:
    577         cleanup = False

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plans.py in inner_scan_nd()
    991     def inner_scan_nd():
    992         for step in list(cycler):
--> 993             yield from per_step(detectors, step, pos_cache)
    994
    995     return (yield from inner_scan_nd())

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plan_stubs.py in one_nd_step(detectors, step, pos_cache, take_reading)
   1112     """
   1113     motors = step.keys()
-> 1114     yield from move_per_step(step, pos_cache)
   1115     yield from take_reading(list(detectors) + list(motors))
   1116

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/plan_stubs.py in move_per_step(step, pos_cache)
   1082             # This step does not move this motor.
   1083             continue
-> 1084         yield Msg('set', motor, pos, group=grp)
   1085         pos_cache[motor] = pos
   1086     yield Msg('wait', None, group=grp)

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426
   1427                     # special case `CancelledError` and let the outer

~/virtualenv/python3.7.1/lib/python3.7/site-packages/bluesky/run_engine.py in _set(self, msg)
   1919         group = kwargs.pop('group', None)
   1920         self._movable_objs_touched.add(msg.obj)
-> 1921         ret = msg.obj.set(*msg.args, **kwargs)
   1922         p_event = asyncio.Event(loop=self._loop_for_kwargs)
   1923         pardon_failures = self._pardon_failures

~/virtualenv/python3.7.1/lib/python3.7/site-packages/ophyd/device.py in __getattr__(self, name)
   1195         # Components will be instantiated through the descriptor mechanism in
   1196         # the Component class, so anything reaching this point is an error.
-> 1197         raise AttributeError(name)
   1198
   1199     def _instantiate_component(self, attr):

AttributeError: set

We’ll add one more column to extract the ‘exit_status’ reported by RE before it errored out.

[6]:
print("HH:MM  plan_name  detectors      motors         exit_status")
for h in db(since=an_hour_ago):
    md = h.start
    print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
          f"{md['plan_name']:11}"
          f"{','.join(md.get('detectors', [])):15}"
          f"{','.join(md.get('motors', [])):15}"
          f"{h.stop['exit_status']}")
HH:MM  plan_name  detectors      motors         exit_status
20:27  scan       ph             motor_ph       success
20:27  count      ph                            success
20:27  scan       slit           motor_slit     success
20:27  count      ph,edge,slit                  success
20:27  scan       edge           motor_edge     success
20:27  scan       edge           motor_edge     success
20:27  scan       motor_ph       ph             fail

Let’s make it easier to reuse this code block by formulating it as a function.

[7]:
def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']}")


summarize_runs(db(since=an_hour_ago))
HH:MM  plan_name  detectors      motors         exit_status
20:27  scan       ph             motor_ph       success
20:27  count      ph                            success
20:27  scan       slit           motor_slit     success
20:27  count      ph,edge,slit                  success
20:27  scan       edge           motor_edge     success
20:27  scan       edge           motor_edge     success
20:27  scan       motor_ph       ph             fail

Getting a little fancy (Too fancy? Maybe….) you can print by default but optionally write to a text file instead.

[8]:
import functools

def summarize_runs(headers, write=functools.partial(print, end='')):
    write("HH:MM  plan_name  detectors      motors         exit_status\n")
    for h in headers:
        md = h.start
        write(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']}\n")


summarize_runs(db(since=an_hour_ago))  # prints as before
HH:MM  plan_name  detectors      motors         exit_status
20:27  scan       ph             motor_ph       success
20:27  count      ph                            success
20:27  scan       slit           motor_slit     success
20:27  count      ph,edge,slit                  success
20:27  scan       edge           motor_edge     success
20:27  scan       edge           motor_edge     success
20:27  scan       motor_ph       ph             fail
[9]:
summarize_runs(db(since=an_hour_ago), write=open('summary.txt', 'w').write)  # writes to 'summary.txt'
[10]:
# cat is a UNIX command for reading a text file. We could also just go open the file like normal people.
!cat summary.txt
HH:MM  plan_name  detectors      motors         exit_status
20:27  scan       ph             motor_ph       success
20:27  count      ph                            success
20:27  scan       slit           motor_slit     success
20:27  count      ph,edge,slit                  success
20:27  scan       edge           motor_edge     success
20:27  scan       edge           motor_edge     success
20:27  scan       motor_ph       ph             fail

If the user tells us more, our report can get richer

[11]:
RE.md['operator'] = 'Dan'

This applies to all future runs until deleted:

del RE.md['operator']

replaced

RE.md['operator'] = 'Tom'

or superceded

RE(count([ph]), operator='Maksim')

In that last example, 'Maksim' takes precedence over whatever is in RE.md, but just for this execution. If next we did

RE(count([ph]))

the operator would revert back to 'Tom'.

[12]:
# User reports the run's "purpose". (That isn't a special name... you can use any terms you want here...)
RE(count([ph]), purpose='test')
RE(count([ph, edge, slit], 3), purpose='test')
RE(scan([edge], motor_edge, -10, 10, 15), purpose='find edge')
RE(scan([edge], motor_edge, -1, 3, 5), purpose='find edge')
RE.md['operator'] = 'Tom'  # Tom takes over.
RE(scan([ph], motor_ph, -1, 3, 5), purpose='data')
RE(scan([slit], motor_slit, -10, 10, 15), purpose='data')


Transient Scan ID: 20     Time: 2020-07-15 20:27:25
Persistent Unique Scan ID: '5d4df1c9-91f6-41c4-bcfd-ccf88f8ed606'
New stream: 'primary'
+-----------+------------+------------+
|   seq_num |       time |     ph_det |
+-----------+------------+------------+
|         1 | 20:27:25.0 |      80990 |
+-----------+------------+------------+
generator count ['5d4df1c9'] (scan num: 20)





Transient Scan ID: 21     Time: 2020-07-15 20:27:25
Persistent Unique Scan ID: '9274320e-0434-44af-b8eb-33e0bc0485cf'
New stream: 'primary'
+-----------+------------+------------+------------+------------+
|   seq_num |       time |     ph_det |   edge_det |   slit_det |
+-----------+------------+------------+------------+------------+
|         1 | 20:27:25.2 |      81158 |      12589 |       7638 |
|         2 | 20:27:25.3 |      82096 |      12576 |       7692 |
|         3 | 20:27:25.4 |      82689 |      12893 |       7846 |
+-----------+------------+------------+------------+------------+
generator count ['9274320e'] (scan num: 21)





Transient Scan ID: 22     Time: 2020-07-15 20:27:25
Persistent Unique Scan ID: 'e762b53c-45ba-4ee6-8d5f-0cc87065e5c7'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_edge |   edge_det |
+-----------+------------+------------+------------+
|         1 | 20:27:25.9 |    -10.000 |          0 |
|         2 | 20:27:25.9 |     -8.571 |          0 |
|         3 | 20:27:26.0 |     -7.143 |          0 |
|         4 | 20:27:26.0 |     -5.714 |          0 |
|         5 | 20:27:26.0 |     -4.286 |          0 |
|         6 | 20:27:26.0 |     -2.857 |          1 |
|         7 | 20:27:26.0 |     -1.429 |         15 |
|         8 | 20:27:26.1 |      0.000 |        259 |
|         9 | 20:27:26.1 |      1.429 |       2234 |
|        10 | 20:27:26.1 |      2.857 |      11616 |
|        11 | 20:27:26.1 |      4.286 |      35793 |
|        12 | 20:27:26.1 |      5.714 |      68942 |
|        13 | 20:27:26.1 |      7.143 |      92681 |
|        14 | 20:27:26.2 |      8.571 |     102099 |
|        15 | 20:27:26.2 |     10.000 |     103698 |
+-----------+------------+------------+------------+
generator scan ['e762b53c'] (scan num: 22)





Transient Scan ID: 23     Time: 2020-07-15 20:27:26
Persistent Unique Scan ID: '3c85fd0e-de77-447a-957c-f277f2c71507'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_edge |   edge_det |
+-----------+------------+------------+------------+
|         1 | 20:27:26.6 |     -1.000 |         28 |
|         2 | 20:27:26.6 |      0.000 |        228 |
|         3 | 20:27:26.6 |      1.000 |       1251 |
|         4 | 20:27:26.6 |      2.000 |       4779 |
|         5 | 20:27:26.6 |      3.000 |      13557 |
+-----------+------------+------------+------------+
generator scan ['3c85fd0e'] (scan num: 23)





Transient Scan ID: 24     Time: 2020-07-15 20:27:26
Persistent Unique Scan ID: '3ac383a0-98c8-4b15-8359-33d67c91e4aa'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |   motor_ph |     ph_det |
+-----------+------------+------------+------------+
|         1 | 20:27:26.8 |     -1.000 |     103114 |
|         2 | 20:27:26.8 |      0.000 |     104762 |
|         3 | 20:27:26.8 |      1.000 |     102309 |
|         4 | 20:27:26.8 |      2.000 |      96456 |
|         5 | 20:27:26.9 |      3.000 |      87187 |
+-----------+------------+------------+------------+
generator scan ['3ac383a0'] (scan num: 24)





Transient Scan ID: 25     Time: 2020-07-15 20:27:27
Persistent Unique Scan ID: '3fd384e0-6a9c-4094-9ce6-f65cdf9b6a3c'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time | motor_slit |   slit_det |
+-----------+------------+------------+------------+
|         1 | 20:27:27.5 |    -10.000 |       7934 |
|         2 | 20:27:27.5 |     -8.571 |      27339 |
|         3 | 20:27:27.5 |     -7.143 |      58198 |
|         4 | 20:27:27.5 |     -5.714 |      84564 |
|         5 | 20:27:27.5 |     -4.286 |      96762 |
|         6 | 20:27:27.5 |     -2.857 |      99717 |
|         7 | 20:27:27.5 |     -1.429 |      99923 |
|         8 | 20:27:27.6 |      0.000 |      99392 |
|         9 | 20:27:27.6 |      1.429 |      99625 |
|        10 | 20:27:27.6 |      2.857 |      98719 |
|        11 | 20:27:27.6 |      4.286 |      94956 |
|        12 | 20:27:27.7 |      5.714 |      83424 |
|        13 | 20:27:27.7 |      7.143 |      57241 |
|        14 | 20:27:27.7 |      8.571 |      26824 |
|        15 | 20:27:27.7 |     10.000 |       7662 |
+-----------+------------+------------+------------+
generator scan ['3fd384e0'] (scan num: 25)



[12]:
('3fd384e0-6a9c-4094-9ce6-f65cdf9b6a3c',)
[13]:
def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status    purpose")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:15}"
              f"{md.get('purpose', '?')}")


summarize_runs(db(since=an_hour_ago))
HH:MM  plan_name  detectors      motors         exit_status    purpose
20:27  scan       ph             motor_ph       success        data
20:27  scan       ph             motor_ph       success        ?
20:27  count      ph                            success        ?
20:27  scan       slit           motor_slit     success        data
20:27  scan       edge           motor_edge     success        find edge
20:27  count      ph                            success        test
20:27  scan       edge           motor_edge     success        find edge
20:27  scan       slit           motor_slit     success        ?
20:27  count      ph,edge,slit                  success        ?
20:27  scan       edge           motor_edge     success        ?
20:27  scan       edge           motor_edge     success        ?
20:27  scan       motor_ph       ph             fail           ?
20:27  count      ph,edge,slit                  success        test

Filtering

We have been filtering based on time. We can filter on user-provided metadata like purpose or automatically-captured metadata like detectors. And we can apply multiple filters at the same time.

[14]:
summarize_runs(db(since=an_hour_ago, purpose='data'))
HH:MM  plan_name  detectors      motors         exit_status    purpose
20:27  scan       ph             motor_ph       success        data
20:27  scan       slit           motor_slit     success        data
[15]:
summarize_runs(db(since=an_hour_ago, detectors='ph', purpose='test'))
HH:MM  plan_name  detectors      motors         exit_status    purpose
20:27  count      ph                            success        test
20:27  count      ph,edge,slit                  success        test

There is a rich query language available here; we are just exercising the basics.

What about getting the data itself?

Wait for the next notebook!

So what’s happening inside print(...)?

A couple handy Python concepts you might not have encountered before…

“f-strings” (new Python 3.6!)

[16]:
name = "Dan"
age = 32

print("Hello my name is {name} and I am {age}.")
Hello my name is {name} and I am {age}.

Add an f before the quote and it becomes a magical “f-string”!

[17]:
print(f"Hello my name is {name} and I am {age}.")
Hello my name is Dan and I am 32.

You can put code inside the {}s.

[18]:
print(f"Hello my name is {name} and next year I will be {1 + age}.")
Hello my name is Dan and next year I will be 33.

Dictionary lookup with defaults

Recall basic dictionary manipulations:

[19]:
md = dict(plan_name='count', detectors=['ph', 'edge'], time=now)
[20]:
md['detectors']  # Look up the value for the 'detectors' key in the md dictionary.
[20]:
['ph', 'edge']
[21]:
md['purpose']  # The user never specified a 'purpose' here, so this raises a KeyError.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-a4691493f52d> in <module>
----> 1 md['purpose']  # The user never specified a 'purpose' here, so this raises a KeyError.

KeyError: 'purpose'
[22]:
md.get('purpose', '?')  # Falls back to a default instead of erroring out.
[22]:
'?'

list -> comma-separated string

[23]:
md.get('detectors', [])
[23]:
['ph', 'edge']
[24]:
', '.join(md.get('detectors', []))
[24]:
'ph, edge'
[25]:
', '.join(md.get('motors', []))  # Remember motors isn't set, so this falls back to the default, an empty list.
[25]:
''

time-munging

[26]:
md['time']  # seconds since 1970, the conventional "UNIX epoch"
[26]:
1594844844.263576
[27]:
datetime.fromtimestamp(md['time'])  # year, month, date, hour, minute, second, microseconds
[27]:
datetime.datetime(2020, 7, 15, 20, 27, 24, 263576)

putting it all together…

[28]:
def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time'])}  "
              f"{md['plan_name']}"
              f"{','.join(md.get('detectors', []))}"
              f"{','.join(md.get('motors', []))}"
              f"{h.stop['exit_status']}")
[29]:
summarize_runs(db(since=an_hour_ago))
HH:MM  plan_name  detectors      motors         exit_status
2020-07-15 20:27:26.820477  scanphmotor_phsuccess
2020-07-15 20:27:22.915417  scanphmotor_phsuccess
2020-07-15 20:27:21.013330  countphsuccess
2020-07-15 20:27:27.071858  scanslitmotor_slitsuccess
2020-07-15 20:27:26.380609  scanedgemotor_edgesuccess
2020-07-15 20:27:25.072468  countphsuccess
2020-07-15 20:27:25.744559  scanedgemotor_edgesuccess
2020-07-15 20:27:23.514606  scanslitmotor_slitsuccess
2020-07-15 20:27:21.129517  countph,edge,slitsuccess
2020-07-15 20:27:22.473818  scanedgemotor_edgesuccess
2020-07-15 20:27:21.811253  scanedgemotor_edgesuccess
2020-07-15 20:27:24.337953  scanmotor_phphfail
2020-07-15 20:27:25.186534  countph,edge,slitsuccess

finishing touch: white space

Use :N to fix width at N characters.

[30]:
f"Hello my name is {name:10} is I am {age:5}."
[30]:
'Hello my name is Dan        is I am    32.'

Format the time.

[31]:
f"{datetime.fromtimestamp(md['time']):%H:%M}"
[31]:
'20:27'
[32]:
def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:11}")
[33]:
summarize_runs(db(since=an_hour_ago))
HH:MM  plan_name  detectors      motors         exit_status
20:27  scan       ph             motor_ph       success
20:27  scan       ph             motor_ph       success
20:27  count      ph                            success
20:27  scan       slit           motor_slit     success
20:27  scan       edge           motor_edge     success
20:27  count      ph                            success
20:27  scan       edge           motor_edge     success
20:27  scan       slit           motor_slit     success
20:27  count      ph,edge,slit                  success
20:27  scan       edge           motor_edge     success
20:27  scan       edge           motor_edge     success
20:27  scan       motor_ph       ph             fail
20:27  count      ph,edge,slit                  success

Exercise

Q1. Add an ‘operator’ column. Hint: Remember that ‘operator’ was reported by the user in some of our example data above, but the name has no special significance to Bluesky and is not guaranteed to be reported. To avoid erroring when it is not reported, you will need to use md.get(...) instead of md[...].

[34]:
# Type your answer here. We have pasted in the latest version of summarize_runs to start from.

def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:15}")
[35]:
%load solutions/summarize_runs_with_operator.py

Q2. Print a table with results filtered by operator, just as we filtered results by purpose.

[36]:
# Fill in the blank
# summarize_runs(db(_____))
[37]:
%load solutions/filter_runs_by_operator.py

Q3. Add seconds to the time columnn.

[38]:
# Type your answer here. We have pasted in the latest version of summarize_runs to start from.

def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:15}")
[39]:
%load solutions/summarize_runs_with_seconds.py

Q4. The md['uid'] is the guaranteed unique identifier for a run. It’s unweildy to print in its entirely. Print just the first 8 characters. (For practical purposes, this is sufficently unique.)

Hint: String truncation in Python works like this:

'supercalifragilisticexpialidocious'[:8] == 'supercal'
[40]:
# Type your answer here. We have pasted in the latest version of summarize_runs to start from.

def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:15}")
[41]:
%load solutions/summarize_runs_with_uid.py

Q5. In all our examples, the columns are left-justified. The format specification mini language documents how to right-justify or center the text. Right-justify the exit_status column. This is a bit of a contrived example, but the feature is more useful when the column have numerical data.

[42]:
# Type your answer here. We have pasted in the latest version of summarize_runs to start from.

def summarize_runs(headers):
    print("HH:MM  plan_name  detectors      motors         exit_status")
    for h in headers:
        md = h.start
        print(f"{datetime.fromtimestamp(md['time']):%H:%M}  "
              f"{md['plan_name']:11}"
              f"{','.join(md.get('detectors', [])):15}"
              f"{','.join(md.get('motors', [])):15}"
              f"{h.stop['exit_status']:15}")
[43]:
%load solutions/summarize_runs_right_justify_exit_status.py