How to store data from the Run Engine

I want to connect RunEngine with Databroker and start saving data.

For small- and medium-sized deployments

Subscribe the Run Engine directly to Databroker.

RE.subscribe(catalog.v1.insert)

Note

You may notice that we are falling back to the v1 API here, where for all other things we show and recommend the new v2 API. This is because we are still discussing the design for this in v2. Until that is sorted out, this is the officially-recommended solution.

It uses Suitcase internally to do the writing.

This will cause the RunEngine to wait for each document it emits to be stored successfully before it proceeds with the next step of data acquisition.

Pro: We are assured that if data is not saved successfully, we will immediately know and the data acquisition will be aborted. We avoid the scary scenario of thinking we are saving data when we are not.

Con: By waiting for data to make it all the way into the database, data acquisition will be marginally slower than if we took a more sophisticated approach.

For facility-scale deployments

At present, all facilities currently using Bluesky (as far as we are aware) are using this straightforward approach described above but a more sophisticated alternative is being tested.

In short, put a message bus such as Kafka between the Run Engine and the database. Tooling for this is under development at bluesky-kafka. Check back here for updates later in 2021.