Annotating Bluesky Plans

Introduction

Queue Server is using information on existing plans and devices stored in the file existing_plans_and_devices.yaml for validating submitted plans. The file can be generated using qserver-list-plans-devices CLI tool or automatically created/updated by RE Manager (see Updating the List of Existing Plans and Devices for available options).

Representations of plans generated by qserver-list-plans-devices contain items such as text descriptions of plans and plan parameters, parameter type annotations, default values and ranges for numerical values. The representations are sufficient to perform validation of parameters of submitted plans without access to startup scripts or RE Worker namespace. Plan representations can also be downloaded by client applications (‘plans_allowed’ 0MQ API) and used to validate plan parameters before the plans are submitted to the server. For details on plan parameter validation see Validation of Plans. Client applications may also use items such as text descriptions, type annotations, default values and ranges for generating or annotating user interfaces, such as GUI forms for plan parameters.

In this manual, the elements of a plan (Python function) header, docstring and the parameters of the optional parameter_annotation_decorator that are used for generating plan representations are referred as plan annotation.

All elements in plan annotations are optional. But properly annotating plans may be beneficial if features such as parameter validation or automated user interface generation are needed. For example in cases when users are manually entering plan parameter values in Qt or web forms, it is preferable to detect errors at the time when the plans are submitted to the queue and reject plans as opposed to waiting for plans to fail when they are sent for execution.

Note

Validation of plan parameters is performed each time a new or modified plan is submitted to Queue Server. Validation can be also performed on the client side before the plan is submitted. To run validation, the client must download the lists of allowed plans and devices (‘plans_allowed’ and :ref:`method_devices_allowed 0MQ API) and call validate_plan() (API for Plan Validation).

Plan annotation may contain the following (optional) elements:

  • Description of the plan: multiline text that describes the plan. The plan description may be displayed to users by client applications.

  • Descriptions for each parameter of the plan: multiline text that describes a plan parameter. Separate description is provided for each parameter. Parameter descriptions may be displayed to users by client applications.

  • Type annotations for each parameter. Parameter types are used for validation of plan parameters. The types may also be used by client applications for generating user interfaces.

  • Default values for each parameter. The parameters with defined default values are optional (following Python rules). The default values are used for parameter validation. The default values may also be used by client applications for generating user interfaces.

  • Minimum, maximum and step values for each numerical parameter. The optional minimum and maximum values define allowed range for numerical parameter values that is used in parameter validation. Step size is passed to client application and may be useful in generating user interfaces.

The elements of plan annotations are defined in the plan header (type hints and default values) and the docstring (parameter descriptions). In addition, Queue Server supports parameter_annotation_decorator (Parameter Annotation Decorator), which allows to define or override any annotation item. The decorator is optional and should be used only when necessary.

Note

When a plan is executed in IPython environment, it behaves as a regular Python generator function. Only the default values defined in the plan header are used. Any elements defined in parameter_annotation_decorator are ignored.

Plans Without Annotation (Default Behavior)

All elements of parameter annotations are optional. Plans without annotations can be successfully managed by Queue Server. Some of the elements, such as text descriptions of plans and plan parameters or step values for numerical parameters are not used by Queue Server, but may be downloaded and used by client applications. The other elements, such as parameter types and default values are used for plan parameter validation in Queue Server. All the elements may be downloaded and used by client applications.

Depending on whether plan annotation contains a default value for a parameter, the parameter is considered required or optional. Plans submitted to the queue must contain values for all required parameters. The default values are used for missing optional parameters.

For each plan parameter, annotation may contain optional type specification. All submitted parameter values undergo type validation. For parameter with type annotation, validation includes verification of the type of the submitted value based on specified parameter type. The parameters with no type annotations are treated according to the default rules:

  • Type checking always succeeds, i.e. any submitted value is accepted and passed to the plan. Plan execution may fail due to incorrect parameter type.

  • All strings found in the submitted parameter value (elements of lists, values of dictionaries, etc.) are matched against the lists of plans and devices allowed for the user submitting the plan. The matching strings are replaced by references to respective objects (plans or devices) from RE Worker namespace, all the other strings are passed as strings.

The validation algorithm is processing each parameter independently. Type validation is applied to the parameters with specified type annotation and default rules to the parameters without specified type.

The examples of the plans with no annotation:

def plan_demo1a(npts, delay):
    # Parameters 'npts' and 'delay' accept values of any type.
    #   No type validation is performed on the parameter values.
    #   The plan may fail during execution if value is not accepted by the plan.
    <code implementing the plan>

def plan_demo1b(npts, delay=1.0):
    # Same as 'plan_demo1' except the default value for parameter 'delay' is
    #   specified, which makes the parameter 'delay' optional.
    #   No type validation is performed for any parameter.
    <code implementing the plan>

Queue Server supports plans with parameters accepting references to devices or other plans. The devices or plans passed as parameters must be defined in startup scripts, loaded in RE Worker namespace and represented in the list of existing devices (existing_plans_and_devices.yaml). When submitting plans to the queue, the devices and plans must be represented by their names (type str). The names are replaced by references to objects in RE Worker namespace before the parameter values are passed to the plans for execution. All submitted parameter values are parsed and each string found in the tree formed by lists, tuples and dictionaries is replaced with a reference to the matching object. If there is no object with the matching name found or the name is not in the list of allowed plans or devices for the user submitting the plan, then the string is not modified and passed directly to the plan. If the parameter value contains dictionaries, the dictionary keys are never modified by the algorithm.

The operation of replacing plan and device names with references to objects from RE Worker namespace is performed for each parameter with no type annotation. This means that every string that matches a name of a device, subdevice or a plan from the list of allowed devices or the list of allowed plans is replaced by the reference to the respective object from RE Worker namespace.

Let’s consider an example of a plan with parameter detectors that is expected to receive a list of detectors:

from ophyd.sim import det1, det2, det3
# Assume that the detectors 'det1', 'det2', 'det3' are included in the list
#   of allowed devices for the user submitting the plan.

def plan_demo1c(detectors, npts):
    # The parameter 'detectors' is expected to receive a list of detectors.
    # There is no type annotation, so the type is not validated.
    <code implementing the plan>

If the plan parameters submitted to the queue contain "detectors": ["det1", "det3"], then the strings "det1" and "det3" are replaced with references to objects det1 and det3 and the plan is executed as if it was called from IPython using

RE(plan_demo1c([det1, det3], <value of npts>))

The default behavior, when Queue Server blindly attempts to convert each string found in each parameter to an object reference may works well in simple cases (especially in demos). In some applications it may be important to guarantee that strings are passed as strings regardless on whether the match is found. In those cases the conversion may be disabled for a given parameter by specifying the parameter type, e.g. using type hints in the plan header. For example, one may need to pass plan or device names to the plan:

import typing
from ophyd.sim import det1, det2, det3
# Assume that the detectors 'det1', 'det2', 'det3' are in the list
#   of allowed devices for the user submitting the plan.

def plan_demo1d(detector_names, npts):
    # The parameter 'detector_names' is expected to receive a list of detector names.
    #   DOES NOT WORK: references to objects are passed to the plan
    <code implementing the plan>

def plan_demo1e(detector_names: typing.List[str], npts):
    # The parameter 'detector_names' is expected to receive a list of detector names.
    #   WORKS: names of detectors are passed without change
    <code implementing the plan>

If the value "detector_names": ["det1", "det3"] is passed to the plan plan_demo1d, then the detector names are converted to references. Adding type hint for the parameter detector_names (see plan_demo1e) disables string conversion and names are passed to the plan unchanged. Adding type hint also enables type validation for parameter detector_names and the plan is going to be rejected by Queue Server if the submitted value is not a list of strings. Type hint may be as restrictive as needed. For example, type hint typing.Any will still disable conversion of strings, but the server will accept value of any type.

The operation of converting strings to objects never fails. If the device name is incorrectly spelled or not in the list of allowed plans or devices, then the plan will be added to the queue and sent for execution. Since the name is passed will be passed to the plan as a string, the plan will likely fail and the queue is going to be stopped. For example, assume that "detectors": ["det1", "det4"] is passed to plan_demo1c. There is no device named det4 in the RE Worker namespace, so it will not be converted to a reference. As a result, the plan will receive the value of detectors=[det1, "det4"] and fail during execution. Queue Server provides parameter_annotation_decorator (Parameter Annotation Decorator), which can be used to define custom types for advanced parameter validation. In particular, the decorator allows to define custom enums based on lists of device or plan names and thus restrict sets of object names that that are accepted by the parameter. Setting up custom enums with specified lists of plans or devices enables the string conversion, but only the listed names will be converted to references:

from ophyd.sim import det1, det2, det3
# Assume that the detectors 'det1', 'det2', 'det3' are in the list
#   of allowed devices for the user submitting the plan.

from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "parameters": {
        "detectors": {
            "annotation": "typing.List[DevicesType1]",
            "devices": {"DevicesType1": ["det1", "det2", "det3"]}
        }
    }
})
def plan_demo1f(detectors, npts):
    # The parameter 'detector_names' is expected to receive a list of detector names.
    <code implementing the plan>

The type annotation in the decorator overrides the type annotation in the function header. Custom enums based on name lists are also used in type validation to guarantee that only the device/plan names from the defined in the enum are accepted. For example, if the submitted plan contains "detectors": ["det1", "det4"], then the plan is rejected, because there is no detector det4 in the enum type DeviceType1.

Note

Value of any type that is serializable to JSON can be passed to the plan if the respective parameter type is not defined or defined as typing.Any. In the latter case the server does not attempt to convert strings to object references.

Supported Types

Queue Server can process limited number of types used in type annotations and default values. If a plan header contains parameter unsupported type hint, Queue Server ignores the hint and the plan is processed as if the parameter contained no type annotation. If unsupported type annotation is defined in parameter_annotation_decorator, then processing of the plan fails and existing_plans_and_devices.yaml can not be generated. The processing also fails if the default value defined in the plan header or in the decorator has unsupported type.

Note

Type annotations and default values defined in parameter_annotation_decorator override type annotations and default values defined in the plan header. If type or default value is defined in the decorator, the respective type and default value from the header are not analyzed. If it is necessary to define a plan parameter with unsupported type hint or default value in the header, use parameter_annotation_decorator to override the type or the default value in order for the plan processing to work.

Supported types for type annotations. Type annotations may be native Python types (such as int, float, str, etc.), NoneType, or generic types that are based on native Python types (such as list[int], typing.List[typing.Union[int, str]]). Technically the type will be accepted if the operation of recreating the type object from its string representation using eval function is successful with the namespace that contains imported typing and collections.abc modules and NoneType type. The server can recognize and properly handle the following types used in the plan headers (see Defining Types in Plan Header and Parameter Types):

  • bluesky.protocols.Readable (replaced by __READABLE__ built-in type);

  • bluesky.protocols.Movable (replaced by __MOVABLE__ built-in type);

  • bluesky.protocols.Flyable (replaced by __FLYABLE__ built-in type);

  • bluesky.protocols.Configurable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Triggerable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Locatable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Stageable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Pausable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Stoppable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Subscribable (replaced by __DEVICE__ built-in type);

  • bluesky.protocols.Checkable (replaced by __DEVICE__ built-in type);

  • collections.abc.Callable (replaced by __CALLABLE__ built-in type);

  • typing.Callable (replaced by __CALLABLE__ built-in type).

Note

Note, that typing.Iterable can be used with the types listed above with certain restrictions. If a parameter is annotated as typing.Iterable[bluesky.protocols.Readable], then the validation will succeed for a list of devices (names of devices), but fails if a single device name is passed to a plan. If a parameter is expected to accept a single device or a list (iterable) of devices, the parameter should be annotated as typing.Union[bluesky.protocols.Readable, typing.Iterable[bluesky.protocols.Readable]]. Validation will fail for a single device if the order of types in the union is reversed.

Supported types of default values. The default values can be objects of native Python types and literal expressions with objects of native Python types. The default value should be reconstructable with ast.literal_eval(), i.e. for the default value vdefault, the operation ast.literal_eval(f"{vdefault!r}") should complete successfully.

The following is an example of a plan with type annotation that discarded by Queue Server. The type annotation is defined in the plan header, so it is ignored and parameter detector is viewed as having no type annotation.

from ophyd import Device

def plan_demo2a(detector: Device, npts=10):
    # Type 'Device' is not recognized by Queue Server, because it is imported
    #   from an external module. Type annotation is ignored by Queue Server.
    #   Use 'parameter_annotation_decorator' to override annotation for
    #   'detector' if type validation is needed.
    <code implementing the plan>

In the following example, the type of the default value of the detector parameter is not supported and processing of the plan fails. The issue can be fixed by overriding the default value using parameter_annotation_decorator (Parameter Annotation Decorator).

from ophyd.sim import det1

def plan_demo2b(detector=det1, npts=10):
    # Default value 'det1' can not be used with the Queue Server.
    #   Fix: use 'parameter annotation decorator to override the default value.
   <code implementing the plan>

Defining Types in Plan Header

Signatures of plans from RE Worker namespace are analyzed each time the list of existing plans is generated (e.g. by qserver-list-plans-devices tool). If a plan signature contains type hints, the processing algorithm verify if the types are supported and saves their string representations. Unsupported types are ignored and the respective parameters are treated as having no type hints (unless type annotations for those parameters are defined in parameter_annotation_decorator).

Note

Queue Server ignores type hints defined in the plan signature for parameters that have type annotations defined in parameter_annotation_decorator.

The acceptable types include Python base types, NoneType and imports from typing and collections.abc modules (see Supported Types). Following are the examples of plans with type hints:

import typing
from typing import List, Optional

def plan_demo3a(detector, name: str, npts: int, delay: float=1.0):
    # Type of 'detector' is not defined, therefore Queue Server will find and attempt to
    #   replace all strings passed to this parameter by references to objects in
    #   RE Worker namespace. Specifying a type hint for the ``detector`` parameter
    #   would disable the automatic string conversion.
    <code implementing the plan>

def plan_demo3b(positions: typing.Union[typing.List[float], None]=None):
    # Generic type using the 'typing' module. Setting default value to 'None'.
    <code implementing the plan>

def plan_demo3c(positions: Optional[List[float]]=None):
    # This example is precisely identical to the previous example. Both hints are
    #   converted to 'typing.Union[typing.List[float], NoneType]' and
    #   correctly processed by the Queue Server.
    <code implementing the plan>

The server can process the annotations containing Bluesky protocols such as bluesky.protocols.Readable, `bluesky.protocols.Movable and bluesky.protocols.Flyable and callable types collections.abc.Callable and typing.Callable with or without type parameters. Those types are replaced with __READABLE__, __MOVABLE__, __FLYABLE__ and __CALLABLE__ built-in types respectively. See the details on built-in types in Parameter Types.

Defining Default Values in Plan Header

Follow Python syntax guidelines for defining default values. The type of the default value must be supported by the Queue Server (see Supported Types). If the default value in the plan header must have unsupported type, override it by specifying the default value of supported type in parameter_annotation_decorator.

Note

If the default value is defined in the parameter_annotation_decorator, Queue Server ignores the default value defined in the header. Processing of the plan fails if the default value for a parameter is defined in the decorator, but missing in the function header. (A default value in the header is required if the default value is defined in the decorator.)

Parameter Descriptions in Docstring

Queue Server collects text descriptions of the plan and parameters from NumPy-style docstrings. Type information specified in docstrings is ignored. The example below shows a plan with a docstring:

def plan_demo4a(detector, name, npts, delay=1.0):
    """
    This is the description of the plan that could be passed
    to the client and displayed to users.

    Parameters
    ----------
    detector : ophyd.Device
        The detector (Ophyd device). Space is REQUIRED before
        and after ':' that separates the parameter name and
        type. Type information is ignored.
    name
        Name of the experiment. Type is optional. Queue Server
        will still successfully process the docstring.
        Documenting types of all parameters is recommended
        practice.
    npts : int
        Number of experimental points.
    delay : float
        Dwell time.
    """
    <code implementing the plan>

Parameter Annotation Decorator

The parameter_annotation_decorator (Plan Annotation API) allows to override any annotation item of the plan, including text descriptions of the plan and parameters, parameter type annotations and default values. The decorator can be used to define all annotation items of a plan, but it is generally advised that it is used only when absolutely necessary.

Note

If the default value of a parameter is defined in the decorator, the parameter must have a default value defined in the header. The default values in the decorator and the header do not have to match. See the use case in notes.

Plan and Parameter Descriptions

Text descriptions of plans and parameters are not used by Queue Server and do not affect processing of plans. In some applications it may be desirable to have different versions of text descriptions for documentation (e.g. technical description) and for user interface (e.g. instructions on how to use plans remotely). The decorator allows to override plan and/or parameter descriptions extracted from the docstring. In this case the descriptions defined in the decorator are displayed to the user.

All parameters in parameter_annotation_decorator are optional. In the following example, the description for the parameter npts is not overridden in the decorator:

from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "description": "Plan description displayed to users.",
    "parameters": {
        "detector": {
            "description":
                "Description of the parameter 'detector'\n" \
                "displayed to Queue Server users",

        }
        "name": {
            "description":
                "Description of the parameter 'name'\n" \
                "displayed to Queue Server users",
        }
    }
})
def plan_demo4a(detector, name, npts):
    """
    Plan description, which is part of documentation.
    It is not visible to Queue Server users.

    Parameters
    ----------
    detector : ophyd.Device
        The detector. Technical description,
        not visible to Queue Server users.
    name
        Name of the experiment. Technical description,
        not visible to Queue Server users.
    npts : int
        Number of experimental points.
        Description remains visible to Queue Server users,
        because it is not overridden by the decorator.
    """
    <code implementing the plan>

Parameter Types

Parameter type hints defined in a plan header can be overridden in parameter_annotation_decorator. The type annotations defined in the decorator do not influence execution of plans in Python. Overriding types should be avoided whenever possible.

Note

Types in the decorator must be represented as string literals. E.g. "str" represents string type, "typing.List[int]" represents an array of integers, etc. Module name typing must be explictly used when defining generic types in the decorator.

Type annotations defined in the decorator may be used to override unsupported type hints in plan headers. But the main application of the decorator is to define custom enum types based on lists of names of plans and devices or string literals. Support for custom enum types is integrated in functionality of Queue Server, including the functionality such as type validation and string conversion. If a parameter type defined in the annotation decorator is using on a custom enum types, which are based on lists of plans or devices, then all strings passed to the parameter that match the names of plans and devices in enum definition are converted to references to plans and devices in RE Worker namespace. The lists of names of plans and devices or string literals may also be used by client applications to generate user interfaces (e.g. populate combo boxes for selecting device names).

from typing import List
from ophyd import Device
from ophyd.sim import det1, det2, det3, det4, det5
from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "parameters": {
        "detector": {
            # 'DetectorType1' is the type name (should be a valid Python name)
            "annotation": "DetectorType1",
            # 'DetectorType1' is defined as custom enum with string values
            #   'det1', 'det2' and 'det3'
            "devices": {"DetectorType1": ["det1", "det2", "det3"]},
        }
    }
})
def plan_demo5a(detector, npts: int, delay: float=1.0):
    # Type hint for the parameter 'detector' in the header is not required.
    # Queue Server accepts the plan if 'detector' parameter value is
    #   a string with values 'det1', 'det2' or 'det3'. The string is
    #   replaced with the respective reference before the plan is executed.
    #   Plan validation fails if the parameter value is not in the set.
    <code implementing the plan>

@parameter_annotation_decorator({
    "parameters": {
        "detectors": {
            # Note that type definition is a string !!!
            # Type names 'DetectorType1' and 'DetectorType2' are defined
            #   only for this parameter. The types with the same names
            #   may be defined differently for the other parameters
            #   of the plan if necessary, but doing so is not recommended.
            "annotation": "typing.Union[typing.List[DetectorType1]" \
                          "typing.List[DetectorType2]]",
            "devices": {"DetectorType1": ["det1", "det2", "det3"],
                        "DetectorType2": ["det1", "det4", "det5"]},
        }
    }
})
def plan_demo5b(detectors: List[Device], npts: int, delay: float=1.0):
    # Type hint contains correct Python type that will be passed to the parameter
    #   before execution.
    # Queue Server accepta the plan if 'detectors' is a list of strings
    #   from any of the two sets. E.g. ['det1', 'det3'] or ['det4', 'det5']
    #   are accepted but ['det2', 'det4'] is rejected (because the
    #   detectors belong to different lists).
    <code implementing the plan>

Similar syntax may be used to define custom enum types for plans (use "plans" dictionary key instead of "devices") or string literals (use "enums" dictionary key). The strings listed as "devices" are converted to references to devices and the strings listed as "plans" are converted to references to plans before plan execution. Strings listed under "enums" are not converted to references, but are still used for plan parameter validation. Mixing devices, plans and enums in one type definition is possible (Queue Server will handle the types correctly), but not recommended.

The lists of plan and device may contain a mix of explicitly listed plan/device names and regular expressions used to select plans and devices. See Lists of Device and Plan Names for detailed reference to writing lists of devices and plans.

The decorator supports the following built-in types: __PLAN__, __DEVICE__, __READABLE__, __MOVABLE__, __FLYABLE__, __PLAN_OR_DEVICE__ and __CALLABLE__. The types __DEVICES__, __READABLE__, __MOVABLE__ and __FLYABLE__ are treated identically by the server, but additional type checks could be added in the future. The built-in types are replaced by str for type validation and conversion of plan and/or device names enabled for this parameter. No plan/device lists are generated and plan/device name is not validated. The built-in types should not be defined in devices, plans or enum sections of the parameter annotation, since it is going to be treated as a regular custom enum type. The __CALLABLE__ type is treated similary to the other built-in types during parameter validation. The strings passed as the parameter values and representing names of python variables or class object attributes are converted to references to the respective objects from the worker namespace.

from ophyd.sim import det1, det2, det3, det4
from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "parameters": {
        "detectors": {
            # '__DEVICE__' is the built-in type. The plan will accept a list of
            # object names (strings), validate the parameter type and attempt to
            # convert all string to device objects (not to plans).
            "annotation": "typing.List[__DEVICE__]",
            # If '__DEVICE__' is explicitly defined in the 'devices' section,
            # it will be treated as a custom enum type(only for this parameter).
        }
    }
})
def plan_demo5c(detectors, npts: int, delay: float=1.0):
    <code implementing the plan>

The lists of custom enum types for devices or plans may include any device or plan names defined in startup scripts and loaded into RE Worker namespace. The type definitions are saved as part of plan representations in the list of existing plans. If built-in enum types are used, the definitions will contain full lists of devices from the namespace. When lists of allowed plans are generated for user groups, custom type definitions are filtered based on user group permissions, so that only the devices and plans that are allowed for the user group remain. This allows to use entries from downloaded lists of allowed plans for validation of plans and for generation of user interfaces directly, without verification user permissions, since it is guaranteed, that the type definitions contain only devices and plans that the current user is allowed to use. Filtering type definitions may cause some lists to become empty in case current user does not have permission to use any devices or plans that are listed in type definition.

Explicitly Enabling/Disabling Conversion of Plan and Device Names

Parameter annotation allows to specify explicitly whether strings passed to this parameter are converted to plan or device objects. Optional boolean parameters convert_device_names and convert_plan_names override any default behavior. If those parameters are not specified, then Queue Server determines whether to convert names to objects based on parameter annotation defined in plan header and parameter_annotation_decorator:

from ophyd.sim import det1, det2, det3, det4
from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "parameters": {
        "dets_1": {
            "annotation": "typing.List[str]",
            # Queue Server attempts to convert each string to a device
            #   or subdevice from the list of allowed devices.
            "convert_device_names": True,
        }
        "dets_2": {
            "annotation": "typing.List[__DEVICE__]",
            # The device names are not converted to device objects and
            #   passed to the plan as strings.
            "convert_device_names": False,
        }
        "dets_3": {
            # Device names are going to be converted to device objects.
            "annotation": "typing.List[__DEVICE__]",
        }
    }
})
def plan_demo5d(detectors, npts: int, delay: float=1.0):
    <code implementing the plan>

Note

Parameters convert_plan_names and convert_device_names control only conversion of plan and device names to objects from the worker namespace and have no effect on the process of validation of plan parameters.

Default Values

Using decorator to override default values defined in plan header with different values is possible, but generally not recommended unless absolutely necessary. Overriding the default value is justified when the type of the default value defined in the header is not supported, and a different default value of supported type can be defined in the decorator so that the plan will behave identically when it is executed in Queue Server or IPython environment.

Note

The default value defined in the decorator must be a Python expression resulting in the value that satisfy requirements in Supported Types (same requirements as for the default values defined in plan header). For the custom enumerated types, the default must be one of the valid strings values.

The following example illustrates the use case which requires overriding the default value. In this example, the default value for the parameter detector is a reference to det1, which has unsupported type (ophyd.Device). When submitting the plan to the queue, the default parameter value must be the string literal "det1", which is then substituted by reference to det1. The decorator contains the definition of custom enum type based on the list of supported device names and sets the default value as a string representing the name of one of the supported devices.

from ophyd.sim import det1, det2, det3
from bluesky_queueserver import parameter_annotation_decorator

@parameter_annotation_decorator({
    "parameters": {
        "detector": {
            "annotation": "DetectorType1",
            "devices": {"DetectorType1": ["det1", "det2", "det3"]},
            "default": "det1",
        }
    }
})
def plan_demo6a(detector=det1, npts: int, delay: float=1.0):
    # The default value for the parameter 'detector' is a reference to 'det1'
    #   when the plan is started from IPython. If the plan is submitted to
    #   the queue and no value is provided for the parameter 'detector', then
    #   the parameter is going to be set to string literal value '"det1"',
    #   which is then substituted with the reference to the detector 'det1'
    #   before the plan is executed.
    <code implementing the plan>

Minimum, Maximum and Step Values

The decorator allows to define optional parameters for numeric values passed to plans, including minimum and maxumum values and step size. The minimum and maximum values determine the allowed range of numerical values used in parameter validation. Step size is not used by Queue Server and intended for generating user interfaces in client applications (e.g. combination of minimum, maximum values and step size may be used to set up a spin box). If maximum and/or minimum values are defined for a parameter, validation includes checking if each numerical value in the data structure passed to the parameter is within this range. The algorithm is searching the data structure for numerical values by iterating through list elements and dictionary values. Non-numeric values are ignored. Dictionary keys are not validated.

Setting both minimum and maximum values defines closed range for the parameter value (including the range boundaries). If only maximum or minimum boundary is set, the range is limited only from above or below (assuming the missing maximum or minimum is -Inf or Inf respectively). If no minimum or maximum value is specified, then the range is not validated.

Note

Minimum, maximum values and step size must be integer or floating point numbers.

@parameter_annotation_decorator({
    "parameters": {
        "v": {
            "default": 50,
            "min": 20,
            "max": 99.9,
            "step": 0.1,
        }
    }
})
def plan_demo7a(v):
    <code implementing the plan>

This example defines the range [20, 99.9] for the parameter v. The plan is accepted by Queue Server in the following cases:

{"v": 30}
{"v": [20, 20.001, 20.002]}
{"v": {"a": 30, "b": [50.5, 90.4]}}

The plan will be rejected if

{"v": 10}  # Value 10 is out of range
{"v": [20, 100.5, 90]}  # Value 100.5 is out of range
{"v": {"a": -2, "b": 80}}  # Value -2 is out of range
{"v": {"a": 30, "b": [50.5, 190.4]}}  # Value 190.4 is out of range

Lists of Device and Plan Names

Annotation of a parameter may contain optional devices and/or plans sections, which contains lists of device or plan names that could be passed with the parameter or patterns based on regular expressions used to pick matching devices or plans from the list of existing devices or plans. The listed or automatically picked names are converted to objects exisiting in Run Engine Worker namespace before being passed to the plan. In the following example there are two lists (Type1 and Type2) defined in the devices section. Names (keys) of the lists are used as types in the string that defines the parameter type:

@parameter_annotation_decorator({
    "parameters": {
        "dets": {
            "annotation": "typing.Union[typing.List[Type1]" \
                          "typing.List[Type2]]",
            # "det1", "det2", "det4" are explicitly stated names
            # ":det3:?val$", ":^det5$" are patterns
            "devices": {"Type1": ["det1", "det2", ":det3:?val$"],
                        "Type2": ["det1", "det4", ":^det5$"]},
        }
    }
})
def plan_demo8a(dets, npts: int, delay: float=1.0):
    <code implementing the plan>

If a plan accepts reference to other plans defined in the worker namespace, one of the plan parameters may have a similar section plans with lists of plan names.

Each list may contain a mix of device name and patterns. The processing algorithm is searching for names of existing plans and devices based on the patterns and adds them to the lists. After removing Duplicate names, the lists are sorted in alphabetical order and saved as part of plan representation in the list of existing plans.

Lists of Device Names

The elements of lists of device names may include:

  • names of devices defined in the global scope of the startup script, e.g. "det1", sim_stage_A, etc.

  • names of subdevices of devices defined in the global scope of startup script, e.g. det1.val, sim_stageA.mtrs.y, etc.

  • patterns based on regular expressions for picking devices and subdevices from the list of existing devices.

The explicitly listed device and subdevice names are added to the parameter annotation in the list of existing plans even if such device or plan does not exist in the worker environment. When plan represenations are preprocessed before being added to a list of allowed plans, the devices/subdevices/plans that are not in the list of allowed devices/plans are removed.

# The following devices will be added to the list:
#   'det1'
#   'det1.val'
#   devices such as 'd3', 'det3', 'detector3' etc. matching reg. expression 'd.*3'
"Type3": ["det1", "det1.val", ":d.*3"]

Note, that the semicolon : is not part of regular expressions and used to tell the processing algorithm that the string contains a pattern as opposed to a device or subdevice name. For example, "det" is an explicitly listed name of the detector det, while ":det" is a regular expression that matches any device name that contains the sequence of characters "d", "e", "t", such as mydetector.

A device name pattern may consist of multiple regular expressions separated by :. The expressions are processed from left to right. The leftmost expression is applied to the name of each device in global worker namespace. If the device name matches the expression, the device name is added to the list and each subdevice is checked using the second expression and matching name is added to the list. The process continues until the end of the pattern or ‘leaf’ devices (that have no subdevices) are reached. On each step only the subdevices of matching devices are searched. The maximum depth of search is defined by the number of expressions in the pattern.

# The item contains three regular expressions: '^sim' is applied to device
#   names (selects all devices starting with 'sim'), '^mt' applies to
#   subdevices of matching devices (all subdevices starting with 'mt')
#   and '^x$' is applied to subdevices of matching subdevices (all subdevices
#   named 'x'). As a result, the list may contain the following
#   devices:
#     'sim_stage_A'
#     'sim_stage_A.mtrs',
#     'sim_stage_A.mtrs.x',
#     'sim_stage_B'.
#     'sim_stage_B.mtrs'.
#     'sim_stage_B.mtrs.x'.
"Type3": [":^sim:^mt:^x$"]

The default behavior of the list generation algorithm is to include in the list all matching devices and subdevices found at each level. The algorithm can be explicitly instructed to do so by inserting + (plus sign) before the regular expression, i.e. the patterns ":^det.val$” and "":+^det.val$" would produce identical results. The default behavior is not always desirable. In order to skip adding to the list matching devices found at a given depth, add - (minus sign) before the regular expression (:-):

# The list will contain devices such as
#   'sim_stage_A.mtrs',
#   'sim_stage_A.mtrs.x',
#   'sim_stage_B.mtrs',
#   'sim_stage_B.mtrs.x'.
"Type4": [":-^sim:^mt:^x$"]

# The list will contain devices such as
#   'sim_stage_A.mtrs.x',
#   'sim_stage_B.mtrs.x'.
"Type5": [":-^sim:-^mt:-^x$"]

Note, that even though - is allowed in the last expression of the pattern, it does not influence processing of the pattern. The devices matching the last expression in the pattern are always inserted in the list.

Regular expressions could be applied to the full device name (such as "sim_stage_A.mtrs.x") or right part a full device name (such as "mtrs.x"). An expression may be labelled as a full-name expression by putting ? (question mark) before the regular expression. The full name regular expression may only be the last component of the pattern. :

# The list may contain devices with names such as
#   'simval',
#   'sim_stage_A.val',
#   'sim_stage_A.det1.val',
#   'sim_stage_A.detectors.det1.val',
"Type6": [":?^sim.*val$"]

# The list will contain devices such as
#   'sim_stage_A',
#   'sim_stage_A.val',
#   'sim_stage_A.det1_val',
#   'sim_stage_A.det1.val',
#   'sim_stage_A.detectors.det1.val',
"Type7": [":^sim_stage_A$:?.*val$"]

Note, that + and - can not be used in conjunction with ?. Using full-name expressions is less efficient, since it the algorithm is forced to searching the full tree of subdevices for matching names. The depth of search may be explicitly limited by adding depth parameter (:?<regex>:depth=N):

# The list will contain devices such as
#   'sim_stage_A',
#   'sim_stage_A.val',
#   'sim_stage_A.det1_val',
#   'sim_stage_A.det1.val',
# The list will not contain 'sim_stage_A.detectors.det1.val',
#   since the depth of search is limited to 2 levels.
"Type8": [":+^sim_stage_A$:?.*val$:depth=2"]

A set of devices matching a pattern may be restricted to devices of certain types by placing one of the type keywords before the patter. The supported keywords are __DETECTOR__ (readable, not movable), __MOTOR__ (readable and movable), __READABLE__, __FLYABLE__ before the expression:

# Select only detectors with names matching the regular expression:
"Type9": ["__DETECTORS__:^sim_stage_A$:?.*:depth=3"]

# Select only motors:
"Type10": ["__MOTORS__:^sim_stage_A$:?.*:depth=3"]

Lists of Plan Names

Similarly to lists of device names, the lists of plan names may include patterns used to pick matching names of the existing plans:

# The list contains the following names: explicitly listed name of the plan
#   ``count`` and regular expression that selects all the plans ending
#   with ``_count``, such as ``_count``, ``my_count`` etc.
"Type11": ["count", ":_count$"]

A plan name pattern may contain only a single regular expression (i.e. a pattern may contain only one : character). The modifiers +, - and ? may still be used, but the do not influence processing of the plan name patterns (they are allowed to make patterns for plans and devices look uniform, but since plan patterns contain only one component and this component is the last, the matching plan names are always added to the list and the regular expression is always applied to the full name). Device type keywords can not be used in plan name patterns.

Plan Annotation API

parameter_annotation_decorator

The decorator allows to attach a custom description to a function or generator function.