New CLI parameters for ‘start-re-manager’ (and matching parameters for ‘qserver-list-plans-devices’):
--use-python-kernel(values ON/OFF, config file parameter
worker/use_ipython_kernel, boolean, env. variable
--ipython-dir(IPython directory, overrides default directory or IPYTHONDIR, config file parameter
--ipython-matplotlib(accepts the same values as
--matplotlibparameter of IPython, config file parameter
worker/ipython_matplotlib, used only in the IPython kernel mode);
--ignore-invalid-plans(values ON/OFF, boolean config file parameter
--device-max-depth(integer, config file parameter
startup/device_max_depth, restricts maximum depth for device components included in the list of existing devices).
An option to start IPython kernel in the worker process. The option is selected by starting RE Manager with the option
--use-ipython-kernel=ON, setting config file parameter
worker/use_ipython_kernel=Trueor environment variable
QSERVER_USE_IPYTHON_KERNEL=true. Users may connect to the kernel directly using Jupyter console. The mode may be useful when transitioning from the existing IPython workflow or for debugging purposes.
New RE Manager status parameters:
ip_kernel_state(None - environment is closed,
disabled- IP kernel is not started or not used in the current mode,
starting- startup in progress,
ip_kernel_captured(None - environment is closed or the kernel is not used, True/False - indicates if the kernel is running tasks started by RE Manager, clients may connect to IP kernel directly and start tasks only when the kernel is not captured);
New worker environment state (
failed. Indicates that the environment failed to start and will be closed. The state is used internally and is unlikely to be reported.
is_ipython_mode()function may be used in startup code to detect if the code is executed in IPython environment. The function returns correct result even if the code is running in the Python-based worker environment with monkeypatched IPython package. The use cases are similar
register_devicefunctions for using in startup code. Currently,
register_planallows to explicitly exclude a given plan from processing by Queue Server (may be useful for problematic plans) and
register_deviceallows to exclude a given device or set maximum depth for the device. This is experimental feature. Functionality may be added in the future.
RE Manager status returns the new
plan_queue_mode/ignore_failuresboolean parameter, which indicates if the mode is enabled.
queue_mode_setAPI now accepts a value for
ignore_failuresmode may be enabled/disabled using
qserverCLI tool (
qserver queue mode set ignore_failures Trueand
qserver queue mode set ignore_failures False).
queue_autostartAPI (enable/disable AUTOSTART mode by passing
queue_autostart_enabledstatus parameter, which indicates if the queue is in the ‘autostart’ mode.
CLI options to enable/disable autostart:
qserver queue autostart enable/disable.
Each item in the list of current runs (
re_runsAPI) now contains
scan_id(integer) of the current scan.
The history items (
history_getAPI) now contains a list of scan IDS (
list(int)) in addition to the list of
config_getAPI. Currently returns IPython kernel connect info (
New parameter of
--ipython-kernel-ipsets IP address of the kernel, the respective config file parameter is
worker/ipython_kernel_ipand environment variable
qserver-consoleCLI tool, which downloads kernel connection info and starts Jupyter Console.
qserver configoption for
qserver-qtconsoleentry point (starts Jupyter Qt Console).
New CLI parameters for
--use-ipython-kernel=ON, then the startup code is loaded as part of initializing IPython kernel. The IPython kernel is created in a separate process and initialized to load startup code (same as in the worker process of RE Manager), but never started.
Changed handling of CLI parameters by
qserver-list-plans-devices. The parameters
--ipython-dirare now handled identically to
qserver environment openand
re_pauseAPI calls are now accepted whenever Run Engine is in the
runningstate. For example, the API may be used to pause the plan that was started in IPython kernel directly using Jupyter console and not managed by RE Manager.
Improved manager and worker stability in case of malfunctioning plans (plans that block Run Engine event loop).
zmq_single_request()functions. The timeout overrides the default timeout
timeout_recvset during instantiation of the respective classes for the particular request.
Support for managing parameters to RE Manager using configuration YML files.
New CLI parameter
--configand environment variable
QSERVER_CONFIGfor passing the path to config file to RE Manager.
Default user group name is changed from
primary. Users of applications that rely on default user group name should change the group name in
Removed built-in protocol support, now relies on
script_uploadAPI. The parameter accepts boolean value (
Trueby default) and allows to disable update of lists of existing and allowed plans and devices after execution of the script. The parameter allows to improve efficiency of execution of scripts that do not add or modify plans and devices in RE worker namespace. Update of the lists may be disabled from CLI as
qserver script upload <path-to-file> keep-lists.
lock_infoAPI. The API are accessible from CLI using
qserver-clear-lockCLI tool for unlocking RE Manager if the lock key is lost and the emergency lock key is not set or unknown.
Foreground tasks (started using
function_executeAPI) are now executed in the main thread of RE Worker.
Plan results (in plan history) now include error message (
msgkey), which contains error message or full traceback in case of failing plan.
creating_environmentRE Manager state. Now the requests to destroy environment are accepted when
status["worker_environment_exists"] is Trueor
status["manager_state"] == "creating_environment".
validate_zmq_keycan now be imported directly from
Patching of IPython-style startup scripts:
__file__variable now returns the path to the original unpatched script.
Capturing console output with updating progress bars (Python 3.8, 3.9).
A bug in the code for management of exceptions that occur during preparation of plans for execution.
A bug that prevented single character device/plan names to be properly handled by the code that converts device/plan names to the respective objects.
exit_status(in plan history) now takes values
re_stopAPI) are considered successful and no longer pushed back in the queue. The
stoppedplans are inserted in the back of the queue in LOOP mode.
Standard names for parameters for CLI tools:
--zmq-control-addris used to pass address of RE Manager control socket and
--zmq-info-addris used to pass the address of RE Manager information socket (currently used for publishing console output). Old parameter names are deprecated, but still supported.
Standard names for environment variables:
QSERVER_ZMQ_INFO_ADDRESSare used to pass control and information socket address to the server (
start-re-manager) and clients (
qserver-console-monitor). An address passed as a parameter overrides the address passed as environment variable. Old environment variable names are deprecated, but still supported.
Changed name of the environment variable used to pass the private encryption key to
QSERVER_ZMQ_PRIVATE_KEYis still supported, but deprecated.) Public key is still passed to
The components of Area Detectors are no longer included in the list of available devices.
Improved handling of IPython-style startup scripts.
Minor change in representation of plan execution results in items of the plan history. If plan execution fails, the
msgparameter contains a brief message that identify the error (may not be helpful) and
tracebackparameter contains full traceback. The parameters are empty strings in case the plan succeeds.
Similar change to representation of task execution results returned by
Nonein case the task fails and
tracebackcontain brief error message and traceback of the raised exception.
Improved default handling of strings in the parameter processing code. Now any string (any combination of characters) can be passed with a parameter, which does not have type annotation. The strings that match one of the allowed device or plan names are going to be converted to the respective objects.
Capturing console output with updating progress bars (Python 3.8, 3.9).
ReceiveConsoleOutputAsync.stop()method now accepts an optional
unsubscribeparameter that controls if 0MQ socket is unsubscribed when the acquisition is stopped. Default is
time_stopare now added to
resultdictionary of each item in plan history.
Implemented proper handling of non-JSON or invalid JSON requests.
Bug in handling of negative indices by
proper update of
Renamed parameters of
reload_permissionsis renamed to
reload_plans_devicesis renamed to
Default BEC: no longer plot with best effort callback (improves performance).
Fixed handling of negative item indices by
pos=-1now adds an item to the back of the queue).
task_statusAPI. The API may be called for a single task from CLI as
qserver task status <task-uid>.
statusAPI is now returning Queue Server version number as part of
"RE Manager v0.0.11".
re_pauseAPI. Now the
optionparameter is optional. The default value is
A bug that allowed classes defined in the global scope of the startup script and recognized as
ophyd.Device) to be included in the list of existing devices. Only instantiated class objects are currently included in the list.
A deficiency in the code that loads Python scripts (not startup scripts from the folder as in IPython
profile_collection) into the environment that failed to load scripts containing definitions of devices with components.
Support for passing subdevice names as values of plan parameters.
Support for regular expressions in the lists of names defined in
__FLYABLE__can be used in conjunction with regular expression to select device of the respective types.
New boolean parameters of the plan parameter annotation (
convert_device_names) for explicitly enabling/disabling conversion of names of plans and/or devices passed as parameter values. Setting those parameters overrides the default behavior and should be used with caution.
Support for subdevice names in ‘user_group_permissions.yaml’.
The algorithm for processing of user group permissions has changed. The old ‘user_group_permissions.yaml’ may no longer work as expected. If the stock ‘user_group_permissions.yaml’ is used for the project, replace it with the updated file from the repository. Otherwise update the existing file using following guidelines:
If the project uses custom ‘user_group_permissions.yaml’, then insert
:before each regular expression in the lists (e.g. change
In previous versions, only the lists with regular expressions were supported. Now the lists may include explicitly listed plan, device or subdevice names, such as
"det1.val"(there is no need to use regular expressions such as
":^count$"to allow the plan
The supported patterns allow to control which subdevices are included. For example, the pattern
:^detincludes all devices with names starting with
det, but no subdevices. The pattern
":^det:?.*"selects all subdevices with unlimited depth. Patterns may include the parameter
depththat limits maximum depth for subdevices, for example
":^det:?.*:depth=2"adds subdevices and subdevices of subdevices. See Configuring User Group Permissions for more detailed instructions.
AllPlanscan no longer be used in parameter annotations of defined in
parameter_annotation_decorator. Use regular expressions in conjunction with keywords
__FLYABLE__to create lists of devices of respective types. Use built-in types
__PLAN_OR_DEVICE__in parameter annotations to selectively enable conversion of names for all plans and/or devices without creating lists of names. Alternatively, use
convert_device_namesparameters of the annotation in order to explicitly enable/disable conversion of all plan/device names.
Numerous fixes related to reliability of Queue Server operation.
Implemented changes to make Queue Server compatible with
--update-existing-plans-devicesCLI parameter of
start-re-managerwas added that controls when the file that stores existing plans and devices is updated.
A new parameter of
reload_plans_devices(boolean, the default value is
False). If set
True, the parameter forces RE Manager to load the list of existing plans and devices from the disk file. The API may be called with
qserverCLI tool as
qserver permissions reload lists.
A new parameter of
reload_permissions(boolean, the default value is
True, permissions are reloaded from the disk file (if allowed), otherwise the currently used permissions are used in computations.
Extended the number of states of worker environment. Currently used states include
A new status fields (
Extended the number of sections in specification of user group permissions (e.g. in
user_group_permissions.yamlfile). The new sections (
forbidden_functions) define conditions for names of functions that are accessible using
function_executeAPI by users from each user groups.
New 0MQ API:
devices_existing. CLI implementation:
qserver script upload(
qserver function execute(
qserver task result(
qserver existing devices(
qserver existing plans(
A new 0MQ API:
permissions_getthat allow uploading and downloading user group permissions. CLI support for the new API:
qserver permissions set <fln.yaml>and
qserver permissions get.
A new parameter of
--user-group-permissions-reload. The parameter accepts values
A new section in documentation on management of user group permissions.
Refactoring of the code for management of user group permissions and lists of existing and allowed plans and devices to make it more consistent. The identical lists of user permissions and existing plans and devices are now maintained by worker and manager processes.
The lists of existing plans and devices used by RE Manager (both manager and worker processes) is automatically updated each time plans and devices in are changed in RE namespace (currently RE namespace is changed only when a new worker environment is opened).
All sections for user group permissions are now optional. The
forbidden_functionssections could be skipped if there are no forbidden items that need to be excluded. Skipping
allowed_...section disables all items (plans, devices or functions) for the group, e.g. if
allowed_plansis skipped, users from this group will not be able to submit or run any plans. Since rules for the
rootgroup are applied to the lists accessible by users from all other groups, skipping
rootdisables all plans for all other groups.
re_pause0MQ API: if
re_pauseis called past the last checkpoint of the plan, the plan is considered successfully completed and execution of the queue is stopped. The stopped queue can be started again using
JSON schemas and code using validation of JSON schemas was modified for compatibility with
jsonschemav4.0.1. Queue server still works with older versions of
A new boolean flag (
pause_pending) added to dictionary returned by
statusAPI. The flag is
Truewhen request to pause a plan (
re_pauseAPI) was accepted by the Queue Server, but not processed by the Run Engine. The flag is set in case of immediate and deferred pause request. The flag is cleared automatically (set to
False) when the request is processed and the plan is paused or the queue is stopped (if deferred pause is requested after the last checkpoint of the plan).
ReceiveConsoleOutputAsync(async version of
ReceiveConsoleOutput) for receiving console output from RE Manager in asyncio-based applications (e.g. HTTP Server).
Renamed parameters of start-re-manager:
--zmq-publishis renamed to
--zmq-publish-addris renamed to
parameter_annotation_decoratornow must be python expressions of supported types (
default) or int or float numbers (
step). In previous versions the parameter values had to be converted to strings in user code.