================== Command-Line Tools ================== The CLI tools are installed with the *bluesky-queueserver* package: - :ref:`start_re_manager_cli` - start RE Manager. - :ref:`qserver_cli` - communicate with RE Manager over 0MQ. - :ref:`qserver_list_plans_devices_cli` - generate list of existing plans and devices, validate startup code. - :ref:`qserver_zmq_keys_cli` - generate key pair for encryption of 0MQ control channel. - :ref:`qserver_console_monitor_cli` - simple monitor of RE Manager console output. - :ref:`qserver_clear_lock_cli` - unlock RE Manager if the lock key is lost. - :ref:`qserver_console_cli` - start Jupyter Console connected to IPython kernel running in the worker. - :ref:`qserver_qtconsole_cli` - start Jupyter Qt Console connected to IPython kernel running in the worker. .. _start_re_manager_cli: start-re-manager ---------------- The command starts Run Engine (RE) Manager on the workstation. If no parameters are specified, RE Manager is started in demo mode (built-in startup files with simulated devices and plans will be used). Demo mode is intended for evaluation and testing of Queue Server. The CLI parameters may be used to customize RE Manager and specify the location of beamline startup scripts. .. _location_of_startup_code: Using a Configuration Files +++++++++++++++++++++++++++ RM Manager can be customized using a config file (YML). Combining the parameters in a single config file may be more convenient in production deployments. The path to config file is passed to the manager using environment variable ``QSERVER_CONFIG`` or using CLI parameter ``--config``. CLI parameters passed to ``start-re-manager`` override the parameters set in config file or passed using environment variables. Location of Startup Code ++++++++++++++++++++++++ RE Manager may create RE Worker environments based based on startup code from IPython-style collection of startup scripts, Python script or Python module. The locations of autogenerated ``existing_plans_and_devices.yaml`` file (see :ref:`update_existing_plans_devices` and :ref:`qserver_list_plans_devices_cli`) and ``user_group_permissions.yaml`` must also be specified. The ``existing_plans_and_devices.yaml`` file is optional and could be generated automatically by opening RE Worker environment, but the path to the file pointing to a location in existing directory must be properly specified. Alternatively, the file can be initially generated by ``qserver_list_plans_devices`` CLI tool. **IPython-style collection of startup scripts:** Use the parameter ``--startup-dir`` to specify the directory path to the collection of startup file. Both ``.yaml`` files are expected to be located in the startup directory. If the location the ``.yaml`` files is different, use the parameters ``--existing-plans-devices`` and ``--user-group-permissions`` to specify the correct paths. **Ipython profile:** If the collection of startup scripts is located in ``startup`` directory in one of the IPython profiles, the profile name may be passed to RE Manager using the parameter ``--startup-profile``. This option is implemented for convenience and allows to avoid specifying long path to startup directory. **Python script:** Use the parameter ``--startup-script`` to specify the path to Python script that contains startup code (``.py`` file). Only the single script will be loaded. The script may contain imports from other scripts or modules. Since there is no obvious location for the ``.yaml`` files, the paths to the files must be explicitely specified using parameters ``--existing-plans-devices`` and ``--user-group-permissions``. **Python module:** Use the parameter ``--startup-module`` to specify the name of the installed module that contains startup code. Paths to both ``.yaml`` files must be explicitly specified. Instances of Run Engine and Databroker ++++++++++++++++++++++++++++++++++++++ By default, RE Manager removes ``RE`` and ``db`` objects from RE Worker namespace and creates the new instances. In many cases, startup scripts contain the code that creates and configures instances ``RE`` and/or ``db``. To disable the default behavior and keep instances of ``RE`` and ``db`` created in startup scripts, start RE Manager with the parameter ``--keep-re``. RE Worker environment will fail to open if ``--keep-re`` is used and startup scripts do not create the instance of ``RE``. RE Manager supports a number of standard options of subscribing ``RE`` to callbacks. All the subscriptions can be performed in startup code, so those options are implemented mostly for convenience. The options work in cases when ``RE`` is created in startup script or RE Manager. (Note, that ``--keep-re`` must be used to preserve the instance of ``RE`` created in the startup script. The parameters supporting the subscription options: - ``--databroker-config`` subscribes RE to Data Broker with the given configuration name. - ``--zmq-data-proxy-addr`` address of 0MQ proxy for publishing documents generated by Run Engine. - ``--kafka-server`` and ``--kafka-topic`` for publishing documents generated by Run Engine. - ``--use-persistent-metadata`` enables msgpack-based persistent storage for scan metadata .. _update_existing_plans_devices: Updating the List of Existing Plans and Devices +++++++++++++++++++++++++++++++++++++++++++++++ The ``--update-existing-plans-devices`` parameter determines when the lists of existing plans and devices stored in a disk file is updated by Queue Server. See notes on the location of the file with the lists of existing plans and devices in :ref:`location_of_startup_code`. The lists are loaded from the disk file by RE Manager process at startup and used to generate the initial lists of allowed plans and devices. Those lists are used for validation of submitted plans, allowing user to submit plans to the queue before startup scripts are loaded. The lists are also distributed to clients and used for generating user interfaces. Therefore, it is important to keep the stored lists of existing plans and devices up to date. The parameter allows to select between the following modes for updating the lists: - ``NEVER`` - Queue Server never overwrites the existing file or create a new file if it is missing. Users need to update the file by manually running ``qserver-list-plans-devices`` CLI tool. Opening the RE Worker environment generates the lists of existing plans and devices that could be used during the current session, but it will not be saved to the file. The next time Queue Server is started, the original lists of existing plans and devices is loaded from disk. :ref:`method_permissions_reload` 0MQ API called with parameter ``restore_plans_devices=True`` will also reload the original list from disk. Restarting the RE Manager process while the environment is open will not read the list of existing plans and devices from disk. This option may be used in workflows with 'static', rarely changed startup scripts where it is preferred that the lists of existing plans and devices is not dynamically changed. - ``ENVIRONMENT_OPEN`` (default) - Queue Server verifies if the lists of existing plans and devices are out of date each time RE Worker environment is opened and saves the new lists to the file when necessary. This ensures that the next time Queue Server is started, the latest up-to-date version of the lists is loaded. In this mode, it is not necessary to call use ``qserver-list-plans-devices`` to generate the lists. Instead it could be left to Queue Server to update the stored lists. Note, that Queue Server will not know about changes to startup code until the operation of opening RE Worker environment is successfully completed. ``qserver-list-plans-devices`` may still be useful for debugging the startup code or generating lists of existing plans and devices in cases when starting Queue Server is inconvenient. - ``ALWAYS`` - Queue Server updates the lists of existing plans and devices stored in file each time plans and devices are added to or removed from to RE namespace. This is option is not fully utilized yet and results in behavior identical to ``ENVIRONMENT_OPEN``. .. _start_re_manager_console_output: Console Ouput +++++++++++++ RE Manager captures output to ``stdout`` and ``stderr`` produced by itself and running plans. The output may be printed to terminal and/or published to a 0MQ socket (different socket from the socket used for control communication). - ``--zmq-info-addr`` allows to set the address for the 0MQ 'PUB' socket. - Setting ``--zmq-publish-console ON`` enables publishing of the collected output to 0MQ socket. - Setting ``--console-output OFF`` disables printing to terminal. Encryption for 0MQ Control Communication Channel ++++++++++++++++++++++++++++++++++++++++++++++++ If the ``QSERVER_ZMQ_PRIVATE_KEY_FOR_SERVER`` environment variable contains a string that represents valid private key, the enryption for control channel is enabled. Clients must use correct public key from public/private key pair to communicate with RE Manager. Public/private key pair can be generated using :ref:`qserver_zmq_keys_cli`. Other Configuration Parameters ++++++++++++++++++++++++++++++ - ``--zmq-control-addr`` is the address of 0MQ socket for the server. The server is managed by RE Manager and used to pass control commands from clients. - ``--redis-addr`` is the address of Redis server used by RE Manager as persistent storage. - ``--verbose``, ``--quiet`` and ``--silent`` options modify logging verbosity level for RE Manager. ``start-re-manager -h`` displays help information: .. code-block:: $ start-re-manager -h usage: start-re-manager [-h] [--config CONFIG_PATH] [--zmq-control-addr ZMQ_CONTROL_ADDR] [--zmq-addr ZMQ_ADDR] [--startup-profile STARTUP_PROFILE] [--startup-module STARTUP_MODULE | --startup-script STARTUP_SCRIPT | --startup-dir STARTUP_DIR] [--ignore-invalid-plans {ON,OFF}] [--device-max-depth DEVICE_MAX_DEPTH] [--existing-plans-devices EXISTING_PLANS_AND_DEVICES_PATH] [--update-existing-plans-devices {NEVER,ENVIRONMENT_OPEN,ALWAYS}] [--user-group-permissions USER_GROUP_PERMISSIONS_PATH] [--user-group-permissions-reload {NEVER,ON_REQUEST,ON_STARTUP}] [--redis-addr REDIS_ADDR] [--redis-name-prefix REDIS_NAME_PREFIX] [--kafka-topic KAFKA_TOPIC] [--kafka-server KAFKA_SERVER] [--zmq-data-proxy-addr ZMQ_DATA_PROXY_ADDR] [--keep-re] [--use-ipython-kernel {ON,OFF}] [--ipython-dir IPYTHON_DIR] [--ipython-matplotlib IPYTHON_MATPLOTLIB] [--ipython-kernel-ip IPYTHON_KERNEL_IP] [--use-persistent-metadata] [--databroker-config DATABROKER_CONFIG] [--zmq-info-addr ZMQ_INFO_ADDR] [--zmq-publish-console-addr ZMQ_PUBLISH_CONSOLE_ADDR] [--zmq-publish-console {ON,OFF}] [--console-output {ON,OFF}] [--verbose | --quiet | --silent] Start Run Engine (RE) Manager bluesky-queueserver version 0.0.20 Encryption for ZeroMQ communication server may be enabled by setting the value of 'QSERVER_ZMQ_PRIVATE_KEY_FOR_SERVER' environment variable to a valid private key (z85-encoded 40 character string): export QSERVER_ZMQ_PRIVATE_KEY_FOR_SERVER='' A public/private key pair may be generated by running 'qserver-zmq-keys'. If RE Manager is configured to use encrypted channel, the encryption must also be enabled at the client side using the public key from the generated pair. Encryption is disabled by default. options: -h, --help show this help message and exit --config CONFIG_PATH Path to a YML config file or a directory containing multiple config files. The path passed as a parameter overrides the path set using QSERVER_CONFIG environment variable. The config path must point to an existing file or directory (may be empty), otherwise the manager can not be started. --zmq-control-addr ZMQ_CONTROL_ADDR The address of ZMQ server (control connection). The parameter overrides the address defined by the environment variable QSERVER_ZMQ_CONTROL_ADDRESS_FOR_SERVER. The default address is used if the parameter or the environment variable is not defined. Address format: 'tcp://*:60615' (default: 'tcp://*:60615'). --zmq-addr ZMQ_ADDR The parameter is deprecated and will be removed in future releases. Use --zmq-control-addr instead. --startup-profile STARTUP_PROFILE The name of IPython profile used to find the location of startup files. Example: if IPython is configured to look for profiles in '~/.ipython' directory (default behavior) and the profile name is 'testing', then RE Manager will look for startup files in '~/.ipython/profile_testing/startup' directory. If IPython-based worker is used, the code in the startup profile or the default profile is always executed before running a startup module or a script --startup-module STARTUP_MODULE The name of the module with startup code. The module is imported each time the RE Worker environment is opened. Example: 'some.startup.module'. Paths to the list of existing plans and devices (--existing-plans-and-devices) and user group permissions (--user- group-permissions) must be explicitly specified if this option is used. --startup-script STARTUP_SCRIPT The path to the script with startup code. The script is loaded each time the RE Worker environment is opened. Example: '~/startup/scripts/scripts.py'. Paths to the list of existing plans and devices (--existing-plans-and-devices) and user group permissions (--user-group-permissions) must be explicitly specified if this option is used. --startup-dir STARTUP_DIR Path to directory that contains a set of startup files (*.py and *.ipy). All the scripts in the directory will be sorted in alphabetical order of their names and loaded in the Run Engine Worker environment. The set of startup files may be located in any accessible directory. The value is ignored if --startup-profile is specified. --ignore-invalid-plans {ON,OFF} Ignore plans with unsupported signatures When loading startup code or executing scripts. The default behavior is to raise an exception. If the parameter is set, the message is printed for each invalid plan and only plans that were processed correctly are included in the list of existing plans (default: OFF). --device-max-depth DEVICE_MAX_DEPTH Default maximum depth for devices included in the list of existing devices: 0 - unlimited depth (full tree of subdevices is included for all devices except areadetectors), 1 - only top level devices are included, 2 - top level devices and subdevices are included, etc. (default: 0). --existing-plans-devices EXISTING_PLANS_AND_DEVICES_PATH Path to file that contains the list of existing plans and devices. The path may be a relative path to the profile collection directory. If the path is directory, then the default file name 'existing_plans_and_devices.yaml' is used. --update-existing-plans-devices {NEVER,ENVIRONMENT_OPEN,ALWAYS} Select when the list of existing plans and devices stored on disk should be updated. The available choices are not to update the stored lists (NEVER), update the lists when the environment is opened (ENVIRONMENT_OPEN) or update the lists each the lists are changed (ALWAYS) (default: ENVIRONMENT_OPEN) --user-group-permissions USER_GROUP_PERMISSIONS_PATH Path to file that contains lists of plans and devices available to users. The path may be a relative path to the profile collection directory. If the path is a directory, then the default file name 'user_group_permissions.yaml' is used. --user-group-permissions-reload {NEVER,ON_REQUEST,ON_STARTUP} Select when user group permissions are reloaded from disk. Options: 'NEVER' - RE Manager never attempts to load permissions from disk file. If permissions fail to load from Redis, they are loaded from disk at the first startup of RE Manager or on request. 'ON_REQUEST' - permissions are loaded from disk file when requested by 'permission_reload' API call. 'ON_STARTUP' - permissions are loaded from disk each time RE Manager is started or when 'permission_reload' API request is received (default: ON_STARTUP) --redis-addr REDIS_ADDR The address of Redis server, e.g. 'localhost', '127.0.0.1', 'localhost:6379' (default: localhost). --redis-name-prefix REDIS_NAME_PREFIX The prefix for the names of Redis keys used by RE Manager (default: qs_default). --kafka-topic KAFKA_TOPIC The kafka topic to publish to. --kafka-server KAFKA_SERVER Bootstrap server to connect (default: 127.0.0.1:9092). --zmq-data-proxy-addr ZMQ_DATA_PROXY_ADDR The address of ZMQ proxy used to publish data. If the parameter is specified, RE is subscribed to 'bluesky.callbacks.zmq.Publisher' and documents are published via 0MQ proxy. 0MQ Proxy (see Bluesky 0MQ documentation) should be started before plans are executed. The address should be in the form '127.0.0.1:5567' or 'localhost:5567'. The address is passed to 'bluesky.callbacks.zmq.Publisher'. It is recommended to use Kafka instead of 0MQ proxy in production data acquisition systems and use Kafka instead. --keep-re Keep RE created in profile collection. If the flag is set, RE must be created in the profile collection for the plans to run. RE will also keep all its subscriptions. Also must be subscribed to the Data Broker inside the profile collection, since '--databroker-config' argument is ignored. --use-ipython-kernel {ON,OFF} Run the Run Engine worker in IPython kernel (default: OFF). --ipython-dir IPYTHON_DIR The path to IPython root directory, which contains profiles. Overrides IPYTHONDIR environment variable. The parameter is ignored if IPython kernel is not used. --ipython-matplotlib IPYTHON_MATPLOTLIB Default Matplotlib backend, typically 'qt5'. The parameter have the same meaning and accepts the same values as --matplotlib parameter of IPython. The value is passed directly to IPython kernel. The parameter is ignored if the worker is running pure Python (--use-ipython-kernel is OFF). --ipython-kernel-ip IPYTHON_KERNEL_IP IP address for IPython kernel. The IP is passed to the IPython kernel at startup and returned to clients as part of kernel connection info ('config_get' API). Accepted values are 'localhost' (sets IP to '127.0.0.1'), 'auto' (attempts to automatically find network IP address of the server), or an explicitly specified IP address of the server. If the IP address is 'localhost' or '127.0.0.1', the kernel can not be accessed from remote machines. The parameter is ignored if worker is not using IPython. Default: localhost. --use-persistent-metadata Use msgpack-based persistent storage for scan metadata. Currently this is the preferred method to keep continuously incremented sequence of Run IDs between restarts of RE. --databroker-config DATABROKER_CONFIG Name of the Data Broker configuration file. Configure console output: The arguments allow to configure printing and publishing of the console output generated by RE Manager. The arguments allow to set the address of 0MQ socket and enable/disable printing and/or publishing of the console output. --zmq-info-addr ZMQ_INFO_ADDR The address of ZMQ server socket used for publishing information on the state of RE Manager and currently running processes. Currently only the captured STDOUT and STDERR published in 'QS_Console' topic. The parameter overrides the address defined by the environment variable 'QSERVER_ZMQ_INFO_ADDRESS_FOR_SERVER'. The default address is used if the parameter or the environment variable is not defined. Address format: 'tcp://*:60625' (default: tcp://*:60625). --zmq-publish-console-addr ZMQ_PUBLISH_CONSOLE_ADDR The parameter is deprecated and will be removed in future releases. Use --zmq-info-addr instead. --zmq-publish-console {ON,OFF} Enable (ON) or disable (OFF) publishing of console output to 0MQ (default: OFF). --console-output {ON,OFF} Enable (ON) or disable (OFF) printing of console output in the Re Manager terminal. (default: ON) Logging verbosity settings: The default logging settings (loglevel=INFO) provide optimal amount of data to monitor the operation of RE Manager. Select '--verbose' option to see detailed data on received and sent messages, added and executed plans, etc. Use options '--quiet' and '--silent' to see only warnings and error messages or disable logging output. --verbose Set logger level to DEBUG. --quiet Set logger level to WARNING. --silent Disables logging output. .. _qserver_cli: qserver ------- ``qserver`` CLI tool allows to communicate with the server by typing commands in command line. The tool is primarily intended for testing or diagnostics of Queue Server and emergency use in production. The tool supports most of the Queue Server 0MQ API including submitting plans, opening and closing of RE Worker environment, starting and stopping the queue, etc. Refer to ``qserver`` help for the full list of supported commands. If RE Manager 0MQ address is different from the default, use the optional ``--zmq-control-addr`` parameter or ``QSERVER_ZMQ_CONTROL_ADDRESS`` to pass the address to ``qserver``. If encryption is enabled at RE Manager, set the environment variable ``QSERVER_ZMQ_PUBLIC_KEY`` to a string representing valid public address of the 0MQ server. Use :ref:`qserver_zmq_keys_cli` tool to generate a new public/private key pair or generate public key from known server private key. `qserver` may used in monitoring mode (``qserver monitor``). In this mode the tool periodically requests and displays the status of Queue Server. ``qserver -h`` displays help information: .. code-block:: $ qserver -h usage: qserver [-h] [--zmq-control-addr ZMQ_CONTROL_ADDR] [--address ADDRESS] [--lock-key LOCK_KEY] command [command ...] Command-line tool for communicating with RE Monitor. bluesky-queueserver version 0.0.20. positional arguments: command a sequence of keywords and parameters that define the command options: -h, --help show this help message and exit --zmq-control-addr ZMQ_CONTROL_ADDR, -a ZMQ_CONTROL_ADDR Address of the control socket of RE Manager. The parameter overrides the address set using the environment variable QSERVER_ZMQ_CONTROL_ADDRESS. The default value is used if the address is not set using the parameter or the environment variable. Address format: 'tcp://127.0.0.1:60615' (default: 'tcp://localhost:60615'). --address ADDRESS The parameter is deprecated and will be removed. Use --zmq-control- addr instead. --lock-key LOCK_KEY, -k LOCK_KEY Lock key. The key is an arbitrary string is used to lock and unlock RE Manager ('lock' and 'unlock' API) and control the manager when the environment or the queue is locked. If RE Manager is configured to use encrypted ZeroMQ communication channel, the encryption must also be enabled before running 'qserver' CLI tool by setting the environment variable QSERVER_ZMQ_PUBLIC_KEY to the value of a valid public key (z85-encoded 40 character string): export QSERVER_ZMQ_PUBLIC_KEY='' Encryption is disabled by default. Examples of CLI commands ------------------------ qserver -h # Display help qserver monitor # Start 'qserver' in monitoring mode qserver ping # Send 'ping' request to RE Manager via ZMQ qserver status # Request status of RE Manager qserver config # Get RE Manager config qserver environment open # Open RE environment qserver environment close # Close RE environment qserver environment destroy # Destroy RE environment (kill RE worker process) qserver environment update # Update the worker state based on contents of worker namespace qserver environment update background # Update the worker state as a background task qserver existing plans # Request the list of existing plans qserver existing devices # Request the list of existing devices qserver allowed plans # Request the list of allowed plans qserver allowed devices # Request the list of allowed devices qserver permissions reload # Reload user permissions and generate lists of allowed plans and devices. qserver permissions reload lists # Same, but reload lists of existing plans and devices from disk. qserver permissions set # Set user group permissions (from .yaml file) qserver permissions get # Get current user group permissions qserver queue add plan '' # Add plan to the back of the queue qserver queue add instruction # Add instruction to the back of the queue qserver queue add plan front '' # Add plan to the front of the queue qserver queue add plan back '' # Add plan to the back of the queue qserver queue add plan 2 '' # Insert plan at position 2 qserver queue add instruction 2 # Insert instruction at position 2 qserver queue add plan -1 '' # Insert plan at position -1 qserver queue add plan before '' '' # Insert the plan before the plan with given UID qserver queue add plan after '' '' # Insert the plan after the plan with given UID NOTE: Position indices are 0-based. Inserting a plan to position 0 pushes it to the front of the queue. Negative position indices are counted from the back of the queue. Request for a plan with index -1 returns the last plan of the queue. Inserting a plan at position -1 makes it previous to last. qserver queue update plan '' # Update item with with a plan qserver queue replace plan '' # Replace item with with a plan qserver queue update instruction '' # Update item with with an instruction qserver queue replace instruction '' # Replace item with with an instruction qserver queue execute plan '' # Immediately execute the plan qserver queue execute instruction # Immediately execute an instruction Example of JSON specification of a plan: '{"name": "count", "args": [["det1", "det2"]], "kwargs": {"num": 10, "delay": 1}}' Supported queue instructions: queue-stop # stops execution of the queue qserver queue get # Request the list of items (plans or instructions) in the queue qserver queue clear # Clear the queue (remove all plans from the queue) qserver queue item get # Request the last item in the queue qserver queue item get back # Request the last item in the queue qserver queue item get front # Request the first item in the queue qserver queue item get 2 # Request the item at position 2 qserver queue item get '' # Request the item with given Item UID qserver queue item remove # Remove the last item from the queue qserver queue item remove back # Remove the last item from the queue qserver queue item remove front # Remove the first item from the queue qserver queue item remove 2 # Remove the item at position 2 qserver queue item remove '' # Remove the item with the given UID qserver queue item move 2 5 # Move item from position 2 to position 5 of the queue qserver queue item move back front # Move item from the back to the front of the queue qserver queue item move front -2 # Move item from the front of the queue to position -2 qserver queue item move '' 5 # Move item with UID to position 5 qserver queue item move 2 before '' # Place item at position 2 before an item with qserver queue item move 2 after '' # Place item at position 2 after an item with qserver queue item move '' before '' # Place item with before item with qserver queue start # Start execution of the queue qserver queue stop # Request execition of the queue to stop after current plan qserver queue stop cancel # Cancel request to stop execution of the queue # Enable and disable autostart qserver queue autostart enable qserver queue autostart disable # Change the queue mode. Enable/disable LOOP and IGNORE_FAILURES modes: qserver queue mode set loop True qserver queue mode set loop False qserver queue mode set ignore_failures True qserver queue mode set ignore_failures False # The following requests are forwarded to the Run Engine: qserver re pause # Request to PAUSE currently executed plan at the next checkpoint qserver re pause deferred # Request to PAUSE currently executed plan at the next checkpoint qserver re pause immediate # Request to immediately PAUSE currently executed plan qserver re resume # RESUME execution of a paused plan qserver re stop # STOP execution of a paused plan qserver re abort # ABORT execution of a paused plan qserver re halt # HALT execution of a paused plan qserver re runs # Get the list of active runs (runs generated by the currently running plans) qserver re runs active # Get the list of active runs qserver re runs open # Get the list of open runs (subset of active runs) qserver re runs closed # Get the list of closed runs (subset of active runs) qserver history get # Request plan history qserver history clear # Clear plan history qserver function execute # Start execution of a function qserver function execute background # ... in the background thread Example of JSON specification of a function ("args" and "kwargs" are optional): '{"name": "function_sleep", "args": [20], "kwargs": {}}' qserver script upload # Upload a script to RE Worker environment qserver script upload background # ... in the background qserver script upload update-re # ... allow 'RE' and 'db' to be updated qserver script upload keep-lists # ... leave lists of allowed and existing plans and devices # unchanged (saves processing time) qserver task result # Load status or result of a task with the given UID qserver task status # Check status of a task with the given UID qserver kernel interrupt # Send interrupt (Ctrl-C) to IPython kernel qserver kernel interrupt task # ... if the manager is executing a task qserver kernel interrupt plan # ... if the manager is executing a plan qserver kernel interrupt task plan # ... if the manager is executing a plan or a task qserver lock environment -k 90g94 # Lock the environment qserver lock environment "Locked for 1 hr" -k 90g94 # Add a text note qserver lock queue -k 90g94 # Lock the queue qserver lock all -k 90g94 # Lock environment and the queue qserver lock info # Load lock status qserver lock info -k 90g94 # Load lock status and validate the key qserver unlock -k 90g94 # Unlock RE Manager qserver manager stop # Safely exit RE Manager application qserver manager stop safe on # Safely exit RE Manager application qserver manager stop safe off # Force RE Manager application to stop NOTE: Exit with 'safe on' option will succeed only if RE Manager is in IDLE state (queue is not running). If called with 'safe off' option, the request will force RE Manager to terminate RE Worker process and exit even if a plan is running. qserver manager kill test # Kills RE Manager by stopping asyncio event loop. Used only for testing. .. _qserver_list_plans_devices_cli: qserver-list-plans-devices -------------------------- ``qserver-list-plans-devices`` loads the startup code and generates the lists of existing plans and devices. The tool may be used for the following purposes: - Validation of startup code. If startup code is successfully loaded by ``qserver-list-plans-devices`` it is very likely that it will be successfully loaded into the RE Worker environment. - Generation of the list of existing plans and devices (``existing_plans_and_devices.yaml``). Queue Server may be configured to automatically generate or update the file (see :ref:`update_existing_plans_devices`), but sometimes it may be more convenient to do it manually or as part of the installation script. The lists of existing plans and devices must be updated each time new devices or plans are added to or removed from the startup code or signatures of the existing plans are modified (e.g. a parameter is added or removed, type annotation or text description is changed etc.), since those changes may affect other functionality such as plan validation or lists of allowed plans and devices distributed to clients. The default name for the output file is ``existing_plans_and_devices.yaml``. The file is always saved to the current directory unless a different path is specified using the ``--file-dir`` parameter. If the output file must have name different from the default, the new name can be specified using the ``--file-name`` parameter. The tool may load startup code from IPython startup script collection, Python script or Python module. Use ``--startup-dir``, ``--startup-script`` and ``--startup-module`` parameters to specify the path to the directory with startup files, the path to a startup script or module name respectively. ``qserver-list-plans-devices -h`` displays help information: .. code-block:: $ qserver-list-plans-devices -h usage: qserver-list-plans-devices [-h] [--file-dir FILE_DIR] [--file-name FILE_NAME] [--startup-profile STARTUP_PROFILE] [--startup-dir STARTUP_DIR | --startup-module STARTUP_MODULE_NAME | --startup-script STARTUP_SCRIPT_PATH] [--ipython-dir IPYTHON_DIR] [--use-ipython-kernel {ON,OFF}] [--ignore-invalid-plans {ON,OFF}] [--device-max-depth DEVICE_MAX_DEPTH] Bluesky-QServer: CLI tool for generating the list of plans and devices from beamline startup scripts. bluesky-queueserver version 0.0.20 options: -h, --help show this help message and exit --file-dir FILE_DIR Directory name where the list of plans and devices is saved. By default, the list is saved to the file 'existing_plans_and_devices.yaml' in the current directory. --file-name FILE_NAME Name of the file where the list of plans and devices is saved. Default file name: 'existing_plans_and_devices.yaml'. --startup-profile STARTUP_PROFILE The name of IPython profile used to find the location of startup files. Example: if IPython is configured to look for profiles in '~/.ipython' directory (default behavior) and the profile name is 'testing', then RE Manager will look for startup files in '~/.ipython/profile_testing/startup' directory. If IPython-based worker is used, the code in the startup profile or the default profile is always executed before running a startup module or a script --startup-dir STARTUP_DIR Path to directory that contains a set of startup files (*.py and *.ipy). All the scripts in the directory will be sorted in alphabetical order of their names and loaded in the Run Engine Worker environment. The set of startup files may be located in any accessible directory. For example, 'qserver-list-plans-devices --startup-dir .' loads startup files from the current directory and saves the lists to the file in current directory. --startup-module STARTUP_MODULE_NAME The name of the module that contains the startup code. The module must be installed in the current environment For example, 'qserver-list- plans-devices --startup-module some.startup.module' loads startup code from the module 'some.startup.module' and saves results to the file in the current directory. --startup-script STARTUP_SCRIPT_PATH The path to the script with startup code. For example, 'qserver-list- plans-devices --startup-script ~/startup/scripts/script.py' loads startup code from the script and saves the results to the file in the current directory. --ipython-dir IPYTHON_DIR The path to IPython root directory, which contains profiles. Overrides IPYTHONDIR environment variable. --use-ipython-kernel {ON,OFF} Run the Run Engine worker in IPython kernel (default: OFF). --ignore-invalid-plans {ON,OFF} Ignore plans with unsupported signatures When loading startup code or executing scripts. The default behavior is to raise an exception. If the parameter is set, the message is printed for each invalid plan and only plans that were processed correctly are included in the list of existing plans (default: OFF). --device-max-depth DEVICE_MAX_DEPTH Default maximum depth for devices included in the list of existing devices: 0 - unlimited depth (full tree of subdevices is included for all devices except areadetectors), 1 - only top level devices are included, 2 - top level devices and subdevices are included, etc. (default: 0). .. _qserver_zmq_keys_cli: qserver-zmq-keys ---------------- Use this tool to generate random public-private key pairs for securing 0MQ control communication channel used by RE Manager: - **private key** - set as a value of ``QSERVER_ZMQ_PRIVATE_KEY_FOR_SERVER`` environment variable at workstation or server running RE Manager - **public key** - set as a value of ``QSERVER_ZMQ_PUBLIC_KEY`` environment variable at the workstation(s) running the client application(s). If server private key is know, the public key may be generated by passing the private key to ``qserver-zmq-keys`` using ``--zmq-private_key``. ``qserver-zmq-keys -h`` displays help information: .. code-block:: $ qserver-zmq-keys -h usage: qserver-zmq-keys [-h] [--zmq-private-key ZMQ_PRIVATE_KEY] Bluesky-QServer: ZMQ security: Generate public-private key pair for ZeroMQ control communication channel. bluesky-queueserver version 0.0.20. Generate new public-private key pair for secured 0MQ control connection between RE Manager and client applications. If private key is passed as ``--zmq-private-key`` parameter, then the generated key pair is based on the provided private key. optional arguments: -h, --help show this help message and exit --zmq-private-key ZMQ_PRIVATE_KEY Private key used by RE Manager. If the private key is provided, then the public key is generated based on the private key. This option allows to create (recover) public key based on known private key. The passed value should be 40 character string containing z85 encrypted key. .. _qserver_console_monitor_cli: qserver-console-monitor ----------------------- ``qserver-console-monitor`` is a simple application that subscribes to the console output (``stdout`` and ``stderr``) published by RE Manager via 0MQ and prints the received messages to terminal (to ``stdout``). The console output printed by the monitor is expected to be identical to the output printed in RE Manager terminal. The monitor may be run on the same workstation as RE Manager or any computer, which can access the workstation running RE Manager over the network. If the address of the 0MQ socket is different from default, it can be passed to the monitor application as a parameter (``--zmq-info-addr``). RE Manager does not publishing the console output to 0MQ socket by default. Publishing can be enabled by starting RE Manager with the parameter ``--zmq-publish-console``: .. code-block:: start-re-manager --zmq-publish-console ON (see :ref:`start_re_manager_console_output`). ``qserver-console-monitor -h`` displays help information: .. code-block:: $ qserver-console-monitor -h usage: qserver-console-monitor [-h] [--zmq-info-addr ZMQ_INFO_ADDR] [--zmq-subscribe-addr ZMQ_SUBSCRIBE_ADDR] Queue Server Console Monitor: CLI tool for remote monitoring of console output published by RE Manager. bluesky-queueserver version 0.0.20 optional arguments: -h, --help show this help message and exit --zmq-info-addr ZMQ_INFO_ADDR The address of RE Manager socket used for publishing console output. The parameter overrides the address set using QSERVER_ZMQ_INFO_ADDRESS environment variable. The default value is used if the address is not set using the parameter or the environment variable. Address format: 'tcp://127.0.0.1:60625' (default: tcp://localhost:60625). --zmq-subscribe-addr ZMQ_SUBSCRIBE_ADDR The parameter is deprecated and will be removed. Use --zmq-info-addr instead. .. _qserver_clear_lock_cli: qserver-clear-lock ------------------ ``qserver-clear-lock`` allows to clear RE Manager lock stored in Redis. The manager lock is not cleared by restarting the manager: it must be explicitly cleared using a valid lock key (used to lock the manager) or an emergency lock key (optional). If the key is lost and the emergency lock key is not set or known, then the lock could be cleared by running ``qserver-clear-lock`` and restarting RE Manager application or service. The utility needs access to Redis server used by RE Manager. If Redis address is different from default, the correct address must be passed using the parameter ``--redis-addr``. .. code-block:: $ qserver-clear-lock -h usage: qserver-clear-lock [-h] [--redis-addr REDIS_ADDR] [--redis-name-prefix REDIS_NAME_PREFIX] Bluesky-QServer: Clear RE Manager lock. bluesky-queueserver version 0.0.20. Recover locked RE Manager if the lock key is lost. The utility requires access to Redis used by RE Manager. Provide the address of Redis service using '--redis-addr' parameter. Restart the RE Manager service after clearing the lock. optional arguments: -h, --help show this help message and exit --redis-addr REDIS_ADDR The address of Redis server, e.g. 'localhost', '127.0.0.1', 'localhost:6379' (default: localhost). --redis-name-prefix REDIS_NAME_PREFIX The prefix for the names of Redis keys used by RE Manager (default: qs_default). .. _qserver_console_cli: qserver-console --------------- Starts Jupyter Console connected to IPython kernel running in the worker process. RE Manager must be started with enabled ``--use-ipython-kernel`` option (using CLI parameter, config file parameter or the environment variable). The console can not be started if the worker environment is closed and the kernel is not running. Use ``Ctrl-D`` to exit the console. Typing ``quit`` or ``exit`` in the console will close the worker environment. .. code-block:: $ qserver-console -h usage: qserver-console [-h] [--zmq-control-addr ZMQ_CONTROL_ADDR] Bluesky-QServer: Start Jupyter console for IPython kernel running in the worker process. bluesky-queueserver version 0.0.20. Requests IPython kernel connection info from RE Manager and starts Jupyter Console. The RE Worker must be running (environment opened) and using IPython kernel. The address of 0MQ control port of RE Manager can be passed as a parameter or an environment variable. If encryption of the control channel is enabled, the public key can be passed by setting QSERVER_ZMQ_PUBLIC_KEY environment variable. Use 'Ctrl-D' to exit the console. Typing 'quit' or 'exit' in the console will close the worker environment. options: -h, --help show this help message and exit --zmq-control-addr ZMQ_CONTROL_ADDR, -a ZMQ_CONTROL_ADDR Address of the control socket of RE Manager. The parameter overrides the address set using the environment variable QSERVER_ZMQ_CONTROL_ADDRESS. The default value is used if the address is not set using the parameter or the environment variable. Address format: 'tcp://127.0.0.1:60615' (default: 'tcp://localhost:60615'). .. _qserver_qtconsole_cli: qserver-qtconsole ----------------- Starts Jupyter Qt Console connected to the IPython kernel running in the worker process. Jupyter Qt Console is extended Qt-based version of Jupyter Console. The command behaves similarly to :ref:`qserver_console_cli`.