Optimizing KB Mirrors with Bayesian Optimization#

In this tutorial, you will learn how to use Blop to optimize a Kirkpatrick-Baez (KB) mirror system. By the end, you will understand:

  • How degrees of freedom (DOFs) represent the parameters you can adjust in an experiment

  • How objectives define what you’re trying to optimize

  • How tracking metrics let you monitor values without optimizing them

  • How to write an evaluation function that extracts results from experimental data

  • How the Agent coordinates the optimization loop

  • How to check optimization health mid-run and continue

We’ll work with a simulated KB mirror beamline, but the concepts apply directly to real experimental setups.

What are KB Mirrors?#

KB mirror systems use two curved mirrors to focus X-ray beams. Each mirror has adjustable curvature—getting both just right produces a tight, intense focal spot. We’ll frame this as a single-objective optimization problem: minimize the beam’s FWHM (full width at half maximum) on the detector, subject to a minimum intensity constraint.

The image below shows our simulated setup: a beam from a geometric source propagates through a pair of toroidal mirrors that focus it onto a screen.

xrt_blop_layout_w.jpg

Setting Up the Environment#

Before we can optimize, we need to set up the data infrastructure. Blop uses Bluesky to run experiments and Tiled to store and retrieve data.

import logging
import warnings
from pathlib import PurePath

import matplotlib.pyplot as plt
import numpy as np
from ax.api.protocols import IMetric

from bluesky.run_engine import RunEngine
from bluesky_tiled_plugins import TiledWriter
from ophyd_async.core import StaticPathProvider, UUIDFilenameProvider
from tiled.client import from_uri  # type: ignore[import-untyped]
from tiled.client.container import Container
from tiled.server import SimpleTiledServer

from blop.ax import Agent, Objective, RangeDOF
from blop.ax.objective import OutcomeConstraint
from blop.protocols import EvaluationFunction

# Import simulation devices (requires: pip install -e sim/)
from blop_sim.backends.models.xrt_kb_model import KBBackend
from blop_sim.devices import DetectorDevice
from blop_sim.devices.xrt import KBMirror

# Suppress noisy logs from httpx and dependency deprecation warnings
logging.getLogger("httpx").setLevel(logging.WARNING)
warnings.filterwarnings("ignore", category=FutureWarning)

# Enable interactive plotting
plt.ion()

DETECTOR_STORAGE = "/tmp/blop/sim"

[INFO 07-28 17:30:55] ax.storage.sqa_store.with_db_settings_base: Ax SQL storage initialized with SQLAlchemy 2.0.51

Next, we create a local Tiled server. The TiledWriter callback will save experimental data to this server, and our evaluation function will read from it.

tiled_server = SimpleTiledServer(readable_storage=[DETECTOR_STORAGE])
tiled_client = from_uri(tiled_server.uri)
tiled_writer = TiledWriter(tiled_client)

RE = RunEngine({})
RE.subscribe(tiled_writer)

Tiled version 0.2.14

0

Defining Degrees of Freedom#

Degrees of freedom (DOFs) are the parameters the optimizer can adjust. In our KB system, we control the curvature radius of each mirror. Let’s define the search space:

# Define search ranges for each mirror's curvature radius
# The optimal values (~38000 and ~21000) are intentionally placed
# away from the center to make the optimization more realistic
VERTICAL_BOUNDS = (25000, 45000)    # Optimal ~38000 is in upper portion
HORIZONTAL_BOUNDS = (15000, 35000)  # Optimal ~21000 is in lower portion

Now we create the simulation backend and individual devices. Each RangeDOF wraps an actuator (something we can move) with bounds that constrain the search space:

# Create XRT simulation backend
backend = KBBackend()

# Create individual KB mirror devices
kbv = KBMirror(backend, mirror_index=0, initial_radius=38000, name="kbv")
kbh = KBMirror(backend, mirror_index=1, initial_radius=21000, name="kbh")

# Create detector device
det = DetectorDevice(backend, StaticPathProvider(UUIDFilenameProvider(), PurePath(DETECTOR_STORAGE)), name="det")

# Define DOFs using mirror radius signals
dofs = [
    RangeDOF(actuator=kbv.radius, bounds=VERTICAL_BOUNDS, parameter_type="float"),
    RangeDOF(actuator=kbh.radius, bounds=HORIZONTAL_BOUNDS, parameter_type="float"),
]

The actuator is the device that physically changes the parameter. The bounds tell the optimizer what range of values to explore. Think of DOFs as the “knobs” the optimizer can turn.

Defining the Objective and Constraints#

For beam focusing, we use a single objective: minimize the beam FWHM (full width at half maximum). This is more sample-efficient than multi-objective optimization because the optimizer only needs to model one response surface.

We also track intensity as a metric without optimizing it directly. An OutcomeConstraint ensures the optimizer avoids configurations where the beam misses the detector entirely:

# Single objective: minimize the geometric-mean FWHM
objectives = [
    Objective(name="fwhm", minimize=True),
]

# Track intensity without optimizing it
intensity_metric = IMetric(name="intensity")

# Soft constraint: reject configurations where most rays miss the screen
outcome_constraints = [
    OutcomeConstraint(constraint="i >= 10000", i=intensity_metric),
]

Using a single objective with an outcome constraint gives us the best of both worlds: focused optimization on spot size, with a safety net ensuring we don’t “optimize” toward configurations where the beam is simply lost.

Writing an Evaluation Function#

The evaluation function is the bridge between raw experimental data and the optimizer. After each measurement, the optimizer needs to know how well that configuration performed. Our evaluation function:

  1. Receives a run UID and the suggestions that were tested

  2. Reads the beam images from Tiled

  3. Computes FWHM from the marginal beam profiles

  4. Returns outcome values for each suggestion

We compute FWHM using marginal profiles — projecting the 2D image onto each axis by summing, then finding where the 1D profile crosses half its peak value. This approach is robust to noise and dead pixels (they get averaged out in the projection) and doesn’t require curve fitting.

class DetectorEvaluation(EvaluationFunction):
    def __init__(self, tiled_client: Container):
        self.tiled_client = tiled_client

    def _fwhm_from_profile(self, profile: np.ndarray) -> float:
        """Compute FWHM from a 1D marginal profile.

        Finds the half-maximum crossing points with sub-pixel interpolation.
        Returns a large value if the beam is too dim or fills the entire detector.
        """
        peak = profile.max()
        if peak == 0:
            return float(len(profile))  # No signal — return detector width as penalty

        half_max = peak / 2.0
        above = profile >= half_max
        if not above.any():
            return float(len(profile))

        indices = np.where(above)[0]
        left_idx = indices[0]
        right_idx = indices[-1]

        # Sub-pixel interpolation at left crossing
        if left_idx > 0:
            left = left_idx - 1 + (half_max - profile[left_idx - 1]) / (profile[left_idx] - profile[left_idx - 1])
        else:
            left = 0.0

        # Sub-pixel interpolation at right crossing
        if right_idx < len(profile) - 1:
            right = right_idx + (half_max - profile[right_idx]) / (profile[right_idx + 1] - profile[right_idx])
        else:
            right = float(len(profile) - 1)

        return right - left

    def _compute_stats(self, image: np.ndarray) -> tuple[float, float]:
        """Compute FWHM and integrated intensity from a beam image.

        Returns
        -------
        fwhm : float
            Geometric mean of the horizontal and vertical FWHM (in pixels).
        intensity : float
            Total integrated intensity (sum of all pixel values).
        """
        gray = image.squeeze().astype(np.float64)
        if gray.ndim == 3:
            gray = gray.mean(axis=-1)

        # Integrated intensity (total flux on detector)
        intensity = gray.sum()

        if intensity == 0:
            return 400.0, 0.0  # No beam — return max FWHM penalty

        # Marginal profiles: project onto each axis
        x_profile = gray.sum(axis=0)  # sum along Y rows -> X profile
        y_profile = gray.sum(axis=1)  # sum along X cols -> Y profile

        fwhm_x = self._fwhm_from_profile(x_profile)
        fwhm_y = self._fwhm_from_profile(y_profile)

        # Geometric mean FWHM — targets a small, round spot
        fwhm = np.sqrt(fwhm_x * fwhm_y)

        return float(fwhm), float(intensity)

    def __call__(self, uid: str, suggestions: list[dict]) -> list[dict]:
        outcomes = []
        run = self.tiled_client[uid]

        # Read beam images from detector
        images = run["primary/det_image"].read()

        # Suggestion IDs stored in start document metadata
        suggestion_ids = [suggestion["_id"] for suggestion in run.metadata["start"]["blop_suggestions"]]

        # Compute statistics from each image
        for idx, sid in enumerate(suggestion_ids):
            image = images[idx]
            fwhm, intensity = self._compute_stats(image)

            outcome = {
                "_id": sid,
                "fwhm": fwhm,
                "intensity": intensity,
            }
            outcomes.append(outcome)
        return outcomes

Note how we:

  1. Project the 2D image onto each axis to get 1D profiles

  2. Find the FWHM of each profile using half-maximum crossings

  3. Combine them into a single geometric-mean FWHM metric

  4. Track integrated intensity for the outcome constraint

  5. Link each outcome back to its suggestion via the _id field

Creating and Running the Agent#

The Agent brings everything together. It:

  • Uses DOFs to know what parameters to adjust

  • Uses objectives to know what to optimize

  • Calls the evaluation function to assess each configuration

  • Builds a surrogate model to predict outcomes across the parameter space

  • Suggests the next configurations to try

agent = Agent(
    sensors=[det],
    dofs=dofs,
    objectives=objectives,
    evaluation_function=DetectorEvaluation(tiled_client),
    outcome_constraints=outcome_constraints,
    name="xrt-blop-demo",
    description="A demo of the Blop agent with XRT simulated beamline",
    experiment_type="demo",
)

# Register intensity as a tracking metric (monitored but not optimized)
agent.ax_client.configure_metrics([intensity_metric])

The sensors list contains any devices that produce data during acquisition. The outcome_constraints tell the optimizer to prefer configurations satisfying the intensity constraint. The configure_metrics call registers intensity as a tracking metric so it appears in analyses and summaries.

Running the Optimization#

Let’s start the optimization. We’ll begin with a batch of 10 points to build an initial model of the parameter space—this includes a center-of-space sample plus quasi-random exploration points.

# Run 1 iteration with a batch of 10 points for initial exploration
RE(agent.optimize(1, n_points=10))



╭───────────────────────────────────────────────── Optimization ──────────────────────────────────────────────────╮
│ Optimizer  AxOptimizer                                                                                          │
│ Actuators  kbv-radius, kbh-radius                                                                               │
│ Sensors    det                                                                                                  │
│ Iterations 1  Points/iter 10                                                                                    │
│ Run UID    5385d54a-987b-434c-8b80-2ec8fcb32f1b                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯



[INFO 07-28 17:31:01] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol', use_existing_trials_for_initialization=True), GenerationNode(name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, generator_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')], suggested_experiment_status=ExperimentStatus.INITIALIZATION, pausing_criteria=[MaxTrialsAwaitingData(threshold=5)]), GenerationNode(name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, generator_key_override=None)], transition_criteria=None, suggested_experiment_status=ExperimentStatus.OPTIMIZATION, pausing_criteria=None)]) chosen based on user input and problem structure.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 0 with parameters {'kbv-radius': 35000.0, 'kbh-radius': 25000.0} using GenerationNode CenterOfSearchSpace.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 1 with parameters {'kbv-radius': 30785.48789, 'kbh-radius': 31834.090948} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 2 with parameters {'kbv-radius': 42970.95716, 'kbh-radius': 19386.426341} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 3 with parameters {'kbv-radius': 36555.225141, 'kbh-radius': 29791.710041} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 4 with parameters {'kbv-radius': 29991.130922, 'kbh-radius': 21409.296561} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 5 with parameters {'kbv-radius': 25742.566008, 'kbh-radius': 26905.404571} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 6 with parameters {'kbv-radius': 37933.317386, 'kbh-radius': 24316.142984} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 7 with parameters {'kbv-radius': 41280.810907, 'kbh-radius': 33930.410799} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 8 with parameters {'kbv-radius': 34721.125867, 'kbh-radius': 17271.625474} using GenerationNode Sobol.
[INFO 07-28 17:31:01] ax.api.client: Generated new trial 9 with parameters {'kbv-radius': 33254.698589, 'kbh-radius': 28155.618124} using GenerationNode Sobol.
[INFO 07-28 17:31:03] ax.api.client: Trial 6 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 0 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 9 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 3 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 1 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 5 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 4 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 8 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 2 marked COMPLETED.
[INFO 07-28 17:31:03] ax.api.client: Trial 7 marked COMPLETED.





────────────────────────────────────────── Iteration 1 / 1  (10 points) ───────────────────────────────────────────



  Acquire UID  a6fc5bd6-e64b-4ed8-96c7-6e97d498438b



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │             0 │      25000 │      35000 │ 139.289 │     47119 │
│     1 │             1 │    31834.1 │    30785.5 │ 299.972 │     28714 │
│     2 │             2 │    19386.4 │      42971 │ 93.9429 │     49798 │
│     3 │             3 │    29791.7 │    36555.2 │ 128.873 │     35919 │
│     4 │             4 │    21409.3 │    29991.1 │ 78.3439 │     41608 │
│     5 │             5 │    26905.4 │    25742.6 │ 302.515 │     23606 │
│     6 │             6 │    24316.1 │    37933.3 │ 78.1923 │     48438 │
│     7 │             7 │    33930.4 │    41280.8 │ 177.505 │     29916 │
│     8 │             8 │    17271.6 │    34721.1 │ 155.104 │     45349 │
│     9 │             9 │    28155.6 │    33254.7 │ 232.251 │     38684 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 78.1923  max: 302.515  mean: 168.599
  intensity  min: 23606  max: 49798  mean: 38915.1
  (10 pts sampled)





                           Summary Statistics                           
┏━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┓
┃ Name       ┃ Type    ┃     Min ┃     Max ┃    Mean ┃     Std ┃ Count ┃
┡━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━┩
│ kbh-radius │ param   │ 17271.6 │ 33930.4 │ 25800.1 │ 5387.29 │    10 │
│ kbv-radius │ param   │ 25742.6 │   42971 │ 34823.5 │ 5221.44 │    10 │
│ fwhm       │ outcome │ 78.1923 │ 302.515 │ 168.599 │ 84.2189 │    10 │
│ intensity  │ outcome │   23606 │   49798 │ 38915.1 │  9150.1 │    10 │
└────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴───────┘



────────────────────────────────────────────── Optimization Complete ──────────────────────────────────────────────





('5385d54a-987b-434c-8b80-2ec8fcb32f1b',
 'a6fc5bd6-e64b-4ed8-96c7-6e97d498438b')













Continuing the Optimization#


The optimization state is preserved, so we can simply run more iterations:






# Run remaining 10 iterations
RE(agent.optimize(10))











╭───────────────────────────────────────────────── Optimization ──────────────────────────────────────────────────╮
│ Optimizer  AxOptimizer                                                                                          │
│ Actuators  kbv-radius, kbh-radius                                                                               │
│ Sensors    det                                                                                                  │
│ Iterations 10 more (1 completed, 11 total)                                                                      │
│ Run UID    44a6f12e-16b1-44d8-9e46-32a0208dde53                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯



[INFO 07-28 17:31:04] ax.api.client: Generated new trial 10 with parameters {'kbv-radius': 41233.596017, 'kbh-radius': 25624.421751} using GenerationNode MBM.
[INFO 07-28 17:31:04] ax.api.client: Trial 10 marked COMPLETED.
[INFO 07-28 17:31:05] ax.api.client: Generated new trial 11 with parameters {'kbv-radius': 29692.492279, 'kbh-radius': 18074.47858} using GenerationNode MBM.
[INFO 07-28 17:31:06] ax.api.client: Trial 11 marked COMPLETED.
[INFO 07-28 17:31:07] ax.api.client: Generated new trial 12 with parameters {'kbv-radius': 35822.911571, 'kbh-radius': 22804.85505} using GenerationNode MBM.
[INFO 07-28 17:31:07] ax.api.client: Trial 12 marked COMPLETED.
[INFO 07-28 17:31:08] ax.api.client: Generated new trial 13 with parameters {'kbv-radius': 38092.241389, 'kbh-radius': 20864.589956} using GenerationNode MBM.
[INFO 07-28 17:31:08] ax.api.client: Trial 13 marked COMPLETED.
[INFO 07-28 17:31:09] ax.api.client: Generated new trial 14 with parameters {'kbv-radius': 39513.712553, 'kbh-radius': 20834.261812} using GenerationNode MBM.
[INFO 07-28 17:31:10] ax.api.client: Trial 14 marked COMPLETED.
[INFO 07-28 17:31:11] ax.api.client: Generated new trial 15 with parameters {'kbv-radius': 45000.0, 'kbh-radius': 23509.152336} using GenerationNode MBM.
[INFO 07-28 17:31:11] ax.api.client: Trial 15 marked COMPLETED.
[INFO 07-28 17:31:12] ax.api.client: Generated new trial 16 with parameters {'kbv-radius': 38767.774778, 'kbh-radius': 19302.192373} using GenerationNode MBM.
[INFO 07-28 17:31:13] ax.api.client: Trial 16 marked COMPLETED.
[INFO 07-28 17:31:14] ax.api.client: Generated new trial 17 with parameters {'kbv-radius': 38899.438456, 'kbh-radius': 21419.676503} using GenerationNode MBM.
[INFO 07-28 17:31:14] ax.api.client: Trial 17 marked COMPLETED.
[INFO 07-28 17:31:15] ax.api.client: Generated new trial 18 with parameters {'kbv-radius': 26256.671034, 'kbh-radius': 21625.783628} using GenerationNode MBM.
[INFO 07-28 17:31:15] ax.api.client: Trial 18 marked COMPLETED.
[INFO 07-28 17:31:17] ax.api.client: Generated new trial 19 with parameters {'kbv-radius': 45000.0, 'kbh-radius': 15000.0} using GenerationNode MBM.
[INFO 07-28 17:31:17] ax.api.client: Trial 19 marked COMPLETED.





──────────────────────────────────────────────── Iteration 2 / 11 ─────────────────────────────────────────────────



  Acquire UID  845d45a7-f260-488b-a1ea-b2d914019167



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            10 │    25624.4 │    41233.6 │ 149.586 │     45422 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 78.1923  max: 302.515  mean: 166.87
  intensity  min: 23606  max: 49798  mean: 39506.6
  (11 pts sampled)



──────────────────────────────────────────────── Iteration 3 / 11 ─────────────────────────────────────────────────



  Acquire UID  d2829e98-052e-4547-b373-e683ac8767da



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃   fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            11 │    18074.5 │    29692.5 │ 230.88 │     39397 │
└───────┴───────────────┴────────────┴────────────┴────────┴───────────┘



  fwhm  min: 78.1923  max: 302.515  mean: 172.204
  intensity  min: 23606  max: 49798  mean: 39497.5
  (12 pts sampled)



──────────────────────────────────────────────── Iteration 4 / 11 ─────────────────────────────────────────────────



  Acquire UID  8b449413-8811-4c17-951f-36fe757fc276



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃   fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            12 │    22804.9 │    35822.9 │ 90.782 │     49791 │
└───────┴───────────────┴────────────┴────────────┴────────┴───────────┘



  fwhm  min: 78.1923  max: 302.515  mean: 165.941
  intensity  min: 23606  max: 49798  mean: 40289.3
  (13 pts sampled)



──────────────────────────────────────────────── Iteration 5 / 11 ─────────────────────────────────────────────────



  Acquire UID  4cb9f018-2b69-4ed1-8c80-1142e87aa94e



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            13 │    20864.6 │    38092.2 │ 22.6215 │     50000 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 155.704
  intensity  min: 23606  max: 50000  mean: 40982.9
  (14 pts sampled)



──────────────────────────────────────────────── Iteration 6 / 11 ─────────────────────────────────────────────────



  Acquire UID  823e9ee6-6bb4-40aa-ad51-5804f6e40668



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            14 │    20834.3 │    39513.7 │ 29.5466 │     50000 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 147.294
  intensity  min: 23606  max: 50000  mean: 41584.1
  (15 pts sampled)



──────────────────────────────────────────────── Iteration 7 / 11 ─────────────────────────────────────────────────



  Acquire UID  a551ed00-ff95-4eae-a61a-3a187041f889



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            15 │    23509.2 │      45000 │ 153.452 │     48343 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 147.678
  intensity  min: 23606  max: 50000  mean: 42006.5
  (16 pts sampled)



──────────────────────────────────────────────── Iteration 8 / 11 ─────────────────────────────────────────────────



  Acquire UID  946aa42e-e2ca-4dab-91fe-50448018e369



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            16 │    19302.2 │    38767.8 │ 52.8638 │     49894 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 142.101
  intensity  min: 23606  max: 50000  mean: 42470.5
  (17 pts sampled)



──────────────────────────────────────────────── Iteration 9 / 11 ─────────────────────────────────────────────────



  Acquire UID  9d07b789-3c55-4f5c-a50e-e9cd5fc1480e



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            17 │    21419.7 │    38899.4 │ 31.3608 │     49997 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 135.949
  intensity  min: 23606  max: 50000  mean: 42888.6
  (18 pts sampled)



──────────────────────────────────────────────── Iteration 10 / 11 ────────────────────────────────────────────────



  Acquire UID  a7e7edaa-b257-440d-a886-5cf7d578a1d7



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            18 │    21625.8 │    26256.7 │ 111.814 │     29477 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 134.679
  intensity  min: 23606  max: 50000  mean: 42182.7
  (19 pts sampled)



──────────────────────────────────────────────── Iteration 11 / 11 ────────────────────────────────────────────────



  Acquire UID  7bfd5a73-5ab1-4e18-bb95-53930573e8e2



┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Event ┃ Suggestion ID ┃ kbh-radius ┃ kbv-radius ┃    fwhm ┃ intensity ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│     0 │            19 │      15000 │      45000 │ 246.897 │     30107 │
└───────┴───────────────┴────────────┴────────────┴─────────┴───────────┘



  fwhm  min: 22.6215  max: 302.515  mean: 140.29
  intensity  min: 23606  max: 50000  mean: 41578.9
  (20 pts sampled)





                           Summary Statistics                           
┏━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┓
┃ Name       ┃ Type    ┃     Min ┃     Max ┃    Mean ┃     Std ┃ Count ┃
┡━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━┩
│ kbh-radius │ param   │   15000 │ 33930.4 │   23353 │ 4919.74 │    20 │
│ kbv-radius │ param   │ 25742.6 │   45000 │ 36325.7 │ 5684.79 │    20 │
│ fwhm       │ outcome │ 22.6215 │ 302.515 │  140.29 │ 86.0115 │    20 │
│ intensity  │ outcome │   23606 │   50000 │ 41578.9 │ 8935.72 │    20 │
└────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴───────┘



────────────────────────────────────────────── Optimization Complete ──────────────────────────────────────────────





('44a6f12e-16b1-44d8-9e46-32a0208dde53',
 '845d45a7-f260-488b-a1ea-b2d914019167',
 'd2829e98-052e-4547-b373-e683ac8767da',
 '8b449413-8811-4c17-951f-36fe757fc276',
 '4cb9f018-2b69-4ed1-8c80-1142e87aa94e',
 '823e9ee6-6bb4-40aa-ad51-5804f6e40668',
 'a551ed00-ff95-4eae-a61a-3a187041f889',
 '946aa42e-e2ca-4dab-91fe-50448018e369',
 '9d07b789-3c55-4f5c-a50e-e9cd5fc1480e',
 'a7e7edaa-b257-440d-a886-5cf7d578a1d7',
 '7bfd5a73-5ab1-4e18-bb95-53930573e8e2')













Understanding the Results#


After optimization, we can examine what the agent learned. Ax’s compute_analyses() runs diagnostics including cross-validation of the surrogate model and optimization trace plots:






_ = agent.ax_client.compute_analyses()









/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/generators/torch/botorch_modular/generator.py:399: BotorchWarning: NSGA-II only returned 1 points.
  candidates, expected_acquisition_value, weights = acqf.optimize(
[WARNING 07-28 17:31:19] ax.adapter.base: TorchAdapter(generator=BoTorchGenerator) was not able to generate 10 unique candidates. Generated arms have the following weights, as there are repeats:
[0.1]
[ERROR 07-28 17:31:23] ax.analysis.analysis: Failed to compute TransferLearningAnalysis
[ERROR 07-28 17:31:23] ax.analysis.analysis: Traceback (most recent call last):
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/analysis/analysis.py", line 115, in compute_result
    card = self.compute(
        experiment=experiment,
        generation_strategy=generation_strategy,
        adapter=adapter,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/analysis/healthcheck/transfer_learning_analysis.py", line 104, in compute
    transferable_experiments = identify_transferable_experiments(
        search_space=experiment.search_space,
    ...<4 lines>...
        experiment_name=experiment.name,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/load.py", line 839, in identify_transferable_experiments
    experiments_search_spaces = _query_historical_experiments_given_parameters(
        parameter_names=list(search_space.parameters.keys()),
        experiment_types=experiment_types,
        config=config,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/load.py", line 759, in _query_historical_experiments_given_parameters
    with session_scope() as session:
         ~~~~~~~~~~~~~^^
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 287, in session_scope
    session = get_session()
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 263, in get_session
    init_engine_and_session_factory()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 191, in init_engine_and_session_factory
    raise ValueError("Must specify either `url` or `creator`.")
ValueError: Must specify either `url` or `creator`.









    

        
        Overview of the Entire Optimization Process 
        
        

        This analysis provides an overview of the entire optimization process. It includes visualizations of the results obtained so far, insights into the parameter and metric relationships learned by the Ax model, diagnostics such as model fit, and health checks to assess the overall health of the experiment.
        

    

    

        



    

        
        Results Analysis
        
        

        Result Analyses provide a high-level overview of the results of the optimization process so far with respect to the metrics specified in experiment design.
        

    

    

        



    

        
        Metric Effects: Predicted and observed effects for all arms in the experiment
        
        

        These pair of plots visualize the metric effects for each arm, with the Ax model predictions on the left and the raw observed data on the right. The predicted effects apply shrinkage for noise and adjust for non-stationarity in the data, so they are more representative of the reproducible effects that will manifest in a long-term validation experiment. 
        

    

    

        



    

        
        Metric Effects Pair for fwhm
        
        

        
        

    

    

        



    



    

        Modeled Arm Effects on fwhm
        
Modeled effects on fwhm. This plot visualizes predictions of the true metric changes for each arm based on Ax's model. This is the expected delta you would expect if you (re-)ran that arm. This plot helps in anticipating the outcomes and performance of arms based on the model's predictions. Note, flat predictions across arms indicate that the model predicts that there is no effect, meaning if you were to re-run the experiment, the delta you would see would be small and fall within the confidence interval indicated in the plot.

        
    

    

        




    
                            
                    




    








    

        Observed Arm Effects on fwhm
        
Observed effects on fwhm. This plot visualizes the effects from previously-run arms on a specific metric, providing insights into their performance. This plot allows one to compare and contrast the effectiveness of different arms, highlighting which configurations have yielded the most favorable outcomes. 

        
    

    

        




    
                            
                    




    








    








    

        
        Metric Effects Pair for intensity
        
        

        
        

    

    

        



    



    

        Modeled Arm Effects on intensity
        
Modeled effects on intensity. This plot visualizes predictions of the true metric changes for each arm based on Ax's model. This is the expected delta you would expect if you (re-)ran that arm. This plot helps in anticipating the outcomes and performance of arms based on the model's predictions. Note, flat predictions across arms indicate that the model predicts that there is no effect, meaning if you were to re-run the experiment, the delta you would see would be small and fall within the confidence interval indicated in the plot.

        
    

    

        




    
                            
                    




    








    

        Observed Arm Effects on intensity
        
Observed effects on intensity. This plot visualizes the effects from previously-run arms on a specific metric, providing insights into their performance. This plot allows one to compare and contrast the effectiveness of different arms, highlighting which configurations have yielded the most favorable outcomes. 

        
    

    

        




    
                            
                    




    








    




    








    

        
        Scatter Plot (Constraints)
        
        

        These plots display the effects of each arm on two metrics displayed on the x- and y-axes. They are useful for understanding the trade-off between the two metrics and for visualizing the Pareto frontier.
        

    

    

        



    



    

        Modeled Effects: fwhm vs. intensity
        
This plot displays the effects of each arm on the two selected metrics. It is useful for understanding the trade-off between the two metrics and for visualizing the Pareto frontier.

        
    

    

        




    
                            
                    




    








    








    



    

        Utility Progression
        
Shows the best fwhm value achieved so far across completed trials (objective is to minimize). The x-axis shows trial index. Only completed or early-stopped trials with complete metric data are included, so there may be gaps if some trials failed, were abandoned, or have incomplete data. The y-axis shows cumulative best utility. Only improvements are plotted, so flat segments indicate trials that didn't surpass the previous best. Infeasible trials (violating outcome constraints) don't contribute to the improvements.

        
    

    

        




    
                            
                    




    








    

        Best Trial for Experiment
        
Displays the trial with the best objective value based on raw observations. This reflects actual measured performance during execution. This trial achieved the optimal objective value and represents the recommended configuration for your optimization goal. Only considering COMPLETED trials.

        
    

    

        

  
    

      
      trial_index
      arm_name
      trial_status
      generation_node
      fwhm
      intensity
      kbv-radius
      kbh-radius
    

  
  
    

      0
      13
      13_0
      COMPLETED
      MBM
      22.621544
      50000.0
      38092.241389
      20864.589956
    

  


    








    

        Summary for xrt-blop-demo
        
High-level summary of the `Trial`-s in this `Experiment`

        
    

    

        

  
    

      
      trial_index
      arm_name
      trial_status
      generation_node
      fwhm
      intensity
      kbv-radius
      kbh-radius
    

  
  
    

      0
      0
      0_0
      COMPLETED
      CenterOfSearchSpace
      139.288699
      47119.0
      35000.000000
      25000.000000
    

    

      1
      1
      1_0
      COMPLETED
      Sobol
      299.971542
      28714.0
      30785.487890
      31834.090948
    

    

      2
      2
      2_0
      COMPLETED
      Sobol
      93.942931
      49798.0
      42970.957160
      19386.426341
    

    

      3
      3
      3_0
      COMPLETED
      Sobol
      128.872795
      35919.0
      36555.225141
      29791.710041
    

    

      4
      4
      4_0
      COMPLETED
      Sobol
      78.343936
      41608.0
      29991.130922
      21409.296561
    

    

      5
      5
      5_0
      COMPLETED
      Sobol
      302.515053
      23606.0
      25742.566008
      26905.404571
    

    

      6
      6
      6_0
      COMPLETED
      Sobol
      78.192314
      48438.0
      37933.317386
      24316.142984
    

    

      7
      7
      7_0
      COMPLETED
      Sobol
      177.505152
      29916.0
      41280.810907
      33930.410799
    

    

      8
      8
      8_0
      COMPLETED
      Sobol
      155.104399
      45349.0
      34721.125867
      17271.625474
    

    

      9
      9
      9_0
      COMPLETED
      Sobol
      232.250840
      38684.0
      33254.698589
      28155.618124
    

    

      10
      10
      10_0
      COMPLETED
      MBM
      149.586029
      45422.0
      41233.596017
      25624.421751
    

    

      11
      11
      11_0
      COMPLETED
      MBM
      230.880303
      39397.0
      29692.492279
      18074.478580
    

    

      12
      12
      12_0
      COMPLETED
      MBM
      90.782024
      49791.0
      35822.911571
      22804.855050
    

    

      13
      13
      13_0
      COMPLETED
      MBM
      22.621544
      50000.0
      38092.241389
      20864.589956
    

    

      14
      14
      14_0
      COMPLETED
      MBM
      29.546612
      50000.0
      39513.712553
      20834.261812
    

    

      15
      15
      15_0
      COMPLETED
      MBM
      153.451719
      48343.0
      45000.000000
      23509.152336
    

    

      16
      16
      16_0
      COMPLETED
      MBM
      52.863832
      49894.0
      38767.774778
      19302.192373
    

    

      17
      17
      17_0
      COMPLETED
      MBM
      31.360815
      49997.0
      38899.438456
      21419.676503
    

    

      18
      18
      18_0
      COMPLETED
      MBM
      111.813889
      29477.0
      26256.671034
      21625.783628
    

    

      19
      19
      19_0
      COMPLETED
      MBM
      246.897415
      30107.0
      45000.000000
      15000.000000
    

  


    








    








    

        
        Insights Analysis
        
        

        Insight Analyses display information to help understand the underlying experiment i.e parameter and metric relationships learned by the Ax model.Use this information to better understand your experiment space and users.
        

    

    

        



    

        
        Outcome Constraints Analysis
        
        

        Understand which trials are likely to meet outcome constraints, and show how outcome constraints are affecting the optimization as a whole.
        

    

    

        



    



    

        Predicted Probability of Feasibility
        
Probability that each arm satisfies all constraints: intensity >= 10000

        
    

    

        




    
                            
                    




    








    

        Modeled  Effect on the Objective vs % Chance of Satisfying the Constraints
        
This plot shows newly generated arms with optimal trade-offs between Ax model-estimated effect on the objective (x-axis) and Ax-model estimated probability of satisfying the constraints (y-axis). This plot is useful for understanding: 1) how tight the constraints are (sometimes the constraints can be configured too conservatively, making it difficult to find an arm that improves the objective(s) while satisfying the constraints), 2) how much headroom there is with the current optimization configuration  (objective(s) and constraints). If arms that are likely feasible (y-axis), do not improve your objective enough, revisiting your optimization config and relaxing the constraints may be helpful. This analysis can be computed adhoc in a notebook environment, and will change with modifications to the optimization config, so you can understand the potential impact of optimization config modifications prior to running another iteration. Get in touch with the Ax developers for pointers on including these arms in a trial or running this via a notebook.

        
    

    

        




    
                            
                    




    








    








    

        
        Top Surfaces Analysis: Parameter sensitivity, slice, and contour plots
        
        

        The top surfaces analysis displays three analyses in one. First, it shows parameter sensitivities, which shows the sensitivity of the metrics in the experiment to the most important parameters. Subsetting to only the most important parameters, it then shows slice plots and contour plots for each metric in the experiment, displaying the relationship between the metric and the most important parameters. 
        

    

    

        



    



    

        Sensitivity Analysis for fwhm
        
Understand how each parameter affects fwhm according to a second-order sensitivity analysis.

        
    

    

        




    
                            
                    




    












    

        
        Slice Plots: Metric effects by parameter value
        
        

        These plots show the relationship between a metric and a parameter. They show the predicted values of the metric on the y-axis as a function of the parameter on the x-axis while keeping all other parameters fixed at their status_quo value (if available), best trial value, or the center of the search space.
        

    

    

        



    



    

        fwhm vs. kbh-radius
        
The slice plot provides a one-dimensional view of predicted outcomes for fwhm as a function of a single parameter, while keeping all other parameters fixed at their best trial value (Arm 17_0). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

        
    

    

        




    
                            
                    




    








    

        fwhm vs. kbv-radius
        
The slice plot provides a one-dimensional view of predicted outcomes for fwhm as a function of a single parameter, while keeping all other parameters fixed at their best trial value (Arm 17_0). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

        
    

    

        




    
                            
                    




    








    








    

        
        Contour Plots: Metric effects by parameter values
        
        

        These plots show the relationship between a metric and two parameters. They show the predicted values of the metric (indicated by color) as a function of the two parameters on the x- and y-axes while keeping all other parameters fixed at their status_quo value (if available), best trial value, or the center of the search space.
        

    

    

        



    



    

        fwhm (Mean) vs. kbv-radius, kbh-radius
        
The contour plot visualizes the predicted outcomes for fwhm across a two-dimensional parameter space, with other parameters held fixed at their best trial value (Arm 17_0). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

        
    

    

        




    
                            
                    




    








    




    








    

        
        Top Surfaces Analysis: Parameter sensitivity, slice, and contour plots
        
        

        The top surfaces analysis displays three analyses in one. First, it shows parameter sensitivities, which shows the sensitivity of the metrics in the experiment to the most important parameters. Subsetting to only the most important parameters, it then shows slice plots and contour plots for each metric in the experiment, displaying the relationship between the metric and the most important parameters. 
        

    

    

        



    



    

        Sensitivity Analysis for intensity
        
Understand how each parameter affects intensity according to a second-order sensitivity analysis.

        
    

    

        




    
                            
                    




    












    

        
        Slice Plots: Metric effects by parameter value
        
        

        These plots show the relationship between a metric and a parameter. They show the predicted values of the metric on the y-axis as a function of the parameter on the x-axis while keeping all other parameters fixed at their status_quo value (if available), best trial value, or the center of the search space.
        

    

    

        



    



    

        intensity vs. kbh-radius
        
The slice plot provides a one-dimensional view of predicted outcomes for intensity as a function of a single parameter, while keeping all other parameters fixed at their best trial value (Arm 17_0). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

        
    

    

        




    
                            
                    




    








    

        intensity vs. kbv-radius
        
The slice plot provides a one-dimensional view of predicted outcomes for intensity as a function of a single parameter, while keeping all other parameters fixed at their best trial value (Arm 17_0). This visualization helps in understanding the sensitivity and impact of changes in the selected parameter on the predicted metric outcomes.

        
    

    

        




    
                            
                    




    








    








    

        
        Contour Plots: Metric effects by parameter values
        
        

        These plots show the relationship between a metric and two parameters. They show the predicted values of the metric (indicated by color) as a function of the two parameters on the x- and y-axes while keeping all other parameters fixed at their status_quo value (if available), best trial value, or the center of the search space.
        

    

    

        



    



    

        intensity (Mean) vs. kbv-radius, kbh-radius
        
The contour plot visualizes the predicted outcomes for intensity across a two-dimensional parameter space, with other parameters held fixed at their best trial value (Arm 17_0). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

        
    

    

        




    
                            
                    




    








    




    




    








    

        
        Diagnostic Analysis
        
        

        Diagnostic Analyses provide information about the optimization process and the quality of the model fit. You can use this information to understand if the experimental design should be adjusted to improve optimization quality.
        

    

    

        



    

        
        Cross Validation: Assessing model fit
        
        

        Cross-validation plots display the model fit for each metric in the experiment. The model is trained on a subset of the data and then predicts the outcome for the remaining subset. The plots show the predicted outcome for the validation set on the y-axis against its actual value on the x-axis. Points that align closely with the dotted diagonal line indicate a strong model fit, signifying accurate predictions. Additionally, the plots include confidence intervals that provide insight into the noise in observations and the uncertainty in model predictions. 

NOTE: A horizontal, flat line of predictions indicates that the model has not picked up on sufficient signal in the data, and instead is just predicting the mean.
        

    

    

        



    



    

        Cross Validation for intensity (R² = 0.98)
        
The cross-validation plot displays the model fit for each metric in the experiment. It employs a leave-one-out approach, where the model is trained on all data except one sample, which is used for validation. The plot shows the predicted outcome for the validation set on the y-axis against its actual value on the x-axis. Points that align closely with the dotted diagonal line indicate a strong model fit, signifying accurate predictions. Additionally, the plot includes 95% confidence intervals that provide insight into the noise in observations and the uncertainty in model predictions. A horizontal, flat line of predictions indicates that the model has not picked up on sufficient signal in the data, and instead is just predicting the mean.

        
    

    

        




    
                            
                    




    








    

        Cross Validation for fwhm (R² = 0.59)
        
The cross-validation plot displays the model fit for each metric in the experiment. It employs a leave-one-out approach, where the model is trained on all data except one sample, which is used for validation. The plot shows the predicted outcome for the validation set on the y-axis against its actual value on the x-axis. Points that align closely with the dotted diagonal line indicate a strong model fit, signifying accurate predictions. Additionally, the plot includes 95% confidence intervals that provide insight into the noise in observations and the uncertainty in model predictions. A horizontal, flat line of predictions indicates that the model has not picked up on sufficient signal in the data, and instead is just predicting the mean.

        
    

    

        




    
                            
                    




    








    

        Summary of model fits
        
R² (coefficient of determination) measures how well the model predicts each metric. Higher values indicate better model fit. Metrics with R² >= 0.1 are highlighted in green.

        
    

    

        




    
                            
                    




    








    








    



    

        Generation Strategy Graph
        
GenerationStrategy: Center+Sobol+MBM:fast

Visualize the structure of a GenerationStrategy as a directed graph. Each node
represents a GenerationNode in the strategy, and edges represent transitions
between nodes based on TransitionCriterion. Edge labels show the criterion
class names that trigger the transition.


        
    

    

        
b'\n\n\n\n\n\nGenerationStrategy\n\n\n\nCenterOfSearchSpace\n\nCenterOfSearchSpace\n()\n\n\n\nSobol\n\nSobol\n\n\n\nCenterOfSearchSpace->Sobol\n\n\nAutoTransitionAfterGen\n\n\n\nMBM\n\nMBM\n(BoTorch)\n\n\n\nSobol->MBM\n\n\nMinTrials(5)\nMinTrials(2)\n\n\n\n'

    








    








    

        
        Health Checks
        
        

        Comprehensive health checks designed to identify potential issues in the Ax experiment. These checks cover areas such as metric fetching, search space configuration, and candidate generation, with the aim of flagging areas where user intervention may be necessary to ensure the experiment's robustness and success.
        

    

    

        



    



    

        Baseline Improvement Healthcheck
        
All 1 objective(s) improved over baseline.

**Metric `fwhm` improved 83.76%** from `139.29` in arm `'0_0'` to `22.62` in arm `'13_0'`.

**Note:** Using the first trial's first arm ('0_0') as the baseline since no explicit baseline was provided.

        
    

    

        

  
    

      
      Metric
      Status
      Details
    

  
  
    

      0
      fwhm
      Improved
      **Metric `fwhm` improved 83.76%** from `139.29` in arm `'0_0'` to `22.62` in arm `'13_0'`.
    

  


    








    

        TransferLearningAnalysis Error
        
ValueError: Must specify either `url` or `creator`.

        
    

    

        
Traceback (most recent call last):
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/analysis/analysis.py", line 115, in compute_result
    card = self.compute(
        experiment=experiment,
        generation_strategy=generation_strategy,
        adapter=adapter,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/analysis/healthcheck/transfer_learning_analysis.py", line 104, in compute
    transferable_experiments = identify_transferable_experiments(
        search_space=experiment.search_space,
    ...<4 lines>...
        experiment_name=experiment.name,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/load.py", line 839, in identify_transferable_experiments
    experiments_search_spaces = _query_historical_experiments_given_parameters(
        parameter_names=list(search_space.parameters.keys()),
        experiment_types=experiment_types,
        config=config,
    )
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/load.py", line 759, in _query_historical_experiments_given_parameters
    with session_scope() as session:
         ~~~~~~~~~~~~~^^
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 287, in session_scope
    session = get_session()
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 263, in get_session
    init_engine_and_session_factory()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/runner/work/blop/blop/.pixi/envs/docs/lib/python3.13/site-packages/ax/storage/sqa_store/db.py", line 191, in init_engine_and_session_factory
    raise ValueError("Must specify either `url` or `creator`.")
ValueError: Must specify either `url` or `creator`.


    








    




    








We can also get a tabular summary of the trials:






agent.ax_client.summarize()













  
    

      
      trial_index
      arm_name
      trial_status
      generation_node
      fwhm
      intensity
      kbv-radius
      kbh-radius
    

  
  
    

      0
      0
      0_0
      COMPLETED
      CenterOfSearchSpace
      139.288699
      47119.0
      35000.000000
      25000.000000
    

    

      1
      1
      1_0
      COMPLETED
      Sobol
      299.971542
      28714.0
      30785.487890
      31834.090948
    

    

      2
      2
      2_0
      COMPLETED
      Sobol
      93.942931
      49798.0
      42970.957160
      19386.426341
    

    

      3
      3
      3_0
      COMPLETED
      Sobol
      128.872795
      35919.0
      36555.225141
      29791.710041
    

    

      4
      4
      4_0
      COMPLETED
      Sobol
      78.343936
      41608.0
      29991.130922
      21409.296561
    

    

      5
      5
      5_0
      COMPLETED
      Sobol
      302.515053
      23606.0
      25742.566008
      26905.404571
    

    

      6
      6
      6_0
      COMPLETED
      Sobol
      78.192314
      48438.0
      37933.317386
      24316.142984
    

    

      7
      7
      7_0
      COMPLETED
      Sobol
      177.505152
      29916.0
      41280.810907
      33930.410799
    

    

      8
      8
      8_0
      COMPLETED
      Sobol
      155.104399
      45349.0
      34721.125867
      17271.625474
    

    

      9
      9
      9_0
      COMPLETED
      Sobol
      232.250840
      38684.0
      33254.698589
      28155.618124
    

    

      10
      10
      10_0
      COMPLETED
      MBM
      149.586029
      45422.0
      41233.596017
      25624.421751
    

    

      11
      11
      11_0
      COMPLETED
      MBM
      230.880303
      39397.0
      29692.492279
      18074.478580
    

    

      12
      12
      12_0
      COMPLETED
      MBM
      90.782024
      49791.0
      35822.911571
      22804.855050
    

    

      13
      13
      13_0
      COMPLETED
      MBM
      22.621544
      50000.0
      38092.241389
      20864.589956
    

    

      14
      14
      14_0
      COMPLETED
      MBM
      29.546612
      50000.0
      39513.712553
      20834.261812
    

    

      15
      15
      15_0
      COMPLETED
      MBM
      153.451719
      48343.0
      45000.000000
      23509.152336
    

    

      16
      16
      16_0
      COMPLETED
      MBM
      52.863832
      49894.0
      38767.774778
      19302.192373
    

    

      17
      17
      17_0
      COMPLETED
      MBM
      31.360815
      49997.0
      38899.438456
      21419.676503
    

    

      18
      18
      18_0
      COMPLETED
      MBM
      111.813889
      29477.0
      26256.671034
      21625.783628
    

    

      19
      19
      19_0
      COMPLETED
      MBM
      246.897415
      30107.0
      45000.000000
      15000.000000
    

  









Visualizing the Surrogate Model#


The plot_objective method shows how the FWHM varies across the DOF space, based on the surrogate model the agent built:






_ = agent.plot_objective(x_dof_name="kbh-radius", y_dof_name="kbv-radius", objective_name="fwhm")













    

        fwhm (Mean) vs. kbh-radius, kbv-radius
        
The contour plot visualizes the predicted outcomes for fwhm across a two-dimensional parameter space, with other parameters held fixed at their best trial value (Arm 17_0). This plot helps in identifying regions of optimal performance and understanding how changes in the selected parameters influence the predicted outcomes. Contour lines represent levels of constant predicted values, providing insights into the gradient and potential optima within the parameter space.

        
    

    

        




    
                            
                    




    









This plot reveals the landscape the optimizer explored. The valley (minimum) shows where the optimal mirror curvatures lie.








Applying the Optimal Configuration#


Let’s retrieve the best configuration found during optimization and apply it to see the resulting beam:






optimal_parameters, metrics, _, _ = agent.ax_client.get_best_parameterization(use_model_predictions=False)
optimal_parameters









{'kbv-radius': 38092.24138939193, 'kbh-radius': 20864.58995647712}









Now move the mirrors to these optimal positions and acquire an image:






from bluesky.plans import list_scan

uid = RE(list_scan(
    [det],
    kbv.radius, [optimal_parameters[kbv.radius.name]],
    kbh.radius, [optimal_parameters[kbh.radius.name]],
))













image = tiled_client[uid[0]]["primary/det_image"].read().squeeze()
plt.imshow(image)
plt.colorbar()
plt.title("Optimized KB Mirror Beam")
plt.show()








../_images/17ce38b1af94ecd694b45d1eacf67899b81b23067c823ddf45100dca38313a07.png










Solving the Same Problem with XoptOptimizer#


Blop also supports Xopt through XoptOptimizer, which plugs directly into the
same optimize plan used throughout the package. This gives you a lower-level
optimization interface while reusing the same devices, acquisition flow, and
evaluation function from above.


For this section, we keep the exact same objective and constraint:


    

  • Minimize fwhm
    • 

  • Require intensity >= 10000
    • 







from xopt.generators.bayesian import ExpectedImprovementGenerator
from xopt.vocs import VOCS

from blop.plans import optimize
from blop.protocols import OptimizationProblem
from blop.xopt.optimizer import XoptOptimizer









Define the Xopt search space (VOCS) from the same mirror bounds and outcome names used earlier:






xopt_vocs = VOCS(
    variables={
        kbv.radius.name: list(VERTICAL_BOUNDS),
        kbh.radius.name: list(HORIZONTAL_BOUNDS),
    },
    objectives={"fwhm": "MINIMIZE"},
    constraints={"intensity": ["GREATER_THAN", 10000.0]},
)

xopt_optimizer = XoptOptimizer(generator=ExpectedImprovementGenerator(vocs=xopt_vocs))

xopt_problem = OptimizationProblem(
    optimizer=xopt_optimizer,
    actuators=[kbv.radius, kbh.radius],
    sensors=[det],
    evaluation_function=DetectorEvaluation(tiled_client),
)









Run an optimization loop. As with the Ax-based flow, this is executed via the Bluesky RunEngine.






RE(optimize(xopt_problem, iterations=10, n_points=1))









('780e02ce-6c66-4529-ab1d-fe350676ed4d',
 'a6a9676d-1cda-4d7c-9b8a-e709c861e3e7',
 '69d9c580-e9c5-4c8f-b1dc-da3f96cf3479',
 'f5c148a6-aff1-45ed-8025-cec49fe7d871',
 '99b76f61-a13e-41da-9497-ce91935b2786',
 'd1174a99-9142-49f1-94ea-551f15145191',
 'bfef6da4-e7c4-4188-af4b-49ffc025f345',
 '9cca556f-1610-4c55-8bce-12c372c097e2',
 '798c5bd0-6040-4c2a-93a9-4edf86a34e00',
 'df7e6647-d42f-481b-956b-19762e5d1aa3',
 '57540610-ccbf-4a8c-a50f-c758b10317a9')









Visualize the GP model learned by the Xopt generator:






fig, _ = xopt_optimizer.generator.visualize_model(
    output_names=["fwhm"],
    variable_names=[kbv.radius.name, kbh.radius.name],
    show_feasibility=True,
)
plt.show()








../_images/f91488aa3fe557faad350eb00ced08cd11ac1377cb683d58cb3759adf0d2aa53.png






Inspect the best point found by Xopt and the collected trial data:






xopt_best_points = xopt_optimizer.get_best_points()
xopt_best_points









[(8,
  {'kbv-radius': np.float64(45000.0),
   'kbh-radius': np.float64(20598.21158289496)},
  {'fwhm': np.float64(55.266219604195), 'intensity': np.float64(48937.0)})]













xopt_optimizer.generator.data.tail()













  
    

      
      _id
      kbv-radius
      kbh-radius
      fwhm
      intensity
    

  
  
    

      5
      5
      25000.00000
      21205.620662
      79.363785
      26142.0
    

    

      6
      6
      27632.68093
      20952.280107
      73.784186
      33554.0
    

    

      7
      7
      25000.00000
      20639.969394
      73.179330
      26250.0
    

    

      8
      8
      45000.00000
      20598.211583
      55.266220
      48937.0
    

    

      9
      9
      39238.78909
      30997.994881
      121.275325
      33719.0
    

  











tiled_server.close()













What You’ve Learned#


In this tutorial, you worked through a complete Bayesian optimization workflow:


    

  1. DOFs define the search space — the parameters you can control and their allowed ranges
    2. 

  2. Objectives specify your optimization goal (here: minimize FWHM for a tight focal spot)
    3. 

  3. Tracking metrics (IMetric) let you monitor values like intensity without optimizing them directly
    4. 

  4. Outcome constraints enforce safety bounds on tracked metrics (e.g., minimum beam intensity)
    5. 

  5. Evaluation functions extract meaningful metrics from experimental data using robust techniques like marginal-profile FWHM
    6. 

  6. The Agent coordinates everything, building a surrogate model of your system and intelligently exploring the parameter space
    7. 

  7. Health checks let you diagnose optimization progress and catch issues early
    8. 



These same components apply to any optimization problem: swap the simulated devices for real hardware, adjust the DOFs and objectives for your system, and write an evaluation function that extracts your metrics.






Next Steps#







See Also#