Testing schedule, mx-bluesky v1.4.3

[rtuck99]

Testing schedule:

GDA changes:

• Screen and collect screens but subsequent collections are not made #703 - screen and collect changes https://gerrit.diamond.ac.uk/c/gda/gda-mx/+/43072
  DONE

Deployment:

• Deployment script should delete tmp #271 - deployment script should delete ./tmp - N/A deployment fails before tests are run due to RH7 control server confluent-kafka build failure
• Deploy script no longer gives console output when running dls_dev_env.sh #242
  • deployment script should give console output when running dls_dev_env.sh - DONE
  • deploy_hyperion_to_k8s.sh should now work - FAIL
  • the container image starts - FAIL
  • see Deployment issues discovered during mx-bluesky 1.4.3 testing #764

Robot load:

• Hyperion should check the beamstop is in position before collection #698 - Hyperion should check the beamstop is in position, robot load after a robot error/puck load with beamstop out should produce an error

Gridscans:

• Deploy script no longer gives console output when running dls_dev_env.sh #242 - non-diffracting samples should now return no hits and be skipped
  - chi is always applied when specified as 0 after previously being specified as non-zero
• GPU processing not working with gda.mx.hyperion.xrc.compare_cpu_and_gpu=true #738 - GPU processing works with gda.mx.hyperion.xrc.compare_cpu_and_gpu=true in domain.properties - FIXED
• Document grid scan coordinate systems #221 - Crystals with x-coordinates near 0 on the end of the pin should not cause {grid_position} is outside the bounds of the grid error messages - FAIL
  • mx-bluesky fails with "outside the bounds of the grid" #759

Rotations:

• Ensure aperture selection based on bounding box takes into account step size #618 - Aperture selection based on bounding box size - test large >= 50um and small crystals

• confirmed for large crystals.

• Rotation start omega uses the wrong sign #247 - Test with omega_flip = True feature flag, rotation_direction Positive in request. This should produce identical results to GDA - FIXED

Single collect:

• Supervised test of skipping bad samples #687 - test samples with bad pin/no diffraction, single collection
• Supervised test of skipping bad samples #687 - test samples with bad pin/no diffraction, multi-rotation
  Screen and collect:
• Screen and collect screens but subsequent collections are not made #703 - screen and collect should actually collect as well as screen
• Supervised test of skipping bad samples #687 - test samples with bad pin/no diffraction, S+C

Acceptance tests:

Neil to soak test energy changes, screen + collect

[rtuck99]

deploy_hyperion_to_k8s.sh should now work

This now fails because deploy_hyperion_to_k8s.sh prints True when passed --print-release-dir

pip install setuptools_scm outputs strings that end up in the version

[rtuck99]

kubernetes container image fails to start with

(.venv) [yqf46943@i03-ws002 hyperion-k8s]$ cat start_log.log 
Traceback (most recent call last):
  File "/usr/local/bin/hyperion", line 5, in <module>
    from mx_bluesky.hyperion.__main__ import main
  File "/app/hyperion/src/mx_bluesky/hyperion/__main__.py", line 36, in <module>
    from mx_bluesky.hyperion.experiment_plans.experiment_registry import (
  File "/app/hyperion/src/mx_bluesky/hyperion/experiment_plans/__init__.py", line 6, in <module>
    from mx_bluesky.hyperion.experiment_plans.flyscan_xray_centre_plan import (
  File "/app/hyperion/src/mx_bluesky/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 50, in <module>
    from mx_bluesky.common.external_interaction.callbacks.xray_centre.ispyb_callback import (
  File "/app/hyperion/src/mx_bluesky/common/external_interaction/callbacks/xray_centre/ispyb_callback.py", line 25, in <module>
    from mx_bluesky.common.external_interaction.callbacks.xray_centre.ispyb_mapping import (
  File "/app/hyperion/src/mx_bluesky/common/external_interaction/callbacks/xray_centre/ispyb_mapping.py", line 5, in <module>
    from dodal.devices.oav import utils as oav_utils
  File "/app/dodal/src/dodal/devices/oav/utils.py", line 13, in <module>
    from dodal.devices.oav.pin_image_recognition import PinTipDetection
  File "/app/dodal/src/dodal/devices/oav/pin_image_recognition/__init__.py", line 16, in <module>
    from dodal.devices.oav.pin_image_recognition.utils import (
  File "/app/dodal/src/dodal/devices/oav/pin_image_recognition/utils.py", line 6, in <module>
    import cv2
  File "/usr/local/lib/python3.11/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/usr/local/lib/python3.11/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

This is probably due to a dependency libGL from cv2

[rtuck99]

box_size_um extra parameter needs removing. Or possibly adding back in to the model, if we actually want it.

• box_size_um cannot be passed from GDA in hyperion requests #762

[rtuck99]

If the previous and current chi are both zero, and the sample is not changed, then doing_chi_change is False and no gridscan is done, which prevents load_centre_collect from doing anything because we then do not find a crystal.

What this should do is return a sentinel-value crystal event that represents current coordinates so that the rest of load_centre_collect knows it can do the rotation without moving.

[rtuck99]

When tested with simulated crystal data with x-coordinate of 0.5, test fails with

<...>
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 358, in run_wrapper
    yield from contingency_wrapper(plan, except_plan=except_plan, else_plan=close_run)
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 604, in contingency_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/src/mx_bluesky/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 252, in run_gridscan_and_fetch_results
    flyscan_results = [
                      ^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/src/mx_bluesky/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 253, in <listcomp>
    _xrc_result_in_boxes_to_result_in_mm(xr, parameters)
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/src/mx_bluesky/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 288, in _xrc_result_in_boxes_to_result_in_mm
    fgs_params.grid_position_to_motor_position(
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/dodal/src/dodal/devices/fast_grid_scan.py", line 113, in grid_position_to_motor_position
    raise IndexError(f"{grid_position} is outside the bounds of the grid")
IndexError: [-0.5  6.5  5.5] is outside the bounds of the grid

This is due to the bounding box also getting transformed in _xrc_result_in_boxes_to_result_in_mm(), and failing because the transformed left edge of the bounding box becomes negative

[rtuck99]

Spurious exceptions from callbacks:

[31m[E 250123 14:23:52.621 plan_reactive_callback:53] �[0m
Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/src/mx_bluesky/common/external_interaction/callbacks/common/plan_reactive_callback.py", line 51, in _run_activity_gated
    return self.emit(name, func(doc))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/.venv/lib/python3.11/site-packages/event_model/__init__.py", line 169, in emit
    emit(name, doc)
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/.venv/lib/python3.11/site-packages/event_model/__init__.py", line 191, in __call__
    return self._dispatch(name, doc, validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/.venv/lib/python3.11/site-packages/event_model/__init__.py", line 199, in _dispatch
    output_doc = getattr(self, name)(doc)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.4.3/mx-bluesky/src/mx_bluesky/common/external_interaction/callbacks/common/zocalo_callback.py", line 85, in event
    assert self.zocalo_interactor is not None

[DominicOram]

Spurious exceptions from callbacks:...

Ok, I think the issue here is:

1. rotation_scan_core in _multi_rotation_scan emits it's start document. This contains information for setting up the nexus file, the ispyb entry and zocalo (the callbacks are called in this order)
2. The nexus callback gets called
3. In the mean time the plan continues in setting up the rotation scan
4. The ispyb callback gets called
5. And starts writing to ispyb
6. rotation_scan_plan emits it's start document
7. The zocalo callback never got it's first start document, this means it was never configured and so throws an exception