Add content assertion XML tags for test output verification using images

[kostrykin]

As part of the Imaging Hackathon in Freiburg. This is a follow-up PR of #17556.

Contributions

This PR adds content assertion XML tags for automated test output verification using 2-D single-channel and multi-channel images. The contributed XML tags are detailed below.

image_has_metadata

The image_has_metadata XML tag can be used for verification of the metadata of the images generated by a Galaxy tool wrapper. Currently supported metadata is:

Attribute	Feature description
width	The width of the image.
height	The height of the image.
channels	Number of channels.

Example:

<test>
  <output name="output">
    <assert_contents>
      <image_has_metadata width="512" height="512" channels="1" />
    </assert_contents>
  </output>
</test>

image_has_intensities

The image_has_intensities XML tag can be used for verification of the image intensities of the images generated by a Galaxy tool wrapper. Verification of the following features is supported:

Attribute	Feature description
mean_intensity	The mean image intensity.
mean_intensity_min	Lower bound of the mean image intensity.
mean_intensity_max	Upper bound of the mean image intensity.
center_of_mass	The center of mass of the image intensities.
eps	Absolute tolerance for mean_intensity and center_of_mass assertions.
channel	Restricts the assertions to only a specific image channel.

The assertions are imposed for all image channels if the channel attribute is omitted.

Example:

<test>
  <output name="output">
    <assert_contents>
      <image_has_intensities channel="0" mean_intensity="0.24" eps="0.01" />
      <image_has_intensities channel="1" mean_intensity="0.66" eps="0.01" />
    </assert_contents>
  </output>
</test>

image_has_labels

The image_has_labels XML tag can be used for verification of label images (label maps) generated by a Galaxy tool wrapper. Verification of the following features is supported:

Attribute	Feature description
number_of_objects	The number of uniquely labeled objects.
mean_object_size	The mean size (number of pixels) of uniquely labeled objects.
mean_object_size_min	Lower bound of the mean size of uniquely labeled objects.
mean_object_size_max	Upper bound of the mean size of uniquely labeled objects.
exclude_labels	Excludes labels from the other assertions.
eps	Absolute tolerance for the mean_object_size assertion.

The primary use case of the exclude_labels attribute is to exclude the image background from the number_of_objects and mean_object_size assertions, which is often labeled by 0 or -1.

Example:

<test>
  <output name="output">
    <assert_contents>
      <image_has_labels number_of_objects="187" exclude_labels="0" />
    </assert_contents>
  </output>
</test>

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[bernt-matthias]

Super cool!

Maybe it would be good to reuse this scheme for asserting numeric values https://github.com/galaxyproject/galaxy/blob/cef07534d54b6e0ad55623534f909f446fb0a1c0/lib/galaxy/tool_util/verify/asserts/size.py#L22C5-L22C19, i.e. use delta, min and max?

[kostrykin]

Super cool!

Maybe it would be good to reuse this scheme for asserting numeric values https://github.com/galaxyproject/galaxy/blob/cef07534d54b6e0ad55623534f909f446fb0a1c0/lib/galaxy/tool_util/verify/asserts/size.py#L22C5-L22C19, i.e. use delta, min and max?

My intention in this PR was to use the same attribute names for image verification as in #17556. There, the absolute tolerance for verification is named eps. The issue that I see with using the delta attribute in #17556 is that it has a default value of 10000, which is rather inappropriate for image verification. This is why I used a dedicated attribute in #17556, for which I could use a meaningful default value (0.01), and, for consistency, also in this PR. What do you think? @bernt-matthias

[kostrykin]

Thanks for your suggestion regarding the lower/upper bounds scheme. I have added *_min and *_max attributes for the mean_intensity and mean_object_size assertions in 5ce6045.

The center_of_mass assertion is about coordinates, and using lower/upper bounds for coordinates is somewhat inconvenient, I think¹. The other assertions (width, height, channels, number_of_objects) concern integer values, for which rather precise values are preferable for the checks, I think.

The _assert_number function cannot be used because it requires n to be int or str, but does not support float (see here). I have doubts whether we should extend this to float, because it contains some logic on parsing sizes (parse_bytesize), and mixing float logics into this function seems misplaced.

Footnotes

1. Axis-wise bounds would induce a rectangular region of validity, but points closer the corner of the rectangle are more erroneous. This issue does not arise when using circular regions of validity, which the tolerance-based scheme is about (where eps is the radius). ↩

[bernt-matthias]

Sounds reasonable to me.

has a default value of 10000, which is rather inappropriate for image verification

We could have a separate default for images.

But anyway I'm happy with it.

[kostrykin]

Thanks for your comments, @bernt-matthias.

In the course of splitting the combined <image_has_*> assertions into individual ones, I have also renamed eps to delta, as you originally suggested (otherwise, I believe it would have been too confusing, why there are the min and max attributes, but delta is called eps). For consistency, we should also rename eps from #17556 to delta now.

You mentioned before, that

We could have a separate default for images.

can you please elaborate how this is possible? See the XSD snippet below.

Caution

We would also need to change to the type of delta to xs:float for images! If this isn't possible, I'd prefer to change delta back to eps in this PR for those assertions, which require a float-typed tolerance:

• <has_image_mean_intensity>
• <has_image_center_of_mass>
• <assert_has_image_mean_object_size>

Then we would have delta for int-typed tolerances, and eps for float-typed.

This is the definition of the default in galaxy.xsd:

<xs:attribute name="delta" type="xs:integer" default="10000">
  ...
</xs:attribute>

---

Edit: After having thought about this for a while, I think that renaming delta back to eps for float-typed assertions is the best option. This establishes consistency with both #17556 and the previous schemes used for assertions. If you agree, I can merge kostrykin#1 into this PR. This establishes the attribute delta for integer-typed assertions (which is consistent with previous assertions), and the attribute eps for float-typed assertions (which is consistent with #17556).

[kostrykin]

For some reason the TestGalaxyOIDCLoginIntegration::test_oidc_login_existing_user test is failing

[nsoranzo]

For some reason the TestGalaxyOIDCLoginIntegration::test_oidc_login_existing_user test is failing

Unrelated to your PR, will be fixed by #17612 .

[bernt-matthias]

Thanks so much. Really cool. This will be really helpful.