Develop Conformance Testsuite

[sdasgup3]

This is about a test suite that consumers can plug into and run on their hardware. The golden results of the tests are expected to be vetted by the reference implementation of the specification.

[bhack]

What is supposed to be tested here?
I am asking this just to understand the scope in the perspective with the thread we have at openxla/community#33 (comment)

[bhack]

Any comment on my point?

[sdasgup3]

Hi @bhack
Thanks for sharing the links and sorry for the delay in replying.

The aim of the conformance suite is to comprehensively cover the StableHLO specification. An important milestone towards that are the tests for the reference implementation which exhaustively covers the behavior of individual ops.. Currently we are focusing on the static shaped opset.

In the future, we are planning to also introduce support for dynamic shapes (#8) and "Quantization" category of StableHLO operations (#588 #530 #531) and the test suite should evolve accordingly to accommodate the augmented specification. The exact details are work in progress.

[bhack]

Yes I understand this. I was only a little bit more confused by

consumers can plug into and run on their hardware

Do we suppose to interface with compiler and runtime (and so the hardware) using openxla-pjrt?

As it seems there we expect to interact with not fully (StableHLO program) conformant compilers/runtimes it is going to impact that topic in the openxla-pjrt-plugin API design when interact with frontends (especially if we want to not just fail fast on the fronted side).

It seems to me that there a space to strictly co-design this.

[sdasgup3]

Do we suppose to interface with compiler and runtime (and so the hardware) using openxla-pjrt?

Interfacing using openxla-pjrt could certainly be a favourable option, but needs further exploration.
At the moment the priority of conformance test-suite is reduced in favour of 2023 StableHLO milestones roadmap.md.