conformance testing of modern SHACL implementations

[VladimirAlexiev]

(Related to #78)

The official Implementation Report https://w3c.github.io/data-shapes/data-shapes-test-suite/ is 5 years old and badly obsolete.

the original impl report was obviously only created to pass the W3C process for SHACL 1.0.

When selecting a SHACL implementation (or when writing shapes), one of the most important considerations is which SHACL features are supported.

• Some implementations list the features, eg https://graphdb.ontotext.com/documentation/10.7/shacl-validation.html#supported-shacl-features
• but it's better to have a conformance test report

Running such tests is up to SHACL implementors.

• tests are defined in https://github.com/w3c/data-shapes/tree/gh-pages/data-shapes-test-suite/tests using the W3C Test Manifest ontology
• But it doesn't help that there's no test runner
• Test reports are processed with https://github.com/w3c/data-shapes/blob/gh-pages/data-shapes-test-suite/reports/earlreport.ui.ttlx, which uses the SPIN Web Pages (SWP) engine in TBC-ME.

Notes:

• JSON-LD tests https://github.com/w3c/json-ld-api/tree/main/tests are processed using Ruby and HAML https://github.com/gkellogg/earl-report. This pertains to
  • manifests (eg https://rawgit2.com/w3c/json-ld-api/main/tests/manifest.html, https://rawgit2.com/w3c/json-ld-api/main/tests/compact-manifest.html and many others)
  • EARL results https://w3c.github.io/json-ld-api/reports/
• Various implementations define their own tests, eg
  • https://github.com/eclipse-rdf4j/rdf4j/tree/main/core/sail/shacl/src/test/resources, https://github.com/eclipse-rdf4j/rdf4j/tree/main/core/sail/shacl/src/test/resources/test-cases
  • https://github.com/apache/jena/tree/main/jena-shacl/src/test/files
  • https://github.com/TopQuadrant/shacl/tree/master/src/test/resources/sh/tests

[afs]

There are two different things here:

To become a REC, there must be an implementation with each feature having at least two implementations. This is frozen at the time of REC. It is showing feasibility. Normally this is via a WG endorsed test suite. From last time, that is https://w3c.github.io/data-shapes/data-shapes-test-suite/.

I don't know why it changed in January this year. It is linked to from https://www.w3.org/TR/shacl/ and as such should be frozen.

An on-going report aimed at users choosing an implementation is different to the formal test suite. It is a community issue.

For test maintenance, there is already https://github.com/w3c/rdf-tests . Sharing that for maintenance beyond the the lifetime of a working group would be good.

But it doesn't help that there's no test runner
There are runners - probably one per implementation being tested across various languages.

Various implementations define their own tests

https://github.com/apache/jena/tree/main/jena-shacl/src/test/files is a copy of the W3C tests included under license. It is included in every Jena release. Relying on the web to download material in a test suite is "unreliable" 😄. There are some addition tests but the bulk is the W3C test suite.

[VladimirAlexiev]

I understand that testing an implementation is the job of the implementor or the community, not the WG. But I'm posting this here as a focal point to gather efforts and links/ resources.

For test maintenance, there is already https://github.com/w3c/rdf-tests .

Can you elaborate? Is there some tooling in the RDF tests that can be reused for the SHACL tests?

Various implementations define their own tests

I mean in addition to the standard tests: so these can be used to enlarge the standard test suite.

[afs]

For test maintenance, there is already https://github.com/w3c/rdf-tests .

Can you elaborate?

It is a way for the test suite to continue to be kept up-to-date, errata applied etc etc. That needs governance and community.

rdf-tests provides up-to-date versions of RDF and SPARQL tests using the common test manifest (the SHACL formats is a little difference but based on the same manifest idea and vocabulary).

Is there some tooling in the RDF tests that can be reused for the SHACL tests?

The WG already provides a test format which a toolkit specific test runner can take in and execute the tests. Every implementation in the report will have done that.

I think you are asking for something else - a single/portable tool for users, not implementers, to evaluate implementations. It would
run tests against any choice of implementation. That requires a common way to invoke "data+shapes" to get a conformance report and to check the conformance report against the expected outcome, then report. It needs a common invocation mechanism (which does not have to be efficient e.g. invoke a process per test) and even then, what about implementation that does not provide a CLI command? That does not sound like WG work within the current charter.

[VladimirAlexiev]

That requires a common way to invoke "data+shapes" to get a conformance report

Yes, but that's one small piece of the complete test runner.

what about implementation that does not provide a CLI command?

I can add a CLI to a SHACL validator, and small invocation scripts per validator.

But if there was a core test runner, it would greatly help the overall task.
I've seen runners implemented:

• in Perl by @tobyink for r2rml testing
• in Ruby by @gkellogg for (I think) JSON-LD testing

[gkellogg]

I have a SHACL test runner as part of my SHACL implementation as well. This generally uses the same mechanism as the JSON-LD test runner by parsing the RDF containing test manifests into JSON-LD and then iterating over each manifest entry in the test runner. I have previously adapted a generic test runner to be able to run against an arbitrary implementation which requires that an implementation be easily installed and available through a CLI with some common options for specifying test locations and options.

Note that the RDFa working group tried to maintain an HTML test runner that would reach out to various endpoints defined by implementations to run the test suite and generate EARL reports. This proved to be very difficult to keep running properly, so my preference is for implementations to continue to be responsible for running tests on their own and send in implementation reports.

[tpluscode]

Note that the RDFa working group tried to maintain an HTML test runner that would reach out to various endpoints defined by implementations to run the test suite and generate EARL reports. This proved to be very difficult to keep running properly, so my preference is for implementations to continue to be responsible for running tests on their own and send in implementation reports.

I once proposed in rml WG to have a repo which would fetch reports from implementors. That never took off and I don't know what they are presently doing but I could rehash that idea for SHACL. The process is that a central repo+generated website exists where implementors submit PRs with links to their published reports.

We can provide an example GitHub workflow which would run your runner and then publish the report on GH Pages.