Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add a doc for running conformance tests from other repos #855

Merged
merged 4 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ It allows users to quickly spin up custom testnets and dev environments to test
- **Use Cases:**
- **Importing as a Module**
- [Chain Integration and GitHub CI](./docs/ciTests.md)
- [Running Conformance Tests](./docs/conformance-tests-lib.md)
- **Running as a Binary**
- [Building Binary](./docs/buildBinary.md)
- [Running Conformance Tests](./docs/conformanceTests.md) - Suite of built-in tests that test high-level IBC compatibility
- [Running Conformance Tests](./docs/conformance-tests-bin.md) - Suite of built-in tests that test high-level IBC compatibility
- [Write Custom Tests](./docs/writeCustomTests.md)
- [Environment Variable Options](./docs/envOptions.md)
- [Retaining Data on Failed Tests](./docs/retainingDataOnFailedTests.md)
Expand Down
File renamed without changes.
106 changes: 106 additions & 0 deletions docs/conformance-tests-lib.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Conformance Tests

`interchaintest` comes with a suite of conformance tests. These tests ensure IBC and relayer compatibility. On a high-level it tests:
- `client`, `channel`, and `connection` creation
- messages are properly relayed and acknowledged
- packets are being properly timed out

You can view all the specific conformance test by reviewing them in the [conformance](../conformance/) folder.

## Importing Conformance Tests In Your Project

`interchaintest` can be imported into your own packages to be used as a library as well as being used from the
binary itself, see [here](conformance-tests-bin.md).

A common pattern when importing `interchaintest` into your own repositories is to use a Go submodule. The reason being
is that there are some replace directives being used in `interchaintest` which may force your main modules `go.mod`
to use specific versions of dependencies. To avoid this issue one will typically create a new package, such as
`interchaintest` or `e2e`, then you can initialize a new Go module via the `go mod init` command.

## Writing Go Tests For Conformance Testing

The main entrypoint exposed by the `conformance` package is a function named `Test`.

Here is the function signature of `Test`:
```go
func Test(t *testing.T, ctx context.Context, cfs []interchaintest.ChainFactory, rfs []interchaintest.RelayerFactory, rep *testreporter.Reporter)
```

It accepts a normal `testing.T` and `context.Context` from the Go standard library as well as a few types defined in `interchaintest`.

- `testreporter.Reporter` is used for collecting detailed test reports, you can read more about it [here](../testreporter/doc.go).
- `interchaintest.ChainFactory` is used to define which chain pairs should be used in conformance testing.
- `interchaintest.RelayerFactory` is used to define which relayer implementations should be used to test IBC functionality between your chain pairs.

It is important to note that the `Test` function accepts a slice of `ChainFactory`, currently the `conformance` tests only work against
a pair of two chains at a time. This means that each `ChainFactory` should only contain definitions for two chains,
which you will define via the `ChainSpec` type. So if you need to run the `conformance` tests against several different chains
you will need to instantiate several instances of `ChainFactory`.

For our example we will run the `conformance` tests against two different instances of the [Cosmos Hub (Gaia)](https://github.com/cosmos/gaia).

```go

package conformance_test

import (
"context"
"testing"

"github.com/strangelove-ventures/interchaintest/v8"
"github.com/strangelove-ventures/interchaintest/v8/conformance"
"github.com/strangelove-ventures/interchaintest/v8/ibc"
"github.com/strangelove-ventures/interchaintest/v8/testreporter"
"go.uber.org/zap/zaptest"
)

func TestConformance(t *testing.T) {
numOfValidators := 2 // Defines how many validators should be used in each network.
numOfFullNodes := 0 // Defines how many additional full nodes should be used in each network.

// Here we define our ChainFactory by instantiating a new instance of the BuiltinChainFactory exposed in interchaintest.
// We use the ChainSpec type to fully describe which chains we want to use in our tests.
cf := interchaintest.NewBuiltinChainFactory(zaptest.NewLogger(t), []*interchaintest.ChainSpec{
{
Name: "gaia",
ChainName: "cosmoshub-1",
Version: "v13.0.1",
NumValidators: &numOfValidators,
NumFullNodes: &numOfFullNodes,
},
{
Name: "gaia",
ChainName: "cosmoshub-2",
Version: "v13.0.1",
NumValidators: &numOfValidators,
NumFullNodes: &numOfFullNodes,
},
})

// Here we define our RelayerFactory by instantiating a new instance of the BuiltinRelayerFactory exposed in interchaintest.
// We will instantiate two instances, one for the Go relayer and one for Hermes.
rlyFactory := interchaintest.NewBuiltinRelayerFactory(
ibc.CosmosRly,
zaptest.NewLogger(t),
)

hermesFactory := interchaintest.NewBuiltinRelayerFactory(
ibc.Hermes,
zaptest.NewLogger(t),
)

// conformance.Test requires a Go context
ctx := context.Background()

// For our example we will use a No-op reporter that does not actually collect any test reports.
rep := testreporter.NewNopReporter()

// Test will now run the conformance test suite against both of our chains, ensuring that they both have basic
// IBC capabilities properly implemented and work with both the Go relayer and Hermes.
conformance.Test(t, ctx, []interchaintest.ChainFactory{cf}, []interchaintest.RelayerFactory{rlyFactory, hermesFactory}, rep)
}


```


Loading