diff --git a/README.md b/README.md index 5a73a1fee4..73ec368660 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,32 @@ -**Move Specification Tester** and **Move Mutator** are tools to test the quality of the Move specifications. - ## Overview -The Move mutator is a tool that mutates Move source code. -It can be used to help test the robustness of Move specifications and tests by generating different code versions (mutants). +The **Move Mutator** is a tool that mutates Move source code. +It can help test the robustness of Move specifications and tests by generating different code versions (mutants). +The tool mutates only the source code. Unit tests and specification blocks are not affected by the tool. + +The **Move Specification Tester** is a tool used to test the quality of the Move specifications. +How it works: +1. Runs the _Move Prover_ on the original source code to ensure the original specification is valid. +2. Internally runs the _Move Mutator_ tool to generate mutants. +3. Runs the _Move Prover_ tool on all mutants to check if the mutants are killed (so _Prover_ will catch an error) by the original specifications. + +If some mutants are not killed, it means that the specification has issues and is incorrect or not tight enough to catch such cases, so it should be improved. -The Move Specification Test tool uses the Move Mutator tool to generate mutants of the Move code. -Then, it runs the Move Prover tool to check if the mutants are killed (so Prover will catch an error) by the original specifications. -If the mutants are not killed, it means that the specification has issues and is incorrect or not tight enough to catch such cases, so it should be improved. +The **Move Mutation Tester** is a tool used to test the quality of the test suite and the source code. +How it works: +1. Runs tests on the original source code to ensure the original tests are valid. +2. Internally runs the _Move Mutator_ tool to generate mutants. +3. Runs the tests for all mutants to check if the mutants are killed by the original test suite. + +If the mutants are not killed, it might indicate the quality of the test suite could be improved, or in some rare cases, it might indicate an error in the original source code. ## Install -To build the tools run: +To build the tools, run: ```bash $ cargo install --git https://github.com/eigerco/move-spec-testing.git move-mutator $ cargo install --git https://github.com/eigerco/move-spec-testing.git move-spec-test +$ cargo install --git https://github.com/eigerco/move-spec-testing.git move-mutation-test ``` That will install the tools into `~/.cargo/bin` directory (at least on MacOS and Linux). @@ -23,23 +35,36 @@ Ensure to have this path in your `PATH` environment. This step can be done with $ export PATH=~/.cargo/bin:$PATH ``` -To uninstall tools run: +To uninstall tools, run: ```bash $ cargo uninstall move-mutator $ cargo uninstall move-spec-test +$ cargo uninstall move-mutation-test ``` ## Usage -To check how Move Specification Test tool works, it must be run over the Move code. Some examples are provided [here](https://github.com/eigerco/move-spec-testing/tree/main/move-mutator/tests/move-assets). +The basic tool overview is shown in the below chapters. To dive more deeply into each tool, please check out the documentation here: + + - [`move-mutator` documentation](move-mutator/README.md) + - [`move-spec-test` documentation](move-spec-test/README.md) + - [`move-mutation-test` documentation](move-mutation-test/README.md) -To start specification testing run the following command (assuming that is downloaded from the provided link): +All tools respect the `RUST_LOG` variable, and it will print out as much information as the variable allows. +There is possibility to enable logging only for the specific modules. +Please refer to the [env_logger](https://docs.rs/env_logger/latest/env_logger/) documentation for more details. + +_To run these tools, example projects shall be used that are provided [here](https://github.com/eigerco/move-spec-testing/tree/main/move-mutator/tests/move-assets)._ + +#### Move Specification Test + +To start specification testing, run the following command from the repo directory: ```bash $ move-spec-test -p move-mutator/tests/move-assets/same_names ``` -There should be output generated similar to the following (there may also be -some additional Prover logs visible): +The generated output should be very similar to the following (there may also be +some additional _Prover_ logs visible): ```text Total mutants tested: 4 Total mutants killed: 4 @@ -57,16 +82,13 @@ Total mutants killed: 4 ╰────────────────────────────────────────────────┴────────────────┴────────────────┴────────────╯ ``` -The specification testing tool respects the `RUST_LOG` variable, and it will print out as much information as the variable allows. -There is possibility to enable logging only for the specific modules. -Please refer to the [env_logger](https://docs.rs/env_logger/latest/env_logger/) documentation for more details. To generate a report in a JSON format, use the `-o` option: ```bash $ move-spec-test -p move-mutator/tests/move-assets/poor_spec -o report.json ``` -The sample report generated for the above test will look as follows: +The sample report generated for the above command will look as follows: ```json { "files": { @@ -87,26 +109,85 @@ The sample report generated for the above test will look as follows: } ``` -To check possible options run: +To check possible options, run: ```bash $ move-spec-test --help ``` -Move Specification tool runs Move Mutator internally, however there is a possibility to run it manually. To check possible options run: +#### Move Mutation Tester + +To start mutation test, run the following command from the repo directory: ```bash -$ move-mutator --help +$ move-mutation-test --package-dir move-mutator/tests/move-assets/simple -o report.txt +``` +A shortened sample output for the above command will look as follows: +```text +Total mutants tested: 229 +Total mutants killed: 203 + +╭────────────────────────────────────────────────┬────────────────┬────────────────┬────────────╮ +│ Module │ Mutants tested │ Mutants killed │ Percentage │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::and │ 3 │ 2 │ 66.67% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::div │ 5 │ 5 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::eq │ 5 │ 5 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::gt │ 6 │ 6 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/StillSimple.move::StillSimple::sample6 │ 30 │ 30 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Sum.move::Sum::sum │ 4 │ 4 │ 100.00% │ +╰────────────────────────────────────────────────┴────────────────┴────────────────┴────────────╯ +``` + +The sample `report.txt` generated for the above command will look as follows: +```json +{ + "files": { + "sources/Operators.move": [ + { + "module_func": "Operators::and", + "tested": 3, + "killed": 2, + "mutants_alive_diffs": [ + "--- original\n+++ modified\n@@ -108,7 +108,7 @@\n }\n\n fun and(x: u64, y: u64): u64 {\n- x & y\n+ y&x\n }\n\n // Info: we won't kill a mutant that swaps places (false-positive)\n" + ] + }, + { + "module_func": "Operators::div", + "tested": 5, + "killed": 5, + "mutants_alive_diffs": [] + }, + [...] +``` + +To explore other tool options, run: +```bash +$ move-mutation-test --help ``` -By default, the output shall be stored in the `mutants_output` directory unless otherwise specified. The mutator tool also respects the `RUST_LOG` variable. +#### Move Mutator + +_Move Specification tool_ runs **Move Mutator** internally, however there is a possibility to run it manually. -To generate mutants for all files within a test project (for the whole Move package) run: +To generate mutants for all files within a test project (for the whole Move package), run: ```bash $ move-mutator -p move-mutator/tests/move-assets/simple/ ``` +By default, the output shall be stored in the `mutants_output` directory unless otherwise specified. +To check possible options, run: +```bash +$ move-mutator --help +``` + + ## License -**Move Specification Tester** and **Move Mutator** are released under the open source [Apache License](LICENSE) +All tools in this repo are released under the open source [Apache License](LICENSE) ## About [Eiger](https://www.eiger.co) diff --git a/move-mutation-test/README.md b/move-mutation-test/README.md index 54a17f9ee4..73baa63a80 100644 --- a/move-mutation-test/README.md +++ b/move-mutation-test/README.md @@ -1,3 +1,151 @@ -# Move Mutation Test tool +# Move Mutation Tester tool -TODO: add docs :) +## Summary + +The tool is used to test the quality of the test suite and the source code. + +## Overview + +The program logic is quite simple, the tool works using the following principles: +1. Runs tests on the original source code to ensure the original tests are valid. +2. Internally runs the _Move Mutator_ tool to generate mutants. +3. Runs the tests for all mutants to check if the mutants are killed by the original test suite. + +If the mutants are not killed, it might indicate the quality of the test suite could be improved, or in some rare cases, it might indicate an error in the original source code. + +**Move Mutation Tester** tool can be used on Move packages (projects) which can compile successfully and have valid tests that are passing. +Using filters, it is possible to run the tool only on certain mutants filtered by: + - Module name (`--mutate-modules` argument) + - Function name (`--mutate-functions` argument) + +The tool cannot be used with single Move files since, to run tests, the whole Move project structure with the manifest file is required. + +The tool generates a report in a JSON format. The report contains information +about the number of mutants tested and killed and also the differences between +the original and modified code. + +## Setup check + +Build the whole repository first: +```bash +cargo build --release +``` + +Check if the tool is working properly by running its tests: +```bash +cargo test -p move-mutation-test +``` + +## Usage + +To start the mutation test, run the following command from the repo directory: +```bash +./target/release/move-mutation-test --package-dir move-mutator/tests/move-assets/simple -o report.txt +``` +The above command will store the report in a file `report.txt`. +A shortened sample output for the above command will look as follows: +```text +Total mutants tested: 229 +Total mutants killed: 203 + +╭────────────────────────────────────────────────┬────────────────┬────────────────┬────────────╮ +│ Module │ Mutants tested │ Mutants killed │ Percentage │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::and │ 3 │ 2 │ 66.67% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::div │ 5 │ 5 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::eq │ 5 │ 5 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Operators.move::Operators::gt │ 6 │ 6 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/StillSimple.move::StillSimple::sample6 │ 30 │ 30 │ 100.00% │ +├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤ +│ sources/Sum.move::Sum::sum │ 4 │ 4 │ 100.00% │ +╰────────────────────────────────────────────────┴────────────────┴────────────────┴────────────╯ +``` + +You should see different results for different modules as it depends on the +quality of the source code and the test suites. Some modules, like `Sum`, have good +tests and all mutants are killed, while some others, like `Operators` +may not and some mutants remain alive. + +It's recommended to generate a report in a JSON format and analyze it to see +which mutants are not killed, and what the differences are between the original +and modified code. This can help improve the test suite, or it may indicate +an error in the original source code. + +The sample `report.txt` generated for the above command will look as follows: +```json +{ + "files": { + "sources/Operators.move": [ + { + "module_func": "Operators::and", + "tested": 3, + "killed": 2, + "mutants_alive_diffs": [ + "--- original\n+++ modified\n@@ -108,7 +108,7 @@\n }\n\n fun and(x: u64, y: u64): u64 {\n- x & y\n+ y&x\n }\n\n // Info: we won't kill a mutant that swaps places (false-positive)\n" + ] + }, + { + "module_func": "Operators::div", + "tested": 5, + "killed": 5, + "mutants_alive_diffs": [] + }, + [...] +``` + +The tool respects `RUST_LOG` variable, and it will print out as much information as the variable allows. +There is possibility to enable logging only for the specific modules. +Please refer to the [env_logger](https://docs.rs/env_logger/latest/env_logger/) documentation for more details. + +You can try to run the tool using other examples from the `move-mutator` tests like: +```bash +./target/release/move-mutation-test -p move-mutator/tests/move-assets/breakcontinue +``` + +## Command-line options + +To check possible options, run: +```bash +./target/release/move-mutation-test --help +The configuration options for running the tests + +Usage: move-mutation-test [OPTIONS] + +Options: + --include-modules Work only over specified modules [default: all] + --mutator-conf Optional configuration file for mutator tool + -o, --output Save report to a JSON file + -u, --use-generated-mutants Use previously generated mutants + --package-dir Path to a move package (the folder with a Move.toml file). Defaults to current directory + --output-dir Path to save the compiled move package + --named-addresses Named addresses for the move binary [default: ] + --override-std Override the standard library version by mainnet/testnet/devnet [possible values: mainnet, testnet, devnet] + --skip-fetch-latest-git-deps Skip pulling the latest git dependencies + --skip-attribute-checks Do not complain about unknown attributes in Move code + --dev Enables dev mode, which uses all dev-addresses and dev-dependencies + --check-test-code Do apply extended checks for Aptos (e.g. `#[view]` attribute) also on test code. NOTE: this behavior will become the default in the future. See [env: APTOS_CHECK_TEST_CODE=] + --optimize Select optimization level. Choices are "none", "default", or "extra". Level "extra" may spend more time on expensive optimizations in the future. Level "none" does no optimizations, possibly leading to use of too many runtime resources. Level "default" is the recommended level, and the default if not + provided + --bytecode-version ...or --bytecode BYTECODE_VERSION + Specify the version of the bytecode the compiler is going to emit. + Defaults to `6`, or `7` if language version 2 is selected + (through `--move-2` or `--language_version=2`), . + --compiler-version ...or --compiler COMPILER_VERSION + Specify the version of the compiler. + Defaults to `1`, or `2` if `--move-2` is selected. + --language-version ...or --language LANGUAGE_VERSION + Specify the language version to be supported. + Currently, defaults to `1`, unless `--move-2` is selected. + --move-2 Select bytecode, language version, and compiler to support Move 2: + Same as `--bytecode_version=7 --language_version=2.0 --compiler_version=2.0` + --dump Dump storage state on failure + -f, --filter A filter string to determine which unit tests to run + --ignore-compile-warnings A boolean value to skip warnings + -h, --help Print help (see more with '--help') +``` + +[aptos-core]: https://github.com/aptos-labs/aptos-core/ diff --git a/move-mutator/README.md b/move-mutator/README.md index ee90c0d6c5..fdac7ba43d 100644 --- a/move-mutator/README.md +++ b/move-mutator/README.md @@ -2,13 +2,13 @@ ## Summary -The `move-mutator` tool is a mutation testing tool for the Move language. +The `move-mutator` tool is a tool that is used to mutate the Move language. ## Overview -The Move mutator is a tool that mutates Move source code. It can be used to -help test the robustness of Move specifications and tests by generating -different code versions (mutants). +The Move mutator is a tool that mutates Move source code. +It can help test the robustness of Move specifications and tests by generating different code versions (mutants). +The tool mutates only the source code. Unit tests and specification blocks are not affected by the tool. Please refer to the design document for more details: [Move Mutator Design](doc/design.md). @@ -26,11 +26,7 @@ cargo test -p move-mutator ## Usage -The `move-mutator` tool can be run using the `move-cli` tool or the `aptos` -tool. The command line options are slightly different for both tools. Last -section of this document describes the command line options for both tools. For -the rest of this document, we will use the `aptos` tool. - +To run the tool, use the command: ```bash ./target/release/move-mutator -m move-mutator/tests/move-assets/simple/sources/Sum.move ``` @@ -69,6 +65,11 @@ It's also possible to generate mutants for a specific module by using the `--mut ```bash ./target/release/move-mutator --package-path move-mutator/tests/move-assets/simple/ --mutate-modules "Sum" ``` +Or use the tool to generate mutants for specific functions: +```bash +# This command will generate mutants only for functions named: 'or', 'and' and 'sum' +./target/release/move-mutator --package-path move-mutator/tests/move-assets/simple/ --mutate-functions or,and,sum +``` The mutator tool generates: - mutants (modified move source code) @@ -76,13 +77,11 @@ The mutator tool generates: Generating mutants for the whole package can be time-consuming. To speed up the process, mutant verification is disabled by default. To enable it, use the -`--verify-mutants - -` option: +`--verify-mutants` option: ```bash ./target/release/move-mutator --package-path move-mutator/tests/move-assets/simple/ --verify-mutants ``` -Mutants verification is done by compiling them. If the compilation fails, +Mutants verification is performed by compiling them. If the compilation fails, the mutant is considered invalid. It's highly recommended to enable this option as it helps to filter out invalid mutants, which would be a waste of time to prove. @@ -95,59 +94,35 @@ directory. They can be used to check the mutator tool as well. To check possible options run: ```bash ./target/release/move-mutator --help + Package and build system for Move code Usage: move-mutator [OPTIONS] Options: - -p, --package-path - The path where to put the output files - -m, --move-sources - The paths to the Move sources - --mutate-modules - Module names to be mutated [default: all] - -o, --out-mutant-dir - The path where to put the output files - --verify-mutants - Indicates if mutants should be verified and made sure mutants can compile - -n, --no-overwrite - Indicates if the output files should be overwritten - --downsampling-ratio-percentage - Remove averagely given percentage of mutants. See the doc for more details - -c, --configuration-file - Optional configuration file. If provided, it will override the default configuration - -d, --dev - Compile in 'dev' mode. The 'dev-addresses' and 'dev-dependencies' fields will be used if this flag is set. This flag is useful for development of packages that expose named addresses that are not set to a specific value - --test - Compile in 'test' mode. The 'dev-addresses' and 'dev-dependencies' fields will be used along with any code in the 'tests' directory - --override-std - Whether to override the standard library with the given version [possible values: mainnet, testnet, devnet] - --doc - Generate documentation for packages - --abi - Generate ABIs for packages - --install-dir - Installation directory for compiled artifacts. Defaults to current directory - --force - Force recompilation of all packages + -p, --package-path The path to the target Move package + -m, --move-sources The paths to the Move sources + --mutate-modules Module names to be mutated [default: all] + -f, --mutate-functions Function names to be mutated [default: all] + -o, --out-mutant-dir The path where to put the output files + --verify-mutants Indicates if mutants should be verified and made sure mutants can compile + -n, --no-overwrite Indicates if the output files should be overwritten + --downsampling-ratio-percentage Remove averagely given percentage of mutants. See the doc for more details + -c, --configuration-file Optional configuration file. If provided, it will override the default configuration + -d, --dev Compile in 'dev' mode. The 'dev-addresses' and 'dev-dependencies' fields will be used if this flag is set. This flag is useful for development of packages that expose named addresses that are not set to a specific value + --test Compile in 'test' mode. The 'dev-addresses' and 'dev-dependencies' fields will be used along with any code in the 'tests' directory + --override-std Whether to override the standard library with the given version [possible values: mainnet, testnet, devnet] + --doc Generate documentation for packages + --abi Generate ABIs for packages + --install-dir Installation directory for compiled artifacts. Defaults to current directory + --force Force recompilation of all packages --arch - - --fetch-deps-only - Only fetch dependency repos to MOVE_HOME - --skip-fetch-latest-git-deps - Skip fetching latest git dependencies - --bytecode-version - Bytecode version to compile move code - --skip-attribute-checks - Do not complain about an unknown attribute in Move code - --compiler-version - Compiler version to use - --language-version - Language version to support - --experiments - Experiments for v2 compiler to set to true - -h, --help - Print help - -V, --version - Print version -``` + --fetch-deps-only Only fetch dependency repos to MOVE_HOME + --skip-fetch-latest-git-deps Skip fetching latest git dependencies + --bytecode-version Bytecode version to compile move code + --skip-attribute-checks Do not complain about an unknown attribute in Move code + --compiler-version Compiler version to use + --language-version Language version to support + --experiments Experiments for v2 compiler to set to true + -h, --help Print help + -V, --version Print version``` diff --git a/move-mutator/doc/design.md b/move-mutator/doc/design.md index 25e21d1356..f9a2a2f0aa 100644 --- a/move-mutator/doc/design.md +++ b/move-mutator/doc/design.md @@ -201,7 +201,6 @@ operators = ["operator1", "operator2"] file = "/path/to/file" verify_mutants = true include_functions = ["function1", "function2"] -exclude_functions = ["function3", "function4"] ``` ### Cross layer