From 89906520120e928196cecc7f0a3734d0360a1ecb Mon Sep 17 00:00:00 2001 From: kuvaldini Date: Mon, 7 Feb 2022 15:14:59 +0200 Subject: [PATCH 1/5] Add .github/_README.md Signed-off-by: kuvaldini --- .github/.gitignore | 1 + .github/_README.make-workflows.md | 95 +++++++++++++++++++++++++++++++ .github/_README.md | 72 +++++++++++++---------- .github/chatops-gen-matrix.sh | 2 + 4 files changed, 141 insertions(+), 29 deletions(-) create mode 100644 .github/.gitignore create mode 100644 .github/_README.make-workflows.md diff --git a/.github/.gitignore b/.github/.gitignore new file mode 100644 index 00000000000..841f5dfee44 --- /dev/null +++ b/.github/.gitignore @@ -0,0 +1 @@ +/matrix* diff --git a/.github/_README.make-workflows.md b/.github/_README.make-workflows.md new file mode 100644 index 00000000000..e465cefbd14 --- /dev/null +++ b/.github/_README.make-workflows.md @@ -0,0 +1,95 @@ +make-workflows.sh for GitHub Actions +==================================== + +GitHub Workflow description in YAML does not support anchors. +There are several workarounds => anyway [they](#links) come to building-editing workflow yaml from source. +So I suggest yet another one `make-workflows.sh` based on YAML tool [`yq`](https://github.com/mikefarah/yq) version 4. + +\All these code and repo is written around `yq eval 'explode(.)'`\ + +### USAGE +0. [Install](#ways-to-install) `make-workflows.sh` to reachable place inside or outside of your repo, i.e. '.github/' +1. Put your workflows to `.github/*.src.yml` +2. (recommended) `pre-commit install` and edit [`.pre-commit-config.yaml`](/.pre-commit-config.yaml) according to where [`make-workflows.sh`](./make-workflows.sh) is placed. + (altenative optional) Copy or link `pre-commit-hook.sh` to `.git/hooks/pre-commit` + Like `ln -s ../../.github/pre-commit-hook.sh .git/hooks/pre-commit` + +``` +$ ./make-workflows.sh --help +make-workflows: + This script expands '*.src.yml' from $1..$[N-1] (default: REPO_ROOT/.github/) + to $N (default:REPO_ROOT/.github/workflows/) with corresponding name '*.yml' + Main goal is to dereference YAML anchors. + Deals only with Git cached/indexed files until --worktree passed. + DEBUG: use option -x + NOTE: spaces in filenames are not allowed to keep code simplicity. +Usage: + make-workflows.sh [--worktree] [dirs_from... [dir_to]] + make-workflows.sh [--help] +Options: + --worktree List files and get contents from working tree + instead of git index + -h, --help show this help + -x, --trace, +x, --no-trace enable/disable bash trace + -i, --install + --update + -V, --version +``` + +### Automate using pre-commit (recommended) +There is a nice tool [pre-commit](https://pre-commit.com) to do checks and some actions just before commit. The tool is called by Git pre-commit hook. + +Making workflows is better being automated – just +```sh +$ pre-commit install +``` +and add next sample to [`.pre-commit-config.yaml`](/.pre-commit-config.yaml) +```yaml +repos: +- repo: local + hooks: + - id: make-workflows + name: Make GitHub workflows from *.src.yml + entry: bash -c '.github/make-workflows.sh && git add .github/workflows' + language: system + files: '.github/.*\.src\.ya?ml' + pass_filenames: false +``` +> NOTE: pay attention to path to `make-workflows.sh` + +> NOTE2: pay attention to path(s) where source files are stored `files: 'PATH_REGEXP'` + + +## Ways to install +1. raw by hand +``` +curl 'https://raw.githubusercontent.com/kuvaldini/make-workflows.sh/main/make-workflows.sh' -LsSf >make-workflows.sh && chmod +x make-workflows.sh +``` + +2. using own facility, installs to /usr/local/bin +``` +curl 'https://raw.githubusercontent.com/kuvaldini/make-workflows.sh/main/make-workflows.sh' -LsSf | bash -s -- --install +``` +> NOTE: may require `...| sudo bash...` + +3. node package manager +``` +npm install kuvaldini/make-workflows.sh +``` +or `npm install git+https://github.com/kuvaldini/make-workflows.sh` + +4. TODO webinstall + + +### Links +1. https://stackoverflow.com/questions/67368724/share-same-steps-for-different-github-actions-jobs +2. https://github.community/t/support-for-yaml-anchors/16128/60 +3. https://github.com/mithro/actions-includes +4. https://github.com/allejo/gha-workflows +5. this repo https://github.com/kuvaldini/make-workflows.sh + +## License, Authors +Authored by [@Kuvaldini](https://github.com/kuvaldini), 2021. +Please keep links to the source code [this repo](https://github.com/kuvaldini/make-workflows.sh) + +Creation of this repo was inspired by [@karfau](https://github.com/karfau) diff --git a/.github/_README.md b/.github/_README.md index 100b4aee2e7..925bb3c86ca 100644 --- a/.github/_README.md +++ b/.github/_README.md @@ -1,31 +1,45 @@ -GitHub Actions -============== +GitHub Actions CI for Iroha +=========================== -GitHub Workflow description in YAML does not support anchors. -There are several workarounds => anyway they come to building-editing workflow yaml from source. -So I suggest yet another one `make-workflows.sh` based on YAML tool `yq`. +> For the **smooth experience** please `pre-commit install` after clone. -### USAGE -0. Move your workflows to `.github/*.src.yml` -1. Put `make-workflows.sh` to directory `.github/` -2. (optional) Copy or link `pre-commit-hook.sh` to `.git/hooks/pre-commit` - Like `ln -s ../../.github/pre-commit-hook.sh .git/hooks/pre-commit` - -### Using pre-commit -```yaml -repos: -- repo: local - hooks: - - id: make-workflows - name: Make GitHub workflows from *.src.yml - entry: bash -c '.github/make-workflows.sh && git add .github/workflows' - language: system - types: [yaml] - pass_filenames: false -``` - -### Links -1. https://stackoverflow.com/questions/67368724/share-same-steps-for-different-github-actions-jobs -2. https://github.community/t/support-for-yaml-anchors/16128/60 -3. https://github.com/mithro/actions-includes -4. https://github.com/allejo/gha-workflows +### List of files +- `build-iroha1.src.yml` + Main file here. GitHub workflow YAML description with ANCHORS, code is not duplicated. + IMPORTANT: regeneration required after after edit, which is automated with pre-commit. +- `workflows/build-iroha1.yml` + Result worflow taken by GitHub and generated with make-workflows script. Long file of repeated code. DO NOT EDIT MANUALLY. +- `make-workflows.sh` + A tool to generate workflows/*.yml from *.src.yml - evaluates anchors. [Read the docs](_README.make-workflows.md). +- `chatops-gen-matrix.sh` + Generates build matrixes form convenient user input. See `--help` + ``` + USAGE: + chatops-gen-matrix.sh --help + echo /build [build_spec...] | chatops-gen-matrix.sh + EXAMPLE build_spec: + /build ubuntu release gcc10 + /build macos llvm release + /build all + /build ubuntu all ## build all possible configurations on Ubuntu + /build ubuntu burrow all ## build all possible configurations on Ubuntu with Burrow + AVAILABLE build_spec keywords: + ubuntu|linux + macos + windows + normal + burrow + ursa + release|Release + debug|Debug + gcc|gcc-9|gcc9 + gcc-10|gcc10 + clang|clang-10|clang10 + llvm + msvc + all|everything|beforemerge|before_merge|before-merge|readytomerge|ready-to-merge|ready_to_merge + ``` +- `pre-commit-hook.sh` + See docs of make-workflows. Use instead of pre-commit as `ln -s ../../.github/pre-commit-hook.sh .git/hooks/pre-commit`, reserv alternative. +- `TESTS_ALLOWED_TO_FAIL` + One day tests of Iroha become failing. To fix CI and postpone fixing tests, this file was invented. It allows CI to pass even when listed tests are failing. HACK. DO NOT USE UNLESS YOU DEFINITELY KNOW WHAT'S GOING. diff --git a/.github/chatops-gen-matrix.sh b/.github/chatops-gen-matrix.sh index a4494248b07..4997b7f5534 100755 --- a/.github/chatops-gen-matrix.sh +++ b/.github/chatops-gen-matrix.sh @@ -39,6 +39,7 @@ END cat < Date: Mon, 7 Feb 2022 20:17:21 +0200 Subject: [PATCH 2/5] /build skip_testing; And more docs Signed-off-by: kuvaldini --- .github/_README.md | 46 ++++++++++++++------ .github/build-iroha1.src.yml | 40 +++++++++++++++--- .github/chatops-gen-matrix.sh | 7 +-- .github/workflows/build-iroha1.yml | 68 ++++++++++++++++++++++++++---- 4 files changed, 131 insertions(+), 30 deletions(-) diff --git a/.github/_README.md b/.github/_README.md index 925bb3c86ca..e8b4c9e3e13 100644 --- a/.github/_README.md +++ b/.github/_README.md @@ -1,28 +1,49 @@ GitHub Actions CI for Iroha =========================== -> For the **smooth experience** please `pre-commit install` after clone. -### List of files -- `build-iroha1.src.yml` - Main file here. GitHub workflow YAML description with ANCHORS, code is not duplicated. +> ### For the **smooth experience** please `pre-commit install` after `git clone`. + +--------------------- + +USAGE +----- +GitHub Actions Workflow [`Iroha1`](build-iroha1.src.yml) solves task of automated build and deployment Iroha1. +There are events when it is running: +- on **pull request** to Iroha1 main and development branches +- on **push** to main or development branches including event when PR is **merged** +- on **workflow dispatch** to run WF manually on special branch with defined buildspec through web interface or via CLI tool +- **scheduled** every night +- _(under construction PR #XX) on **comment to PR** which contains buildspec._ + +Default `buildspec` is _`/build all`_ + +### Buildspec +Build matrix is a way to select among number of configurations to be built. +Build matrix is generated from buildspec string and handled by script [`chatops-gen-matrix.sh`](./chatops-gen-matrix.sh) + +## List of files +- `build-iroha1.src.yml` + Main file here. GitHub workflow YAML description with ANCHORS, code is not duplicated. IMPORTANT: regeneration required after after edit, which is automated with pre-commit. -- `workflows/build-iroha1.yml` +- `workflows/build-iroha1.yml` Result worflow taken by GitHub and generated with make-workflows script. Long file of repeated code. DO NOT EDIT MANUALLY. -- `make-workflows.sh` +- `make-workflows.sh` A tool to generate workflows/*.yml from *.src.yml - evaluates anchors. [Read the docs](_README.make-workflows.md). -- `chatops-gen-matrix.sh` +- `chatops-gen-matrix.sh` Generates build matrixes form convenient user input. See `--help` ``` USAGE: chatops-gen-matrix.sh --help + chatops-gen-matrix.sh /build ubuntu clang + chatops-gen-matrix.sh '/build ubuntu clang; /build macos release ursa' echo /build [build_spec...] | chatops-gen-matrix.sh EXAMPLE build_spec: /build ubuntu release gcc10 - /build macos llvm release + /build macos llvm release; /build macos clang ursa release /build all /build ubuntu all ## build all possible configurations on Ubuntu - /build ubuntu burrow all ## build all possible configurations on Ubuntu with Burrow + /build ubuntu burrow all ## build release and debug on Ubuntu with Burrow AVAILABLE build_spec keywords: ubuntu|linux macos @@ -37,9 +58,10 @@ GitHub Actions CI for Iroha clang|clang-10|clang10 llvm msvc + skip-testing|skip_testing all|everything|beforemerge|before_merge|before-merge|readytomerge|ready-to-merge|ready_to_merge ``` -- `pre-commit-hook.sh` +- `pre-commit-hook.sh` See docs of make-workflows. Use instead of pre-commit as `ln -s ../../.github/pre-commit-hook.sh .git/hooks/pre-commit`, reserv alternative. -- `TESTS_ALLOWED_TO_FAIL` - One day tests of Iroha become failing. To fix CI and postpone fixing tests, this file was invented. It allows CI to pass even when listed tests are failing. HACK. DO NOT USE UNLESS YOU DEFINITELY KNOW WHAT'S GOING. +- `TESTS_ALLOWED_TO_FAIL` + One day tests of Iroha became failing. To fix CI and postpone fixing tests, this file was invented. It allows CI to pass even when listed tests are failing. DO NOT USE UNLESS YOU DEFINITELY KNOW WHAT'S GOING. KEEP IT EMPTY. diff --git a/.github/build-iroha1.src.yml b/.github/build-iroha1.src.yml index 8d824764f77..b778e1e2b49 100644 --- a/.github/build-iroha1.src.yml +++ b/.github/build-iroha1.src.yml @@ -15,7 +15,7 @@ name: Iroha1 ## and from 2hrs to 27min without cache ## GitHub's default runners also idles much time before started when build often ## TODO 5. [speed,optimization,resources] Cancel previous runs if their duration is less than 10 minutes, protect almost done builds from being killed -## TODO [prettify,documentation] update status badges in README.md +## TODO [prettify,documentation] update status badges in /README.md ## TODO [minor] windows ## TODO actions/create-release for main branch and tags ## TODO [cmake,dockerimage,iroha-builder] To improve speed of vcpkg step install to iroha-builder @@ -55,9 +55,31 @@ on: ## NOTE: Able to run via cmdline: gh workflow run Iroha1 inputs: build_spec: - description: 'See chatops-gen-matrix.sh, example "/build ubuntu macos gcc-9 burrow"' - required: false - default: '/build' + description: | + See chatops-gen-matrix.sh, example "/build ubuntu macos gcc-9 burrow" + EXAMPLE build_spec: + /build ubuntu release gcc10 + /build macos llvm release; /build macos clang ursa release + /build all + /build ubuntu all ## build all possible configurations on Ubuntu + /build ubuntu burrow all ## build release and debug on Ubuntu with Burrow + AVAILABLE build_spec keywords: + ubuntu|linux + macos + windows + normal + burrow + ursa + release|Release + debug|Debug + gcc|gcc-9|gcc9 + gcc-10|gcc10 + clang|clang-10|clang10 + llvm + skip-testing|skip_testing + all|everything|beforemerge|before_merge|before-merge|readytomerge|ready-to-merge|ready_to_merge + required: true + default: '/build skip_testing ' # issue_comment: # types: [created, edited] schedule: @@ -447,6 +469,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - *step_detect_commented_pr - &step_checkout_full <<: *step_checkout @@ -511,8 +538,8 @@ jobs: -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE - -DTESTING=ON - -DBENCHMARKING=ON + -DTESTING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) + -DBENCHMARKING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DPACKAGE_DEB=ON #-DCMAKE_VERBOSE_MAKEFILE=ON ## Note: use for debug - &step_cmake_build @@ -574,6 +601,7 @@ jobs: build/test_data/** - &step_ctest timeout-minutes: 40 + if: env.skip_testing == '' name: CTest run: | echo ::group::'boilerplate' diff --git a/.github/chatops-gen-matrix.sh b/.github/chatops-gen-matrix.sh index 4997b7f5534..39b306e452a 100755 --- a/.github/chatops-gen-matrix.sh +++ b/.github/chatops-gen-matrix.sh @@ -40,6 +40,7 @@ END USAGE: $(basename $0) --help $(basename $0) /build ubuntu clang + $(basename $0) '/build ubuntu clang; /build macos release ursa' echo /build [build_spec...] | $(basename $0) END --help-buildspec @@ -73,7 +74,7 @@ handle_user_line(){ return fi shift - local oses compilers cmake_opts build_types + local oses compilers cmake_opts build_types skip_testing= while [[ $# > 0 ]] ;do case "$1" in @@ -91,7 +92,7 @@ handle_user_line(){ clang|clang-10|clang10) compilers+=" clang clang-10" ;; llvm) compilers+=" $1 " ;; msvc) compilers+=" $1 " ;; - skip-testing|skip_testing) skip_testing=yes ;; + skip-testing|skip_testing) skip_testing='skip_testing' ;; all|everything|beforemerge|before_merge|before-merge|readytomerge|ready-to-merge|ready_to_merge) oses=${oses:-"$ALL_oses"} build_types=${build_types:-"$ALL_build_types"} @@ -127,7 +128,7 @@ handle_user_line(){ for co in $cmake_opts ;do if test $os = macos -a $co = burrow; then continue; fi ##Reduce macos load on CI if test $os = macos -a $co = ursa; then continue; fi ##Reduce macos load on CI - MATRIX+="$os $cc $bt $co"$'\n' + MATRIX+="$os $cc $bt $co $skip_testing"$'\n' done done done diff --git a/.github/workflows/build-iroha1.yml b/.github/workflows/build-iroha1.yml index 43b5eda8f73..1e84431c791 100644 --- a/.github/workflows/build-iroha1.yml +++ b/.github/workflows/build-iroha1.yml @@ -17,7 +17,7 @@ name: Iroha1 ## and from 2hrs to 27min without cache ## GitHub's default runners also idles much time before started when build often ## TODO 5. [speed,optimization,resources] Cancel previous runs if their duration is less than 10 minutes, protect almost done builds from being killed -## TODO [prettify,documentation] update status badges in README.md +## TODO [prettify,documentation] update status badges in /README.md ## TODO [minor] windows ## TODO actions/create-release for main branch and tags ## TODO [cmake,dockerimage,iroha-builder] To improve speed of vcpkg step install to iroha-builder @@ -57,9 +57,31 @@ on: ## NOTE: Able to run via cmdline: gh workflow run Iroha1 inputs: build_spec: - description: 'See chatops-gen-matrix.sh, example "/build ubuntu macos gcc-9 burrow"' - required: false - default: '/build' + description: | + See chatops-gen-matrix.sh, example "/build ubuntu macos gcc-9 burrow" + EXAMPLE build_spec: + /build ubuntu release gcc10 + /build macos llvm release; /build macos clang ursa release + /build all + /build ubuntu all ## build all possible configurations on Ubuntu + /build ubuntu burrow all ## build release and debug on Ubuntu with Burrow + AVAILABLE build_spec keywords: + ubuntu|linux + macos + windows + normal + burrow + ursa + release|Release + debug|Debug + gcc|gcc-9|gcc9 + gcc-10|gcc10 + clang|clang-10|clang10 + llvm + skip-testing|skip_testing + all|everything|beforemerge|before_merge|before-merge|readytomerge|ready-to-merge|ready_to_merge + required: true + default: '/build skip_testing ' # issue_comment: # types: [created, edited] schedule: @@ -497,6 +519,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -560,7 +587,7 @@ jobs: # sys time 0,70 secs 575,00 micros 0,70 secs - name: CMake configure ## Takes 13s on regular GitHub runner - run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=ON -DBENCHMARKING=ON -DPACKAGE_DEB=ON + run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DBENCHMARKING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DPACKAGE_DEB=ON #-DCMAKE_VERBOSE_MAKEFILE=ON ## Note: use for debug - name: CMake build run: | @@ -612,6 +639,7 @@ jobs: build/test_bin/** build/test_data/** - timeout-minutes: 40 + if: env.skip_testing == '' name: CTest run: | echo ::group::'boilerplate' @@ -777,6 +805,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -840,7 +873,7 @@ jobs: # sys time 0,70 secs 575,00 micros 0,70 secs - name: CMake configure ## Takes 13s on regular GitHub runner - run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=ON -DBENCHMARKING=ON -DPACKAGE_DEB=ON + run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DBENCHMARKING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DPACKAGE_DEB=ON #-DCMAKE_VERBOSE_MAKEFILE=ON ## Note: use for debug - name: CMake build run: | @@ -892,6 +925,7 @@ jobs: build/test_bin/** build/test_data/** - timeout-minutes: 40 + if: env.skip_testing == '' name: CTest run: | echo ::group::'boilerplate' @@ -1070,6 +1104,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - name: Homebrew run: brew install cmake ninja coreutils ccache bash ## Takes 22 seconds with default github runner @@ -1147,7 +1186,7 @@ jobs: # sys time 0,70 secs 575,00 micros 0,70 secs - name: CMake configure ## Takes 13s on regular GitHub runner - run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=ON -DBENCHMARKING=ON -DPACKAGE_DEB=ON + run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DBENCHMARKING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DPACKAGE_DEB=ON #-DCMAKE_VERBOSE_MAKEFILE=ON ## Note: use for debug - name: CMake build run: | @@ -1192,7 +1231,8 @@ jobs: build/test_data/** - name: Install Postgres on MacOS run: brew install postgresql - - name: CTest + - if: env.skip_testing == '' + name: CTest run: | echo ::group::'boilerplate' set -euo pipefail @@ -1375,7 +1415,7 @@ jobs: # sys time 0,70 secs 575,00 micros 0,70 secs - name: CMake configure ## Takes 13s on regular GitHub runner - run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=ON -DBENCHMARKING=ON -DPACKAGE_DEB=ON + run: cmake -B build -DCMAKE_TOOLCHAIN_FILE=$PWD/vcpkg-build/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=${{ env.BuildType }} -GNinja $CMAKE_USE -DTESTING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DBENCHMARKING=$( test "$skip_testing" = skip_testing && echo OFF || echo ON ) -DPACKAGE_DEB=ON #-DCMAKE_VERBOSE_MAKEFILE=ON ## Note: use for debug - name: CMake build run: | @@ -1460,6 +1500,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -1635,6 +1680,11 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing + # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF + # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON + # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > From d4d26e976926d691956d1a0dab0404a16f85132f Mon Sep 17 00:00:00 2001 From: kuvaldini Date: Mon, 7 Feb 2022 21:46:23 +0200 Subject: [PATCH 3/5] /build all skip_testing Signed-off-by: kuvaldini From 22c7f2720880752eb5b55352529f3b3b9546d28d Mon Sep 17 00:00:00 2001 From: kuvaldini Date: Mon, 7 Feb 2022 22:55:37 +0200 Subject: [PATCH 4/5] Fix 'check if docker image exists' /build all skip_testing Signed-off-by: kuvaldini --- .github/build-iroha1.src.yml | 8 +++++--- .github/workflows/build-iroha1.yml | 16 +++++++++------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/.github/build-iroha1.src.yml b/.github/build-iroha1.src.yml index b778e1e2b49..5dfbee00310 100644 --- a/.github/build-iroha1.src.yml +++ b/.github/build-iroha1.src.yml @@ -387,9 +387,11 @@ jobs: id: dockertag_already run: | echo "::set-output name=container::$DOCKERHUB_ORG/iroha-builder:$dockertag" + docker pull "$DOCKERHUB_ORG/iroha-builder:$dockertag" - - name: Possible ERROR, Dockerfile edited but image cannot be pushed - if: ${{ steps.docker_login.outcome != 'success' || steps.build_and_push.outcome != 'success' }} + name: Possible ERROR, Dockerfile edited, image was build, but seems not pushed, CANNOT PULL. + if: failure() + #if: ${{ steps.docker_login.outcome != 'success' || steps.build_and_push.outcome != 'success' }} env: container: ${{steps.dockertag_already.outputs.container}} dockertag: ${{env.dockertag}} @@ -469,7 +471,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON diff --git a/.github/workflows/build-iroha1.yml b/.github/workflows/build-iroha1.yml index 1e84431c791..ac64b0694af 100644 --- a/.github/workflows/build-iroha1.yml +++ b/.github/workflows/build-iroha1.yml @@ -415,8 +415,10 @@ jobs: id: dockertag_already run: | echo "::set-output name=container::$DOCKERHUB_ORG/iroha-builder:$dockertag" - - name: Possible ERROR, Dockerfile edited but image cannot be pushed - if: ${{ steps.docker_login.outcome != 'success' || steps.build_and_push.outcome != 'success' }} + docker pull "$DOCKERHUB_ORG/iroha-builder:$dockertag" + - name: Possible ERROR, Dockerfile edited, image was build, but seems not pushed, CANNOT PULL. + if: failure() + #if: ${{ steps.docker_login.outcome != 'success' || steps.build_and_push.outcome != 'success' }} env: container: ${{steps.dockertag_already.outputs.container}} dockertag: ${{env.dockertag}} @@ -519,7 +521,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON @@ -805,7 +807,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON @@ -1104,7 +1106,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON @@ -1500,7 +1502,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON @@ -1680,7 +1682,7 @@ jobs: *) echo "::error::Unknown features '$features'"; false ;; esac echo >>$GITHUB_ENV features="$features" - echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fq skip_testing) + echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON From 98aa6a3d3c9d4e5c4f365ef298bb70409b599259 Mon Sep 17 00:00:00 2001 From: kuvaldini Date: Tue, 8 Feb 2022 15:08:52 +0200 Subject: [PATCH 5/5] Reduce text, one important TODO Signed-off-by: kuvaldini --- .github/_README.make-workflows.md | 33 +++--------------------------- .github/build-iroha1.src.yml | 5 +---- .github/workflows/build-iroha1.yml | 21 +------------------ 3 files changed, 5 insertions(+), 54 deletions(-) diff --git a/.github/_README.make-workflows.md b/.github/_README.make-workflows.md index e465cefbd14..89154d7442a 100644 --- a/.github/_README.make-workflows.md +++ b/.github/_README.make-workflows.md @@ -11,7 +11,7 @@ So I suggest yet another one `make-workflows.sh` based on YAML tool [`yq`](https 0. [Install](#ways-to-install) `make-workflows.sh` to reachable place inside or outside of your repo, i.e. '.github/' 1. Put your workflows to `.github/*.src.yml` 2. (recommended) `pre-commit install` and edit [`.pre-commit-config.yaml`](/.pre-commit-config.yaml) according to where [`make-workflows.sh`](./make-workflows.sh) is placed. - (altenative optional) Copy or link `pre-commit-hook.sh` to `.git/hooks/pre-commit` + (altenative optional) Copy or link `pre-commit-hook.sh` to `.git/hooks/pre-commit` Like `ln -s ../../.github/pre-commit-hook.sh .git/hooks/pre-commit` ``` @@ -39,7 +39,7 @@ Options: ### Automate using pre-commit (recommended) There is a nice tool [pre-commit](https://pre-commit.com) to do checks and some actions just before commit. The tool is called by Git pre-commit hook. -Making workflows is better being automated – just +Making workflows is better being automated – just ```sh $ pre-commit install ``` @@ -60,36 +60,9 @@ repos: > NOTE2: pay attention to path(s) where source files are stored `files: 'PATH_REGEXP'` -## Ways to install -1. raw by hand -``` -curl 'https://raw.githubusercontent.com/kuvaldini/make-workflows.sh/main/make-workflows.sh' -LsSf >make-workflows.sh && chmod +x make-workflows.sh -``` - -2. using own facility, installs to /usr/local/bin -``` -curl 'https://raw.githubusercontent.com/kuvaldini/make-workflows.sh/main/make-workflows.sh' -LsSf | bash -s -- --install -``` -> NOTE: may require `...| sudo bash...` - -3. node package manager -``` -npm install kuvaldini/make-workflows.sh -``` -or `npm install git+https://github.com/kuvaldini/make-workflows.sh` - -4. TODO webinstall - - ### Links 1. https://stackoverflow.com/questions/67368724/share-same-steps-for-different-github-actions-jobs 2. https://github.community/t/support-for-yaml-anchors/16128/60 3. https://github.com/mithro/actions-includes 4. https://github.com/allejo/gha-workflows -5. this repo https://github.com/kuvaldini/make-workflows.sh - -## License, Authors -Authored by [@Kuvaldini](https://github.com/kuvaldini), 2021. -Please keep links to the source code [this repo](https://github.com/kuvaldini/make-workflows.sh) - -Creation of this repo was inspired by [@karfau](https://github.com/karfau) +5. dedicated repo https://github.com/kuvaldini/make-workflows.sh diff --git a/.github/build-iroha1.src.yml b/.github/build-iroha1.src.yml index 5dfbee00310..0870f13a111 100644 --- a/.github/build-iroha1.src.yml +++ b/.github/build-iroha1.src.yml @@ -1,5 +1,6 @@ name: Iroha1 +## TODO IMPORTANT DISALLOW deploying tags and main and develop builds where skip_testing was set. ## TODO 1. [vcpkg,optimization-space,optimization-speed] ## Build only Debug or only Release - reduce vcpkg build duration and output size 2times ## see directory triplets/, `vcpkg help triplets` and link: https://stackoverflow.com/a/52781832/3743145 @@ -472,10 +473,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - *step_detect_commented_pr - &step_checkout_full <<: *step_checkout diff --git a/.github/workflows/build-iroha1.yml b/.github/workflows/build-iroha1.yml index ac64b0694af..08f72af2555 100644 --- a/.github/workflows/build-iroha1.yml +++ b/.github/workflows/build-iroha1.yml @@ -2,6 +2,7 @@ ## Generated from build-iroha1.src.yml with make-workflows.sh name: Iroha1 +## TODO IMPORTANT DISALLOW deploying tags and main and develop builds where skip_testing was set. ## TODO 1. [vcpkg,optimization-space,optimization-speed] ## Build only Debug or only Release - reduce vcpkg build duration and output size 2times ## see directory triplets/, `vcpkg help triplets` and link: https://stackoverflow.com/a/52781832/3743145 @@ -522,10 +523,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -808,10 +805,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -1107,10 +1100,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - name: Homebrew run: brew install cmake ninja coreutils ccache bash ## Takes 22 seconds with default github runner @@ -1503,10 +1492,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: > @@ -1683,10 +1668,6 @@ jobs: esac echo >>$GITHUB_ENV features="$features" echo >>$GITHUB_ENV skip_testing=$(echo ${{matrix.buildspec}} | grep -Fo skip_testing) - # if [[ " ${{matrix.buildspec}} " = *" skip_testing "* ]] #echo ${{matrix.buildspec}} | grep -Fq skip_testing - # then echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=OFF CMAKE_BENCHMARKING=OFF - # else echo >>$GITHUB_ENV CMAKE_TESTING=-DTESTING=ON CMAKE_BENCHMARKING=ON - # fi - name: REF and SHA of commented PR to ENV if: github.event.comment run: >