Skip to content

Commit

Permalink
DAC-1431 Create redshift migration workflow (#275)
Browse files Browse the repository at this point in the history 
Add new run-flyway-command workflow
Add scripts to be called from new workflow
Add needed role and VPC ingress to IaC code
Add new skip-tests option to pull-request-deploy-and-test
Fix formatting of code-quality-sonarcloud workflow and make file extension consistent
Document sonarcloud and flyway workflows in README
  • Loading branch information
@hdavey-gds
hdavey-gds authored Oct 23, 2023
1 parent 003b1c1 commit 1079513
Show file tree
Hide file tree
Showing 7 changed files with 220 additions and 16 deletions.
6 changes: 4 additions & 2 deletions ...ub/workflows/code-quality-sonarcloud.yaml → ...hub/workflows/code-quality-sonarcloud.yml
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
name: SonarCloud Code Analysis

on:
push:
branches:
- main
pull_request:
types: [opened, synchronize, reopened]

jobs:
sonarcloud:
name: SonarCloud
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
8 changes: 8 additions & 0 deletions .github/workflows/pull-request-deploy-and-test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@ name: ✳️ Pull request deploy and test

on:
workflow_dispatch:
inputs:
skip-tests:
description: Skip running the smoke and integration tests (i.e. just deploy to feature)
type: boolean
required: false
default: false
pull_request_review:
types:
- submitted
Expand Down Expand Up @@ -73,6 +79,7 @@ jobs:
sam deploy --region eu-west-2 --stack-name dap --parameter-overrides "$PARAMETER_OVERRIDES" --signing-profiles "$PROFILES" --resolve-s3 --no-confirm-changeset --no-fail-on-empty-changeset
run-integration-tests:
if: inputs.skip-tests == false
needs: [deploy-to-feature]
# These permissions are needed to interact with GitHub's OIDC Token endpoint (enabling the aws-actions/configure-aws-credentials action)
permissions:
Expand Down Expand Up @@ -105,6 +112,7 @@ jobs:
TXMA_QUEUE_URL: https://sqs.eu-west-2.amazonaws.com/655068466146/feature-placeholder-txma-event-queue

run-smoke-tests:
if: inputs.skip-tests == false
needs: [deploy-to-feature]
# These permissions are needed to interact with GitHub's OIDC Token endpoint (enabling the aws-actions/configure-aws-credentials action)
permissions:
Expand Down
58 changes: 58 additions & 0 deletions .github/workflows/run-flyway-command.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: ✳️ Run flyway command on redshift

on:
workflow_dispatch:
inputs:
environment:
type: string
required: true
description: AWS environment - one of DEV, TEST, FEATURE, BUILD, STAGING, INTEGRATION or PRODUCTION
command:
type: choice
required: true
description: Flyway command to run
options:
- info
- migrate
- clean
- clean migrate

jobs:
validate-environment:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
- name: Validate input environment
run: scripts/validate-environment.sh ${{ inputs.environment }}

run-flyway:
needs: [validate-environment]
# These permissions are needed to interact with GitHub's OIDC Token endpoint (enabling the aws-actions/configure-aws-credentials action)
permissions:
id-token: write
contents: read
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
- name: Assume AWS DB migration role
uses: aws-actions/configure-aws-credentials@v2
with:
aws-region: eu-west-2
role-to-assume: ${{ secrets[format('DB_MIGRATION_ROLE_ARN_{0}', inputs.environment)] }}
- name: Get AWS account id
run: echo "account_id=$(aws sts get-caller-identity --output text --query 'Account')" >> $GITHUB_ENV
- name: Prepare migrations folder
run: scripts/prepare-redshift-migrations.sh ${{ inputs.environment }} $account_id
- name: Download JARs
run: |
mkdir ${{ github.workspace }}/redshift-scripts/drivers
cd ${{ github.workspace }}/redshift-scripts/drivers
wget https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/2.1.0.20/redshift-jdbc42-2.1.0.20.jar
wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.3/jackson-databind-2.15.3.jar
- name: Run flyway command
run: |
ENVIRONMENT=$(echo ${{ inputs.environment }})
WORKGROUP_NAME="${ENVIRONMENT,,}-redshift-serverless-workgroup"
scripts/run-flyway-command.sh $WORKGROUP_NAME ${{ inputs.command }}
30 changes: 16 additions & 14 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,20 +61,22 @@ The [AWS SAM](https://aws.amazon.com/serverless/sam) config is at [samconfig.tom
#### Workflows

[Workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) that enable [GitHub Actions](https://docs.github.com/en/actions) can be found in the [.github/workflows](.github/workflows) directory.
Below is a list of workflows:

| Name | File | Triggers | Purpose |
|-----------------------------------|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| Deploy to an AWS environment | deploy-to-aws.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li></ul> | Deploys to a deployable AWS environment (dev, build, test) |
| Deploy to the test environment | deploy-to-test.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Deploys IaC and lambda code to the test AWS |
| Deploy to the dev environment | deploy-to-dev.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Deploys IaC and lambda code to the dev AWS |
| Deploy to the build environment | deploy-to-build.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li></ul> | Deploys IaC and lambda code to the build AWS |
| Test and validate iac and lambdas | test-and-validate.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li><li>[pull requests](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Runs linting, formatting and testing of lambda code, and linting and scanning of IaC code |
| Upload Athena files to S3 | upload-athena-files.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Uploads athena scripts for a particular environment (under [athena-scripts](athena-scripts)) to S3 |
| Pull request deploy and test | pull-request-deploy-and-test.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li><li>[pull requests (on open, reopen and update)](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li></ul> | Deploys a pull request branch to the feature environment and runs integration tests when a pull request is opened, reopened or updated |
| Pull request tear down | pull-request-tear-down.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li><li>[pull requests (on close)](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li></ul> | Tears down the feature environment when a pull request is merged or otherwise closed |
| Upload testing image to ECR | upload-testing-image.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li></ul> | Builds a testing dockerfile in `tests/scripts/` and uploads the image to ECR |
| Upload testing images to ECR | upload-testing-images.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Builds one or more testing dockerfiles in `tests/scripts/` and uploads the images to ECR. Which dockerfiles to build can be specified via inputs |
Below is a list of workflows. The ✳️ symbol at the start of a workflow name indicates that it can be run manually.

| Name | File | Triggers | Purpose |
|--------------------------------------|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| Deploy to an AWS environment | deploy-to-aws.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li></ul> | Deploys to a deployable AWS environment (dev, build, test) |
| ✳️ Deploy to the test environment | deploy-to-test.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Deploys IaC and lambda code to the test AWS |
| ✳️ Deploy to the dev environment | deploy-to-dev.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Deploys IaC and lambda code to the dev AWS |
| Deploy to the build environment | deploy-to-build.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li></ul> | Deploys IaC and lambda code to the build AWS |
| ✳️ Test and validate iac and lambdas | test-and-validate.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li><li>[pull requests](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Runs linting, formatting and testing of lambda code, and linting and scanning of IaC code |
| ✳️ Upload Athena files to S3 | upload-athena-files.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Uploads athena scripts for a particular environment (under [athena-scripts](athena-scripts)) to S3 |
| ✳️ Pull request deploy and test | pull-request-deploy-and-test.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li><li>[pull requests (on open, reopen and update)](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li></ul> | Deploys a pull request branch to the feature environment and runs integration tests when a pull request is opened, reopened or updated |
| ✳️ Pull request tear down | pull-request-tear-down.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li><li>[pull requests (on close)](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li></ul> | Tears down the feature environment when a pull request is merged or otherwise closed |
| Upload testing image to ECR | upload-testing-image.yml | <ul><li>[other workflows](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_call)</li></ul> | Builds a testing dockerfile in `tests/scripts/` and uploads the image to ECR |
| ✳️ Upload testing images to ECR | upload-testing-images.yml | <ul><li>[merge to main](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push)</li><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Builds one or more testing dockerfiles in `tests/scripts/` and uploads the images to ECR. Which dockerfiles to build can be specified via inputs |
| SonarCloud Code Analysis | code-quality-sonarcloud.yml | <ul><li>[pull requests (on open, reopen and update)](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)</li></ul> | Runs a SonarCloud analysis on the repository |
| ✳️ Run flyway command on redshift | run-flyway-command.yml | <ul><li>[manual](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch)</li></ul> | Runs a specified flyway command on the redshift database in a specified environment |

## Testing

Expand Down
Loading

0 comments on commit 1079513

Please sign in to comment.