Reliability and security updates

For: uc-cdis/cloud-automation
Notes since tag: pybase3-1.5.0
Notes to tag/commit: pybase3-1.5.1
Generated: 2021-05-24

Bug Fixes

• Disable npm 7 update notifier because it is causing Portal pods to stuck at
  starting up after 7 days (#1592)
• Change domain to correct one (#1586)
• Capture logs from failed pods when K8sReset fails (before the k8s namespace
  is teared down). (#1581)
• Updated hostname so the connection to auspice works (#1571)
• Prevent hung gen3qa-check-bucket-access pods. (#1575)

Improvements

• covid19-bayes-model-(cron)job 20Gi memory (#1583)
• PRC covid19-bayes-model job: add configurable state list (#1579)

Dependency Updates

• Due to a security incident Hashicorp has updated the GPG keys used to sign
  its releases (including providers and plugins etc) (#1595)
• That's why we see cloud-automation CI runs fail on Terraform tests. (#1595)
• We need to bump them to a new version that support the new keys. (#1595)
• Ref: (#1595)
• https://discuss.hashicorp.com/t/terraform-updates-for-hcsec-2021-12/23570
  (#1595)

Python 3 base image: Rust toolchain

Improvements

• Install Rust and Rust toolchain in python-nginx image to fix issues
  installing cryptography (#1568)

Observability Improvements

Release Notes

uc-cdis/cloud-automation

New Features

• chore(observability): Enable metrics scraping for revproxy (#1376)
• Improving prometheus exporter side-cars for fence. (#1376)
• Script for download metrics (#1473)
• New Gen3 CLI command (gen3 gen3qa-run <test_job>) (#1474)
• Google cloud setup for covid19 (#1460)
• Add ability to sync docker images from docker hub to quay.io (#1455)
• Added terraform to create sftp server (#1437)

Deployment Changes

• Put google cloud credential into Gen3Secrets/g3auto/covid19-etl/default.json (#1460)

Dependency Updates

• remove python-pygit2 (#1447)
• pylib2 added (#1446)

Bug Fixes

• fixed helm repo (#1467)
• gen3release has new dependency
  https://github.com/uc-cdis/gen3-release-utils/blob/1b611e6b13371e039a725c24519a7b7ba3354c3f/gen3release-sdk/pyproject.toml#L16
  which requires pip version >19. (#1448)

Improvements

• chore(pybase): Adopt parent img from quay to avoid pull rate limit errors (#1485)
• Clarify metadata exchange instructions for cognito-adfs integration (#1466)
• revproxy sets cache-control: no-store for lw-workspace/proxy (#1469)
• tty service (#1468)
• awshelper changes for tty service - include ubuntu-user sshd (#1468)
• gen3 infra helper - collect infrastructure reports on a VPC for pen
  tests, security reviews, whatever (#1468)
• gen3 api indexd-delete $did (#1468)
• k8s rbac to v1 api (#1468)
• add anaconda.com to whitelist (#1462)
• patch gen3 jupyter idle to properly consider the length limit on
  ambassador cluster names - the hatchery-reaper should properly garbage
  collect hatchery pods with long names now ... (#1452)
• automation for ws-storage service (#1450)
• fix kube-setup-wts - mkdir before writing creds (#1449)
• fix hatchery-reaper - add namespace from metadata (#1449)
• fix gen3 gitops configmaps key1 key2 ... (#1445)
• make kube-setup-wts fail fast on failure to setup OIDC client with fence
  (#1445)
• add AWS_STS_REGIONAL_ENDPOINTS=regional environment to ssjdispatcher
  (#1445)

Python 3 base dockerfile 1.4.0

uc-cdis/cloud-automation

Enabling metrics to be scraped by Prometheus.

Python 3 base dockerfile 1.3.0

uc-cdis/cloud-automation

New Features

• RDS cluster autoscaling can now be enabled in terraform by just setting the
  variables to their desired value (#1362)
• Added default encryption to rds databases. Also changed default size to
  t2.small because encryption is not available for t2.micro instances. (#1346)
• make mariner available for dev and qa testing (#1352)
• Move data replicate jobs from
  https://github.com/uc-cdis/dcf-datareplicate/jobs to cloud-automation
  (#1335)
• Ignore changes if the data-upload-bucket has cors_rules, (#1331)
• Added option to assume role and refactored code (#1327)
• Add auspice service's .yaml file (#1319)
• For CSOC attached commons, logs will now be sent over onto logDNA (#1324)
• Added other run option to allow for Jenkins to get output file with
  information about the run (#1317)
• Added cookbook to manage adminvm (#1288)
• Added wildcard *.chef.io to squid whitelist (#1295)
• Added bucket replication job that uses aws batch (#1294)
• Whitelist *.census.org (#1280)
• Added netpolicy rule for sowerjobs to reach revproxy and utilize internal
  routing (#1266)
• aws batch job for bucket manifest generating tool (#1219)
• COVID19 ETL jobs: add "S3_BUCKET" optional configuration variable + handle
  underscores in job names (#1252)
• .adfs.federation.va.gov whitelisted (#1248)
• cognito integration for SAML authentication. (#1247)
• added mran.microsoft.com to the whitelist (#1241)
• Selenium Hub (#1232)
• kube-setup-seleniumhub script is TBD. (#1232)
• Azure terraform modules. (#1226)
• Added job (#1217)
• Added option to replicate from different source account than adminvm (#1217)
• new kube-setup-sower-jobs command that sets up S3 bucket, service account,
  and fine-grained IAM controls for sower jobs (#1224)
• Added uwsgi timeout optional param to extend read-timeout for fence (#1120)
• AdminVM module off utility VM (utility_admin) (#1208)
• Remove old & unused jobs for covid19 etl (#1207)
• Improve running new jobs for covid19 etl: now they will have unique names
  (#1207)
• gen3 util for creating aws lambda function (#1189)
• gen3 awslambda create funcname description role_arn (#1189)
• New Ansible playbook to add a cronjob to commons user to check on terraform
  resources on daily basis and alert if there are changes outside the
  template. Would also alert if there are uncommitted changes in
  cloud-automation repo locally. (#1194)
• Created bucket replicate script (#1186)
• You can now choose the version you want the ElasticSearch cluster to be
  deployed on. (#1183)
• Notebook ETL job (#1178)
• Doc update (#1181)
• Remove PR template, cloud-automation will use the organization one (#1179)
• Migrated non-sensitive, externally helpful docs from cdis-wiki (#1154)
• Added www.dph.illinois.gov to Squid whitelist (#1166)
• Add new kubernetes job, the data-ingestion-job, which is specific to
  DataSTAGE. (#1012)
• ETL job for Illinois Department for Public Health data (#1162)
• Ability to deploy k8s workers on a /22 subnet, allowing more workers and
  pods in the cluster. (#1152)
• Add COVID-19 ETL job (#1150)
• Added keys for new bdcat cluster to squid (#1140)
• get hostname to indexd for DRS field self_uri (#1133)
• Added script to update ebs volumes (#1130)
• Run WTS DB migration during "kube-setup-wts" (#1128)
• Add empty "external_oidc" field to WTS configuration file (#1128)
• gen3 squid info to get information about the HA-proxy instances (#1137)
• gen3 workers-cycle to cycle a node or all nodes (#1126)
• Switch proxy, let the stand by instance become the active one, or if the
  cluster has more than two instance, a single one will be picked up
  (different from the current instance) as active. (#1125)
• RDS module now creates an Option Group by default that you assign to the
  instance for backing up against s3 (#1119)
• gen3 secrets rotate postgres indexd|sheepdog|fence (#1114)
• kube-dev-namespace sets up new db users for indexd, sheepdog, and fence
  db's (#1114)
• added fence ssh keys from internalanvil to squid (#1115)
• Setup sower job for indexd_utils (#1066)
• AWS inspec implementation for the security team. (#1112)
• added qa-dcf key to squid (#1109)
• metadata service automation (#1087)
• Remediate CIS issues with Amazon Linux workers (#1094)
• Single squid instance type is a variable. (#1092)
• HA squid (#1046)
• add OWASP rules to default modsecurity configuration (#1082)
• ability to run gen3 commands remotely using adminVMs as proxy (#1072)
• EX: (#1072)

• ssh cdistest.csoc -C "~/cloud-automation/files/script/remote-gen3.sh

  kube-setup-revproxy (#1072)

• ansible a-hosts -m shell -a "cloud-automation/files/script/remote-gen3.sh

  kube-setup-revproxy (#1072)
• implement gen3 cmd for creating gs bucket for data refresh (#1060)
• Networkpolicy fixes from VA: Kubernetes YAML syntax fix (#1049)

Dependency Updates

• Bumps lodash from 4.17.15 to 4.17.19.
  (#1329)

Deployment Changes

• (#1362)
• (#1354)
• (#1353)
• (#1352)
• (#1350)
• (#1351)
• (#1345)
• (#1347)
• (#1337)
• (#1341)
• (#1334)
• (#1331)
• (#1332)
• (#1330)
• (#1327)
• (#1328)
• (#1325)
• (#1324)
• (#1318)
• (#1317)
• (#1316)
• (#1298)
• (#1309)
• (#1297)
• (#1288)
• (#1293)
• (#1292)
• (#1287)
• (#1280)
• (#1277)
• (#1274)
• (#1267)
• (#1269)
• (#1266)
• (#1248)
• (#1247)

Python 3 base dockerfile 1.2.0

Add Nginx rate limit to help Fence with its RPS throughput

Python 3 base dockerfile 1.1.0

By default, disable uwsgi cheaper mode and run 2 uwsgi processes

Release new pybase2

pybase2-1.0.2

chore(defaults): values change (#871)

update timeout json response

pybase2-1.0.1

fix(timeout): return valid JSON (#782)

feat(alpine-base): add alpine base dockerfile

Feat/alpine base (#590)

* feat(alpine-base): add alpine base dockerfile