rekey ansible vault keys

[kayiwa]

This issue tracks the necessary steps to re-key all Ansible vaults used in our infrastructure and subsequently re-run the affected playbooks to update encrypted data with the new keys.

Reason for Re-keying:

Proactive security measure: Regularly rotating encryption keys is a best practice to enhance security.
Former employee departure: An employee with knowledge of the vault password has left the organization.

Affected Vaults:

[List all affected vault files]

• group_vars/abid/vault.yml

• group_vars/all/vault.yml

• group_vars/allsearch_api/vault.yml

• group_vars/allsearch_frontend/vault.yml

• group_vars/ansible_tower/vault.yml

• group_vars/approvals/vault.yml

• group_vars/bibdata/vault.yml

• group_vars/byzantine/vault.yml

• group_vars/checkmk/vault.yml

• group_vars/cicognara/vault.yml

• group_vars/crowdstrike/vault.yml

• group_vars/dpul/vault.yml

• group_vars/drupal/vault.yml

• group_vars/dss/vault.yml

• group_vars/ealapps/vault.yml

• group_vars/ezproxy/vault.yml

• group_vars/figgy/vault.yml

• group_vars/gitlab/vault.yml

• group_vars/globus/vault.yml

• group_vars/iiif/vault.yml

• group_vars/lae/vault.yml

• group_vars/lib_jobs/vault.yml

• group_vars/lib_svn/vault.yml

• group_vars/libwww/vault.yml

• group_vars/lockers_and_study_spaces/vault.yml

• group_vars/mflux/vault.yml

• group_vars/mysql/vault.yml

• group_vars/nomad/dpulc/vault.yml

• group_vars/nomad/imagecat/vault.yml

• group_vars/nomad/logging/vault.yml

• group_vars/nomad/vault.yml

• group_vars/oawaiver/vault.yml

• group_vars/ojs/vault.yml

• group_vars/orangelight/vault.yml

• group_vars/orcid/vault.yml

• group_vars/ouranos/vault.yml

• group_vars/pas/vault.yml

• group_vars/pdc_describe/vault.yml

• group_vars/pdc_discovery/vault.yml

• group_vars/postgresql/vault.yml

• group_vars/pulfalight/vault.yml

• group_vars/pulmap/vault.yml

• group_vars/recap/vault.yml

• group_vars/redis/vault.yml

• group_vars/repec/vault.yml

• group_vars/researchdata/vault.yml

• group_vars/sftp/vault.yml

• group_vars/solr8cloud/vault.yml

• group_vars/solr9cloud/vault.yml

• group_vars/special_collections/vault.yml

• group_vars/static_tables/vault.yml

• group_vars/tigerdata/PeopleSoft-Departments.csv

• group_vars/tigerdata/vault.yml

• group_vars/towerdeploy/vault.yml

• group_vars/vsphere/vault.yml

• group_vars/whichiso/vault.yml

• roles/bibdata/files/bibdata-worker.smb.credentials

• roles/bibdata/files/bibdata_share.smb.credentials

• roles/bibdata/files/scratch.smb.credentials

• roles/drupal/files/drupalweb.smb.credentials

• roles/ealapps/files/id_rsa

• roles/ezproxy/files/config.txt

• roles/ezproxy/files/id_rsa

• roles/ezproxy/files/princeton_allow.txt

• roles/ezproxy/files/ssl/priv/ezproxy-test_princeton_edu_priv.key

• roles/ezproxy/files/user.txt

• roles/figgy/files/archives.smb.credentials

• roles/figgy/files/archives_bd.smb.credentials

• roles/figgy/files/bitcur-archives.smb.credentials

• roles/figgy/files/hydradev.smb.credentials

• roles/figgy/files/illiad.smb.credentials

• roles/figgy/files/libimages2.smb.credentials

• roles/figgy/files/maplab.smb.credentials

• roles/figgy/files/marquand.smb.credentials

• roles/figgy/files/mendel.smb.credentials

• roles/figgy/files/microforms.smb.credentials

• roles/figgy/files/mudd.smb.credentials

• roles/figgy/files/numis.smb.credentials

• roles/figgy/files/plum_mount.smb.credentials

• roles/figgy/files/production-google_cloud_credentials.json

• roles/figgy/files/pudl.smb.credentials

• roles/figgy/files/staging-google_cloud_credentials.json

• roles/figgy/files/studio.new.smb.credentials

• roles/figgy/files/studio.smb.credentials

• roles/hr_share/files/hr_share.smb.credentials

• roles/lib_jobs/files/onbase.smb.credentials

• roles/lib_jobs/files/peoplesoft.smb.credentials

• roles/lib_sftp/files/id_ed25519

• roles/libstatic/files/id_rsa

• roles/libstatic/templates/mssimages_ed25519

• roles/libwww/files/id_rsa

• roles/mflux/files/gssapi_jaas.conf

• roles/mflux/files/krb5.conf

• roles/mflux/files/licence-ci.xml

• roles/mflux/files/licence-docker.xml

• roles/mflux/files/licence-staging.xml

• roles/nginxplus/files/conf/http/dev/templates/rapid7.conf

• roles/nginxplus/files/conf/http/dev/templates/rate-limit-allow-list.conf

• roles/nginxplus/files/conf/http/dev/templates/restrict.conf

• roles/nginxplus/files/conf/http/templates/htc_restrict.conf

• roles/nginxplus/files/conf/http/templates/libnet.conf

• roles/nginxplus/files/conf/http/templates/rapid7.conf

• roles/nginxplus/files/conf/http/templates/rate-limit-allow-list.conf

• roles/nginxplus/files/conf/http/templates/restrict.conf

• roles/nginxplus/files/license/nginx-repo.crt

• roles/nginxplus/files/license/nginx-repo.jwt

• roles/nginxplus/files/license/nginx-repo.key

• roles/nginxplus/files/ssl/cicognara_org_priv.key

• roles/obsd_httpd/files/conf/restrict.conf

• roles/ojs/files/private/ojs-prod1.princeton.edu_priv.key

• roles/ojs/files/private/ojs-prod2.princeton.edu_priv.key

• roles/ojs/files/private/ojs-staging1.princeton.edu_priv.key

• roles/ojs/files/private/ojs-staging2.princeton.edu_priv.key

• roles/pas/files/license.key

• roles/pas/files/pas.smb.credentials

• roles/pulmap/files/maplab.smb.credentials

• roles/pulmap/files/production-google_cloud_credentials.json

• roles/pulmap/files/staging-google_cloud_credentials.json

• roles/pulmirror/files/pulmirror_princeton_edu_priv.key

• roles/shared_data/files/shared_data.smb.credentials

• roles/solrcloud/files/solr.smb.credentials

• roles/studio_proc/files/archives.smb.credentials

• roles/studio_proc/files/archives_bd.smb.credentials

• roles/studio_proc/files/bluemountain.smb.credentials

• roles/studio_proc/files/ddd.smb.credentials

• roles/studio_proc/files/historicalperiodicals.smb.credentials

• roles/studio_proc/files/ingest_scratch.smb.credentials

• roles/studio_proc/files/libimages.smb.credentials

• roles/studio_proc/files/princetonperiodicals.smb.credentials

• roles/studio_proc/files/pudl.smb.credentials

• roles/studio_proc/files/pul_store.smb.credentials

• roles/studio_proc/files/studio.new.smb.credentials

• roles/video_reserves/files/id_rsa

• roles/video_reserves/files/private/lib-vr-prod1.princeton.edu_priv.key

• roles/video_reserves/files/private/lib-vr-staging1.princeton.edu_priv.key

• keys/cicognara_org_priv.key

• keys/dataspace-dev_princeton_edu_priv.key

• keys/dataspace-staging_princeton_edu_priv.key

• keys/dataspace_princeton_edu_priv.key

• keys/oar-dev_princeton_edu_priv.key

• keys/oar-staging_princeton_edu_priv.key

• keys/oar_princeton_edu_priv.key

• keys/oaworkflow-dev_princeton_edu_priv.key

• keys/oaworkflow_princeton_edu_priv.key

• keys/pulfleet_pulcloud_io.csr

• keys/pulfleet_pulcloud_io.key

• keys/pulmirror_princeton_edu_priv.key

• keys/thesis-central_princeton_edu_priv.key

• keys/tigris_princeton_edu_priv.key

Tasks:

1. Generate a New Vault Password:

• Create a strong, unique password for the Ansible vaults.
• Securely store the new password in our password manager

2. Re-key Existing Vaults:

• Use the ansible-vault rekey command to update each affected vault file with the new password.
  Example:

 ansible-vault rekey group_vars/all/vault.yml --new-vault-password-file /path/to/new_vault_password_file

3. Update Playbooks (if necessary):

• Review playbooks to see if the vault password is used directly (this should ideally be avoided).
• If the vault password is used in any playbooks, update them to use a secure method like --vault-password-file or environment variables.

4. Re-run Playbooks:

• Identify all playbooks that utilize the affected vaults.
• Re-run these playbooks in a controlled manner (e.g., starting with a staging environment, if applicable) to ensure encrypted variables are correctly updated.

[hackartisan]

@kayiwa can you take a look at pulibrary/pul-it-handbook#359 ? is this still what / where we want that? I was going off of notes on #3854.