Postgresql migration epic

[acozine]

Component parts:

• all projects should run on new postgresql servers (migrate from lib-postgres3 and lib-postgres-prod1 to the new replicated servers, lib-postgres-prod3 and its replicated box lib-postgres-prod2)
• all projects should upgrade from psql 10 to psql 13 (does this involve code changes?)

Check off when the project has been migrated to the new servers and to psql 13 if necessary. Projects that run on postgres:

• ansible-tower
• approvals [lib-postgres3, pgsql 10]
• approvals-staging [lib-postgres3, pgsql 10]
• bibdata [lib-postgres-prod, pgsql 13]
• cicognara [lib-postgres3, pgsql 10]
• cicognara-staging [lib-postgres3, pgsql 10]
• dpul [lib-postgres3, pgsql 10]
• dpul-staging [lib-postgres3, pgsql 10]
• dss [lib-postgres3, pgsql 10]
• dss-staging [lib-postgres3, pgsql 10]
• figgy [lib-postgres3, pgsql 10]
• figgy-staging [lib-postgres3, pgsql 10]
• geaccirc [lib-postgres3, pgsql 10]
• gscatalog/geoserver (investigate)
• lae [lib-postgres3, pgsql 10]
• lae-staging [lib-postgres3, pgsql 10]
• libjobs [lib-postgres3, pgsql 10]
• lib-jobs-staging [lib-postgres-staging1, pgsql 13]
• libstatistics [lib-postgres3, pgsql 10]
• libstatistics-staging [lib-postgres-staging1, pgsql 13]
• lockers and study spaces [lib-postgres3, pgsql 10]
• lockers and study spaces staging [lib-postgres3, pgsql 10]
• mudd [lib-postgres3, pgsql 10]
• mudd-staging [lib-postgres3, pgsql 10]
• ojs [lib-postgres-prod1, pgsql 13]
• ojs-staging [lib-postgres-staging1, pgsql 13]
• openmonograph
• openmonograph-staging
• ouranos-staging [lib-postgres3, pgsql 10]
• pulfalight [lib-postgres3, pgsql 10]
• pulfalight-staging [lib-postgres3, pgsql 10]
• pulmap [lib-postgres3, pgsql 10]
• pulmap-staging [lib-postgres3, pgsql 10]
• researchdata

Related postgres tickets include #2210 #1522 #2305 #2405 #2444 #2464.
Edit: make the list a checklist.

[jrgriffiniii]

pdc_discovery may not need to be migrated, but this must be verified.

[acozine]

We prioritized migrating services that were connecting to postgres on lib-postgres-prod1, which we will retire. All services have migrated EXCEPT the CDH services as of today. Once that goal is met, we will migrate the rest of the services, project by project.

Some projects have staging running on postgresql 13 while production is still running on postgresql 10. These will require extra testing before we migrate and upgrade production.

[acozine]

The Ansible Tower database was moved to its own database server running Rocky Linux, see #5606.
The cicognara project was migrated to a static site, see #5570.
So the only remaining item is gscatalog/geoserver.

[acozine]

We migrated gscatalog/geoserver to a lambda on AWS.

[acozine]

Remaining work is documented in #4383.