[Support] Queens University Hub Outage

[yuvipanda]

The Freshdesk ticket link

https://2i2c.freshdesk.com/a/tickets/2645

Ticket request type

Something is not working

Ticket impact

🟥 Critical

Short ticket description

New nodes don't spin up, so no new users can log in

(Optional) Investigation results

No response

[yuvipanda]

First I couldn't login, but this was because I hadn't set up MFA initially. That was quickly resolved by Queens IT, and we could continue.

[yuvipanda]

It looks like an upscaling operation on the existing user pool is 'stuck' somewhere, and nothing can happen on the cluster. I tried creating a new nodepool, which got stuck too. Same for trying to manually scale them.

I was able to abort the 'operation' with az aks nodepool operation-abort --nodepool-name usere8sv5 --cluster-name hub-cluster --resource-group 2i2c-jupyterhub-prod let's see if that helps

[yuvipanda]

I created a VM with a public ip just to test that works, and it does. So this isn't some new security policy (at least on the surface).

I'm trying to now 'unwedge' this.

[yuvipanda]

I tried to change the tier, but that too got stuck in 'updating' for far too long (~30min). Cancelling, and trying to upgrade master instead. If that doesn't get it unstuck, I'll bounce things back, as this now needs Azure support to be involved.

[yuvipanda]

ya, the cluster is wedged af. I've responded in https://2i2c.freshdesk.com/a/tickets/2645 to ask them to involve Azure Support.

[yuvipanda]

As a reminder, when something like this happened with UToronto a few years ago, things were down for like 48h until we just gave up and moved to CILogon