connection_validator not recovering from errors

[HoneyryderChuck]

Hi 👋 ,

I'm using the "connection_validator" plugin on this AWS setup, where the connection from the ruby processes to the database has an SSL proxy in the middle. My naive interpretation of this plugin was that, if a certain class of errors would happen, the connection would be thrashed and replaced by a freshly initiated connection, which is the use case I'd be most looking for (the SSL proxy frequently drops down connections, and reports it back as timed out, errors, or others).

Today we had an incident, where such an error brought the workers down:

PQconsumeInput() SSL connection has been closed unexpectedly (PG::ConnectionBad)

This error was not handled by the connection validator and bubbled up. This led me to investigate why the plugin didn't handle it, and from a quick glance, the plugin is handling only timeouts.

Is my assumption correct? If so, what's the best way to implement the recovery strategy that would allow me to recover from such errors, or is there a plugin for that? Or am I wrong, and the "connection_validator" plugin should have worked, but I'm missing some configuration?

[HoneyryderChuck]

I've looked into this a bit further, and am applying the DB.pool.connection_validation_timeout = -1 patch suggested in the docs, which is probably what caused the incident (SSL proxy may terminate connections during traffic surges, regardless of liveness).

It was a bit unclear to me why such a lenient timeout (1 hour seems very long) was chosen as the default, but then again, my judgement may be clouded by today's experience, and the documentation was pretty clear about the alternative (I just overlooked it).

[jeremyevans]

I suppose 1 hour may be too long as a default. That's something we could probably consider changing in Sequel 6. I definitely don't want -1 as a default, as that can have a significant negative performance effect. But maybe 300 would be a better default than 3600.

[HoneyryderChuck]

Agreed that -1 shouldn't be a default.