Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Timeout handling and improves obscuring of DNS failure messages being swallowed by the base timeout (circuit breaker) #123

Merged
merged 5 commits into from
Feb 17, 2019
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,5 @@ mkmf.log
#Intellij files
*.iml
*.idea

vendor/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ This project adheres to [Semantic Versioning](http://semver.org/).
This CHANGELOG follows the format listed [here](https://github.com/sensu-plugins/community/blob/master/HOW_WE_CHANGELOG.md)

## [Unreleased]
### Changed
- `check-http.rb`: Add options to set open-timeout and read-timeout for Net:HTTP. Improve output on what Net::HTTP timeout was encountered.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally I'd like to see the option names being called out here so someone reading this knows exactly what to update to leverage.

For example:

## [Unreleased]
### Added
- `check-http.rb`: Add options to set `--open-timeout` and `--read-timeout` for Net:HTTP. Additionally rescue `Net::OpenTimeout` and `Net::ReadTimeout` exception classes (@mygithubuser)
- `check-http.rb`: exposed `--dns-timeout` and rescue on `Resolv::ResolvTimeout` exception class (@mygithubuser)

### Changed
- `check-http.rb`: switched to using rubies DNS resolver (which looks at `/etc/resolv.conf`) and rescue `Resolv::ResolvError` exception class for generic DNS failures (@mygithubuser)

- `check-http.rb`: Use ruby DNS resolver, and set DNS resolution timeout.

## [4.0.0] - 2018-12-17
### Breaking Changes
Expand Down
28 changes: 25 additions & 3 deletions bin/check-http.rb
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
require 'net/http'
require 'net/https'
require 'digest'
require 'resolv-replace'

#
# Check HTTP
Expand Down Expand Up @@ -165,7 +166,19 @@ class CheckHttp < Sensu::Plugin::Check::CLI
short: '-t SECS',
long: '--timeout SECS',
proc: proc(&:to_i),
description: 'Set the timeout',
description: 'Set the total execution timeout in seconds',
default: 15

option :open_timeout,
long: '--open-timeout SECS',
proc: proc(&:to_i),
description: 'Number of seconds to wait for the connection to open',
default: 15

option :read_timeout,
long: '--read-timeout SECS',
proc: proc(&:to_i),
description: 'Number of seconds to wait for one block to be read',
default: 15

option :redirectok,
Expand Down Expand Up @@ -250,10 +263,19 @@ def run
config[:port] ||= config[:ssl] ? 443 : 80
end

# Use Ruby DNS Resolver and set DNS resolution timeout to 800ms
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update comment when you expose it as a var.

dns_resolver = Resolv::DNS.new
dns_resolver.timeouts = 0.8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya let's expose that as an option so its configurable and make sure to cast it as a float.

Resolv::DefaultResolver.replace_resolvers([dns_resolver])

begin
Timeout.timeout(config[:timeout]) do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside acquire_resource below we should use this to still provide a way for people to account for slow overloaded web servers.

Copy link
Contributor Author

@johanek johanek Jan 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timeout options to Net::HTTP are already being set to the timeout value passed as config:

http.read_timeout = config[:timeout]
http.open_timeout = config[:timeout]
http.ssl_timeout = config[:timeout]
http.continue_timeout = config[:timeout]
http.keep_alive_timeout = config[:timeout]

acquire_resource
end
rescue Net::OpenTimeout
critical 'Request timed out opening connection'
rescue Net::ReadTimeout
critical 'Request timed out reading data'
rescue Timeout::Error
critical 'Request timed out'
rescue StandardError => e
Expand All @@ -279,8 +301,8 @@ def acquire_resource
else
http = Net::HTTP.new(config[:host], config[:port])
end
http.read_timeout = config[:timeout]
http.open_timeout = config[:timeout]
http.read_timeout = config[:read_timeout]
http.open_timeout = config[:open_timeout]
http.ssl_timeout = config[:timeout]
http.continue_timeout = config[:timeout]
http.keep_alive_timeout = config[:timeout]
Expand Down