Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return the set of metrics that are over desired thresholds rather than just the first one #30

Open
himmat62 opened this issue Apr 15, 2016 · 4 comments

Comments

@himmat62
Copy link

I need to generate multiple warnings/critical's in one run of check (for checking the disk-usage of 20 machines) by using check-graphite-data.rb  and send it to mailer handler.

As of now i am getting only single output and mail also working.But my requirement is getting multiple(20 machines ,which matches given threshold values ) warnings/critical's.

I have followed link: https://sensuapp.org/docs/0.21/getting-started-with-checks  for setting alerts.
i ran below script to check threshold :
ruby check-graphite-data.rb -s localhost:8080 -t 'highestCurrent(internal.dev3.*.1.disk-usage.root.used,6)' -w 180 -c 20000 -a 120

send(type, "#{@value['target']} has passed #{type} threshold (#{@data.last})") if below?(type) || above?(type)
So, instead of above method, i tried below method because in above method next loop is not running to get multiple warnings/critical's.:
if below?(type) || above?(type)
puts "CheckGraphiteData #{type} : #{@value['target']} has passed #{type} threshold (#{@data.last})"
end

But still issue persists.Please suggest me feasible solution.

@smitchelus
Copy link

I have this exact same issue. Trying to figure out a solution. As I read the code it appears to me that the intent is to loop through the returned matches from Graphite and evaluate each of them, but it seems like the only alert I get is from the first match returned by Graphite.

@smitchelus
Copy link

The code I was looking at that made me believe that it might report for each match is:

  results = proxy.retrieve_data!
  results.each_pair do |_key, value|
    @value = value
    @data = value['data']
    check_age || check(:critical) || check(:warning)
  end

Followed by:

def check(type)
if config[type]
send(type, "#{@value['target']} has passed #{type} threshold (#{@data.last})") if below?(type) || above?(type)
end
end

I haven't had time to add any logging to see why this doesn't work though. I wonder if that send() method is throwing an exception or something which breaks the loop.

@majormoses
Copy link
Member

We could change this logic to loop over them and append them to an array or hash and then make a check after evaluating each of them which is a common approach in other plugins.

@majormoses
Copy link
Member

Would you be willing to attempt a pr for this?

@majormoses majormoses changed the title Generate multiple warnings/critical's in one run of check. Return the set of metrics that are over desired thresholds rather than just the first one Jun 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants