Error handling; don't panic. #61
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- feat: added done and error channels in sparrow - feat: added global shutdown function for main components - chore: edited most functions to return errors and not panic
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
puffitos
requested review from
y-eight,
niklastreml,
lvlcn-t and
eumel8
as code owners
puffitos
added
feature
Signed-off-by: Bruno Bressi <[email protected]>
|
Ready for review. |
Signed-off-by: Bruno Bressi <[email protected]>
niklastreml
reviewed
Jan 12, 2024
Signed-off-by: Bruno Bressi <[email protected]>
Signed-off-by: Bruno Bressi <[email protected]>
lvlcn-t
reviewed
Jan 14, 2024
Signed-off-by: Bruno Bressi <[email protected]>
niklastreml
approved these changes
Jan 16, 2024
lvlcn-t
approved these changes
Jan 16, 2024
y-eight
approved these changes
Jan 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Addresses #53
Still a WIP. I just wanted to put everything together as documentation so we can discuss this together next week.
Changes
A friendly guide for the changes:
Runand write their errors into the error channel (when something irrecoverable happens)handleErrors), which may be extended to handle errors in other ways (currently only fatal errors are expected in the error channel)handleErrorswill gracefully shutdown all sparrow components (if possible) and return the error(s) which led to the shutdown, along with other errors that happened during shutdown.Additionally, I couldn't resist the urge and did the following (sorry!):
withGroupfrom all loggersTests done
Normal run, metrics from remote config
Cancel context after 30 secs
A timeout context was set in the top-level sparrow.Run(ctx) call. After running for a bit, we get the following logs. Note the many errors in the various checks, the api shutdown and the target manager shutdown. Finally, sparrow exits with 1 as it should :)
{"time":"2024-01-05T18:54:15.235186838+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).FetchFiles","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/gitlab/gitlab.go","line":147},"msg":"Successfully fetched all target files","files":8} {"time":"2024-01-05T18:54:16.622747226+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/config.(*HttpLoader).Run","file":"/home/bbressi/dev/repos/sparrow/pkg/config/http.go","line":71},"msg":"Successfully got remote runtime configuration"} {"time":"2024-01-05T18:54:20.957935964+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).handleErrors","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/run.go","line":330},"msg":"Context done, shutting down sparrow"} {"time":"2024-01-05T18:54:20.957941044+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/checks.getLatency","file":"/home/bbressi/dev/repos/sparrow/pkg/checks/latency.go","line":274},"msg":"Error while checking latency","url":"https://google.com/","error":"Get \"https://www.google.com/\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959002788+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).registerCheck.func1","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/run.go","line":257},"msg":"Failed to run check","name":"health","error":"context deadline exceeded"} {"time":"2024-01-05T18:54:20.959042803+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/checks.getLatency","file":"/home/bbressi/dev/repos/sparrow/pkg/checks/latency.go","line":274},"msg":"Error while checking latency","url":"https://example.com/","error":"Get \"https://example.com/\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959082609+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).fetchFile","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/gitlab/gitlab.go","line":171},"msg":"Failed to fetch file","file":"sparrow-dev-cool.de.json","error":"Get \"https://gitlab.devops.telekom.de/api/v4/projects/237078/repository/files/sparrow-dev-cool.de.json/raw?ref=main\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959101665+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).FetchFiles","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/gitlab/gitlab.go","line":142},"msg":"Failed fetching files","error":"Get \"https://gitlab.devops.telekom.de/api/v4/projects/237078/repository/files/sparrow-dev-cool.de.json/raw?ref=main\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959088671+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/checks.getLatency","file":"/home/bbressi/dev/repos/sparrow/pkg/checks/latency.go","line":274},"msg":"Error while checking latency","url":"https://yam.telekom.de","error":"Get \"https://yam-united.telekom.com/pages\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959117124+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).refreshTargets","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/targets/gitlab.go","line":203},"msg":"Failed to update global targets","error":"Get \"https://gitlab.devops.telekom.de/api/v4/projects/237078/repository/files/sparrow-dev-cool.de.json/raw?ref=main\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959044988+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/checks.getLatency","file":"/home/bbressi/dev/repos/sparrow/pkg/checks/latency.go","line":274},"msg":"Error while checking latency","url":"https://gitlab.devops.telekom.de","error":"Get \"https://gitlab.devops.telekom.de/users/sign_in\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959173922+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).Shutdown","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/targets/gitlab.go","line":149},"msg":"Stopping gitlab reconciliation routine"} {"time":"2024-01-05T18:54:20.959185023+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/checks.(*Latency).Run","file":"/home/bbressi/dev/repos/sparrow/pkg/checks/latency.go","line":90},"msg":"Context canceled","err":"context deadline exceeded"} {"time":"2024-01-05T18:54:20.959235329+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).registerCheck.func1","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/run.go","line":257},"msg":"Failed to run check","name":"latency","error":"context deadline exceeded"} {"time":"2024-01-05T18:54:20.959233375+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).api.func1","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/api.go","line":86},"msg":"Failed to serve api","error":"http: Server closed"} {"time":"2024-01-05T18:54:20.959139457+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).Reconcile","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/targets/gitlab.go","line":99},"msg":"Failed to get global targets","error":"Get \"https://gitlab.devops.telekom.de/api/v4/projects/237078/repository/files/sparrow-dev-cool.de.json/raw?ref=main\": context deadline exceeded"} {"time":"2024-01-05T18:54:20.959275775+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).handleErrors","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/run.go","line":335},"msg":"Error in sparrow component, shutting down","error":"context deadline exceeded"} {"time":"2024-01-05T18:54:20.959285975+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).Reconcile","file":"/home/bbressi/dev/repos/sparrow/pkg/sparrow/targets/gitlab.go","line":94},"msg":"Gitlab target reconciliation ended"} {"time":"2024-01-05T18:54:20.95933082+01:00","level":"ERROR","source":{"function":"github.com/caas-team/sparrow/cmd.NewCmdRun.run.func1","file":"/home/bbressi/dev/repos/sparrow/cmd/run.go","line":123},"msg":"Error while running sparrow","error":"sparrow was shut down"}TODO