Be more resilient against loss of internet connection

[czepluch]

Abstract

If internet connection is lost just briefly on the machine running a Raiden node, the node crashes. This is not an ideal user experience and it makes it harder to guarantee uptime.

Motivation

The user should worry as little as possible about running a Raiden node. If it crashes from just a quick dropped internet connection the user has to worry about running on a very stable connection. Back in the days Raiden handled this more gracefully and the user could even put his laptop to sleep and resume afterwards without the node crashing.

Specification

Allow Raiden to be more graceful when connection is lost instead of crashing the node.

[fredo]

Complex Problem because state machine must stop if one service is not available.

Soft measurements:

• Add retries to the http requests
• Send notification what happened
• restart-on-error flag with potebntial error emssage on next restart.

[fredo]

closed in favor of #7186

[ulope]

One more option we discussed:

• Add a soft-restart feature that would allow the client to self restart from the outer layer (cli) in case of an exception, removing the need for the user to setup a process manager