Change remote write behaviour in the case of additional external endpoints

[philipgough]

Related to https://issues.redhat.com/browse/ACM-6685

Return main upstream response when using multiple endpoints

This changes refactors the Proxy remote write handler to do
as much work outside of the handler as possible.

There is a major change in behaviour.
The extra endpoints are now written to async in the background of
the main request. Extra endpoints will be written to with best
effort and logged.

The response code from the thanos backend will be returned to the
client.

This will cause the client to retry on specific errors and will
have a knock on effect of a successfully written request prior
being attempted once again to the extra remote write endpoints.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: philipgough

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [philipgough]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sonarqubecloud]

Quality Gate failed

Failed conditions
B Security Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[openshift-ci]

@philipgough: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/pr-image-mirror	19a4ac1	link	true	/test pr-image-mirror
ci/prow/images	19a4ac1	link	true	/test images
ci/prow/sonar-pre-submit	19a4ac1	link	true	/test sonar-pre-submit
ci/prow/test	19a4ac1	link	true	/test test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[philipgough]

need to add more cases here

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.