Traffic split across clusters not working as expected

[Vamsi0473]

Hi Team, I have following scenario
Service A is deployed in internal EKS clusters located in both the US and EU regions.
Multi-cluster communication is established between both internal clusters.
A traffic split is configured such that if Service A in the US internal cluster becomes unavailable, traffic is redirected to Service A running in the EU internal cluster.
External Exposure:
Service A is exposed through the Kong gateway, which is running in external clusters in both the US and EU regions.
Services from the internal clusters (US & EU) are mirrored to the corresponding external clusters.
Configuration Details:
Service A is configured in the Kong gateway using the service DNS format {servicename}.{internalclustername}-{namespace}. This setup allows the Kong gateway to proxy requests from the external to the internal cluster.
Expected Behavior:
Under normal conditions, when Service A is operational in the US internal cluster, calls from the Kong gateway in the external cluster are routed to Service A in the US internal cluster.
If Service A in the US internal cluster is down, the traffic split configuration should redirect the calls to Service A running in the EU internal cluster.
But this is not happening we are receiving 500 error similar setup when services and traffic splits running and configured in external clusters they are working fine.
Can you please let me know what is causing this issue

[adleong]

Hi @Vamsi0473

Traffic splits use fixed weighting for distributing traffic. This means that each backend service will receive a portion of the traffic proportional to its weight. If one of the backend services is down, the same portion of traffic will still be sent to that backend service unless its weight is adjusted.
https://github.com/servicemeshinterface/smi-spec/blob/main/apis/traffic-split/v1alpha4/traffic-split.md#specification

[Vamsi0473]

Hi @adleong I understand the concept, but my question is about a service deployed in internal clusters located in the EU and US regions, with traffic splits configured to perform a failover if the service goes down in one of the regions. This service is mirrored to external clusters in both the EU and US regions using Linkerd multicluster and is exposed to end users via a gateway. The host of the service is {servicename}-{internalclustername}.{namespace}.

When users hit the US gateway endpoint for this service, and if the service is not running in the internal US cluster, our expectation is that the request should be redirected from the external US cluster to the internal EU cluster. However, this is not happening. Despite the traffic split switching the traffic weight to the internal EU cluster, we are seeing 500 server errors because the external US cluster treats the service in US internal cluster as not working.