Ingress routes to incorrect podIP

[stingerpk]

Is this a BUG or FEATURE REQUEST?:
Bug

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?:
Yes

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version: 0.7.1
kubectl version: v1.9.6-gke.1

Is Istio Auth enabled or not ?

Not using auth
Used standard Istio 0.7.1 scripts with automatic sidecar injection
The ingress is used with HTTPS

What happened:
We have a cluster with a lot of services. This doesn't always happen, but is frequent enough. Ingress returns 503 error, browser displays: upstream connect error or disconnect/reset before headers.

Output of curl http://istio-pilot:15003/v1/registration contains these lines:

{
   "service-key": "testing-client.app.svc.cluster.local|http-testing-c",
   "hosts": [
    {
     "ip_address": "10.4.3.28",
     "port": 80
    }
   ]
  },

Pod YAML:
podIP: 10.4.3.28
hostIP: 10.132.0.11

Service YAML:
clusterIP: 10.7.255.54

However, the ingress somehow doesn't route to the correct IP addresses for host and pod and tries routing the requests to some other IP:

"GET / HTTP/2" 503 UF 0 57 1004 - "10.132.0.9" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" "eb88831b-5569-9d52-98af-10d574882fd7" "testing-client.website.com" "10.4.4.247:80"

So basically the pilot has the right IP address for the service but ingress is routing to incorrect IP.

Restarting pilot pods fix the issue temporarily. Restarting any other components or the service has no effect.

What you expected to happen:
Ingress should always route to correct IP.

How to reproduce it:
No idea, seems to happen randomly. Some deployments go alright, others don't. The manifests for deploying the app are pretty standard and are deployed through helm.

Feature Request:
N

[ku-s-h]

I am also facing the same issue. Istio ingress routes a small part of the traffic (0.01 ops) to incorrect pods.

[louiscryan]

Is this still an issue with 0.8. There were a bunch of improvements in that release that might have resolved this issue.