nats streaming - client connection fails (timeout) #242

mmanna-sapfgl · 2019-04-17T15:11:32Z

I have started a nats streaming server using the attached conf file and the below docker command:

docker run --rm -ti -p 4223:4223 -p 4222:4222 -p 6223:6223 -p 8223:8223 -p 6222:6222 --name gnatsd-streamer --net host -v /home/docker/streamer_conf.conf:/streamer_conf.conf -v /home/docker/streamer1:/streamer1 nats-streaming:latest -p 4223 -m 8223 -c /strea
mer_conf.conf

But Upon stan-bench startup, I am getting connection timeout (tried with telnet first to confirm if the connection works):

.\stan-bench.exe -s nats://192.168.99.100:4223 -qgroup test1 -np 1 -ns 1 -n 10 -ms 7212212 -csv 10x10x100kx7m_result.csv foo

Subscriber benchmark-sub-0 can't connect: stan: connect request timeout

i am running it on Single Node, so no cluster info is required. I couldn't see any connection issue on the docker logs, so this issue is slightly unusual for me.

kozlovic · 2019-04-17T15:28:10Z

As explained on slack, you have set a cluster name in your server, so you need to use that same cluster name in the client when connecting. With the bench program, that would be -c <cluster name>. Note that the message size is above the default limit, so you need to ensure that you have increased the max payload on all NATS servers that will be part of your cluster to allow for such size. Note that it is not recommended on NATS to use big payloads.

mmanna-sapfgl · 2019-04-17T16:09:00Z

PS C:\Users\XXXX\go\src\github.com\nats-io\go-nats-streaming\examples\stan-bench> .\stan-bench.exe -c streamer1 -s nats://192.168.99.100:4223 -qgroup test1 -np 1 -ns 1 -n 10 -ms 7212212 -csv 10x10x100kx7m_result.csv foo
Subscriber benchmark-sub-0 can't connect: stan: connect request timeout

I am still getting the error after using -c streamer1

kozlovic · 2019-04-17T16:22:59Z

Can you telnet the server machine from your client machine? By that I mean you should be able to telnet the machine and NATS Streaming port and get an INFO protocol from the embedded NATS server. (do you use external NATS or is it embedded)?

mmanna-sapfgl · 2019-04-17T22:04:42Z

Hey @kozlovic - yes I did the telnet test before posting. It does return the JSON with all info correctly. I also hit the url on browser and it returns the raw text same as telnet. And I am running docker on windows using Virtual Box. I am using Windows 10 and no external NATS. My intention is to run ants-streaming container (no connection to external NATS server).

kozlovic · 2019-04-17T22:08:06Z

Please post your config file..

kozlovic · 2019-04-17T22:34:18Z

Are you sure about how you start the docker image? As described here, for Windows images if you pass any argument, you actually need to specify the executable name, like so:

$ docker run -p 4223:4223 -p 8223:8223 nats-streaming nats-streaming-server -p 4223 -m 8223

From the command that you posted, it does not look like this is what you are doing.
Also if I try with --net host I get the error reported here. Not sure how you are doing this.

When I run a docker image of nats streaming for Windows, I usually do this (I run from macOS):

docker run --rm -ti -p4223:4222 nats-streaming

then my client uses the IP of the VM and 4223 for the port. But the server itself runs on port 4222 in the container. If I want it to run on 4223, I would have to export a different port on my host. Like so:

docker-run --rm -ti -p 4224:4223 nats-streaming nats-streaming-server -p 4223

that makes the container listen on port 4223 but then client needs to use port 4224.

mmanna-sapfgl · 2019-04-17T22:44:22Z

streamer_conf.conf.txt

mmanna-sapfgl · 2019-04-17T22:46:19Z

Are you sure about how you start the docker image? As described here, for Windows images if you pass any argument, you actually need to specify the executable name, like so:
$ docker run -p 4223:4223 -p 8223:8223 nats-streaming nats-streaming-server -p 4223 -m 8223
From the command that you posted, it does not look like this is what you are doing.
Also if I try with --net host I get the error reported here. Not sure how you are doing this.

When I run a docker image of nats streaming for Windows, I usually do this (I run from macOS):
docker run --rm -ti -p4223:4222 nats-streaming
then my client uses the IP of the VM and 4223 for the port. But the server itself runs on port 4222 in the container. If I want it to run on 4223, I would have to export a different port on my host. Like so:
docker-run --rm -ti -p 4224:4223 nats-streaming nats-streaming-server -p 4223
that makes the container listen on port 4223 but then client needs to use port 4224.

kozlovic · 2019-04-17T22:48:47Z

Ok, the problem is that you specify discover_prefix: \"discover\" in your configuration file, but don't use that in your client! The default discovery prefix is _STAN.discover and the client library uses that by default. If you want to change it, you need to have both servers and clients use the same. Remove this setting from your config and it should work.

mmanna-sapfgl · 2019-04-17T22:59:01Z

Are you sure about how you start the docker image? As described here, for Windows images if you pass any argument, you actually need to specify the executable name, like so:
$ docker run -p 4223:4223 -p 8223:8223 nats-streaming nats-streaming-server -p 4223 -m 8223
From the command that you posted, it does not look like this is what you are doing.
Also if I try with --net host I get the error reported here. Not sure how you are doing this.

When I run a docker image of nats streaming for Windows, I usually do this (I run from macOS):
docker run --rm -ti -p4223:4222 nats-streaming
then my client uses the IP of the VM and 4223 for the port. But the server itself runs on port 4222 in the container. If I want it to run on 4223, I would have to export a different port on my host. Like so:
docker-run --rm -ti -p 4224:4223 nats-streaming nats-streaming-server -p 4223
that makes the container listen on port 4223 but then client needs to use port 4224.

No its incorrect. This command will be rejected by docker saying "invalid command nets-streaming-server". I have just tried with MacOS on my person machine - it's the same error.

kozlovic · 2019-04-17T23:05:17Z

No its incorrect. This command will be rejected by docker saying "invalid command nets-streaming-server". I have just tried with MacOS on my person machine - it's the same error.

My host is macOS and the docker image is the NATS Streaming image nano server 0.12.2. If I try to pass an argument without the executable name:

IvanMBP:~ ivan$ docker run --rm -ti -p 4224:4223 nats-streaming:0.12.2 -p 4223
docker: Error response from daemon: container 23abbd8ac35f3f795c2e763f0c8e1e5c6d534e3f8c946d2f1ef0568f5b9cb93e encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {"ApplicationName":"","CommandLine":"-p 4223","User":"","WorkingDirectory":"C:\\nats-streaming-server","Environment":{"NATS_DOCKERIZED":"1"},"EmulateConsole":true,"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":false,"ConsoleSize":[0,0]}.
IvanMBP:~ ivan$

If I pass the executable name:

IvanMBP:~ ivan$ docker run --rm -ti -p 4224:4223 nats-streaming:0.12.2 nats-streaming-server -p 4223

[1648] 2019/04/09 18:26:18.469083 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.12.2
[1648] 2019/04/09 18:26:18.471960 [INF] STREAM: ServerID: iS73mM2UNh1T8Rn0xGAwuK
[1648] 2019/04/09 18:26:18.471960 [INF] STREAM: Go version: go1.11.6
[1648] 2019/04/09 18:26:18.473940 [INF] STREAM: Git commit: [4489c46]
[1648] 2019/04/09 18:26:18.473940 [INF] Starting nats-server version 1.4.1
[1648] 2019/04/09 18:26:18.476851 [INF] Git commit [3e64f0b]
[1648] 2019/04/09 18:26:18.477865 [INF] Listening for client connections on 0.0.0.0:4223
[1648] 2019/04/09 18:26:18.478805 [INF] Server is ready
[1648] 2019/04/09 18:26:18.510096 [INF] STREAM: Recovering the state...
[1648] 2019/04/09 18:26:18.514926 [INF] STREAM: No recovered state
[1648] 2019/04/09 18:26:18.772746 [INF] STREAM: Message store is MEMORY
[1648] 2019/04/09 18:26:18.772746 [INF] STREAM: ---------- Store Limits ----------
[1648] 2019/04/09 18:26:18.774136 [INF] STREAM: Channels:                  100 *
[1648] 2019/04/09 18:26:18.774136 [INF] STREAM: --------- Channels Limits --------
[1648] 2019/04/09 18:26:18.775705 [INF] STREAM:   Subscriptions:          1000 *
[1648] 2019/04/09 18:26:18.775705 [INF] STREAM:   Messages     :       1000000 *
[1648] 2019/04/09 18:26:18.778603 [INF] STREAM:   Bytes        :     976.56 MB *
[1648] 2019/04/09 18:26:18.778603 [INF] STREAM:   Age          :     unlimited *
[1648] 2019/04/09 18:26:18.782508 [INF] STREAM:   Inactivity   :     unlimited *
[1648] 2019/04/09 18:26:18.782508 [INF] STREAM: ----------------------------------

I wrote the docker file and instructions on Docker hub..

kozlovic · 2019-04-17T23:12:19Z

Here the docker file used to create the official image: https://github.com/nats-io/nats-streaming-docker/blob/master/windows/nanoserver/Dockerfile

mmanna-sapfgl · 2019-04-17T23:13:49Z

No its incorrect. This command will be rejected by docker saying "invalid command nets-streaming-server". I have just tried with MacOS on my person machine - it's the same error.

My host is macOS and the docker image is the NATS Streaming image nano server 0.12.2. If I try to pass an argument without the executable name:

IvanMBP:~ ivan$ docker run --rm -ti -p 4224:4223 nats-streaming:0.12.2 -p 4223
docker: Error response from daemon: container 23abbd8ac35f3f795c2e763f0c8e1e5c6d534e3f8c946d2f1ef0568f5b9cb93e encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {"ApplicationName":"","CommandLine":"-p 4223","User":"","WorkingDirectory":"C:\\nats-streaming-server","Environment":{"NATS_DOCKERIZED":"1"},"EmulateConsole":true,"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":false,"ConsoleSize":[0,0]}.
IvanMBP:~ ivan$

If I pass the executable name:

IvanMBP:~ ivan$ docker run --rm -ti -p 4224:4223 nats-streaming:0.12.2 nats-streaming-server -p 4223

[1648] 2019/04/09 18:26:18.469083 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.12.2
[1648] 2019/04/09 18:26:18.471960 [INF] STREAM: ServerID: iS73mM2UNh1T8Rn0xGAwuK
[1648] 2019/04/09 18:26:18.471960 [INF] STREAM: Go version: go1.11.6
[1648] 2019/04/09 18:26:18.473940 [INF] STREAM: Git commit: [4489c46]
[1648] 2019/04/09 18:26:18.473940 [INF] Starting nats-server version 1.4.1
[1648] 2019/04/09 18:26:18.476851 [INF] Git commit [3e64f0b]
[1648] 2019/04/09 18:26:18.477865 [INF] Listening for client connections on 0.0.0.0:4223
[1648] 2019/04/09 18:26:18.478805 [INF] Server is ready
[1648] 2019/04/09 18:26:18.510096 [INF] STREAM: Recovering the state...
[1648] 2019/04/09 18:26:18.514926 [INF] STREAM: No recovered state
[1648] 2019/04/09 18:26:18.772746 [INF] STREAM: Message store is MEMORY
[1648] 2019/04/09 18:26:18.772746 [INF] STREAM: ---------- Store Limits ----------
[1648] 2019/04/09 18:26:18.774136 [INF] STREAM: Channels:                  100 *
[1648] 2019/04/09 18:26:18.774136 [INF] STREAM: --------- Channels Limits --------
[1648] 2019/04/09 18:26:18.775705 [INF] STREAM:   Subscriptions:          1000 *
[1648] 2019/04/09 18:26:18.775705 [INF] STREAM:   Messages     :       1000000 *
[1648] 2019/04/09 18:26:18.778603 [INF] STREAM:   Bytes        :     976.56 MB *
[1648] 2019/04/09 18:26:18.778603 [INF] STREAM:   Age          :     unlimited *
[1648] 2019/04/09 18:26:18.782508 [INF] STREAM:   Inactivity   :     unlimited *
[1648] 2019/04/09 18:26:18.782508 [INF] STREAM: ----------------------------------

I wrote the docker file and instructions on Docker hub..

I am getting error on both windows and MacOS with your command. Also, you are mapping Host 4224 to container 4223 (I am mapping 4223 to 4223). when I telnet, I get the following response

INFO {"server_id":"W2L6anhIotnnvrLWjUkdSq","version":"1.4.1","proto":1,"git_commit":"3e64f0b","go":"go1.11.6","host":"0.0.0.0","port":4223,"max_payload":1048576,"client_id":16}

Also, below is my server startup screen

mne$ docker run --rm -ti --name gnatsd-streamer -p 4223:4223 -p 8223:8223 -v /Users/mne/Desktop/streamer_conf.conf:/streamer_conf.conf -v /Users/mne/streamer1:/streamer1 nats-streaming:latest -p 4223 -m 8223 -c /streamer_conf.conf
[1] 2019/04/17 22:57:59.734110 [INF] STREAM: Starting nats-streaming-server[streamer1] version 0.12.2
[1] 2019/04/17 22:57:59.734613 [INF] STREAM: ServerID: v6zkefe4BZKkD6CJWS0JWX
[1] 2019/04/17 22:57:59.734621 [INF] STREAM: Go version: go1.11.6
[1] 2019/04/17 22:57:59.734626 [INF] STREAM: Git commit: [4489c46]
[1] 2019/04/17 22:57:59.735894 [INF] Starting nats-server version 1.4.1
[1] 2019/04/17 22:57:59.735918 [INF] Git commit [3e64f0b]
[1] 2019/04/17 22:57:59.736035 [INF] Starting http monitor on 0.0.0.0:8223
[1] 2019/04/17 22:57:59.736095 [INF] Listening for client connections on 0.0.0.0:4223
[1] 2019/04/17 22:57:59.736222 [INF] Server is ready
[1] 2019/04/17 22:57:59.781052 [INF] STREAM: Starting in standby mode
[1] 2019/04/17 22:58:01.106784 [INF] STREAM: Server is active
[1] 2019/04/17 22:58:02.120748 [INF] STREAM: Recovering the state...
[1] 2019/04/17 22:58:02.128402 [INF] STREAM: No recovered state
[1] 2019/04/17 22:58:02.130771 [DBG] STREAM: Discover subject: discover.streamer1
[1] 2019/04/17 22:58:02.130784 [DBG] STREAM: Publish subjects root: _STAN.pub.streamer1
[1] 2019/04/17 22:58:02.130788 [DBG] STREAM: Subscribe subject: _STAN.sub.streamer1
[1] 2019/04/17 22:58:02.130791 [DBG] STREAM: Subscription Close subject: _STAN.subclose.streamer1
[1] 2019/04/17 22:58:02.130794 [DBG] STREAM: Unsubscribe subject: _STAN.unsub.streamer1
[1] 2019/04/17 22:58:02.130797 [DBG] STREAM: Close subject: _STAN.close.streamer1
[1] 2019/04/17 22:58:02.131125 [INF] STREAM: Message store is FILE
[1] 2019/04/17 22:58:02.131143 [INF] STREAM: Store location: /streamer1
[1] 2019/04/17 22:58:02.131235 [INF] STREAM: ---------- Store Limits ----------
[1] 2019/04/17 22:58:02.131275 [INF] STREAM: Channels: 20
[1] 2019/04/17 22:58:02.131279 [INF] STREAM: --------- Channels Limits --------
[1] 2019/04/17 22:58:02.131282 [INF] STREAM: Subscriptions: unlimited
[1] 2019/04/17 22:58:02.131284 [INF] STREAM: Messages : unlimited
[1] 2019/04/17 22:58:02.131287 [INF] STREAM: Bytes : unlimited
[1] 2019/04/17 22:58:02.131289 [INF] STREAM: Age : 15m0s
[1] 2019/04/17 22:58:02.131324 [INF] STREAM: Inactivity : 900h0m0s
[1] 2019/04/17 22:58:02.131329 [INF] STREAM: -------- List of Channels --------
[1] 2019/04/17 22:58:02.131331 [INF] STREAM: company_config
[1] 2019/04/17 22:58:02.131334 [INF] STREAM: company
[1] 2019/04/17 22:58:02.131336 [INF] STREAM: foo
[1] 2019/04/17 22:58:02.131339 [INF] STREAM: supplier_table
[1] 2019/04/17 22:58:02.131374 [INF] STREAM: shared_table
[1] 2019/04/17 22:58:02.131378 [INF] STREAM: ----------------------------------

kozlovic · 2019-04-17T23:17:20Z

On the subject of the entry point and need for me to pass the executable name:
nats-io/nats-streaming-docker@9596dfa#diff-e2b7dae7d59ab0b01b6fa33dfcc4c7af
and
docker-library/official-images#3622

You may have missed my earlier comment regarding the actual issue: the problem is with the discover_prefix.

kozlovic · 2019-04-17T23:19:15Z

To be clear, issue of connection timeout can be solved if you follow instructions here

mmanna-sapfgl · 2019-04-17T23:28:05Z

On the subject of the entry point and need for me to pass the executable name:
nats-io/nats-streaming-docker@9596dfa#diff-e2b7dae7d59ab0b01b6fa33dfcc4c7af
and
docker-library/official-images#3622

You may have missed my earlier comment regarding the actual issue: the problem is with the discover_prefix.

streamer_conf.conf.txt

I have tried without it (Attached new conf file) and I am still getting the same issue. I am using nats-streaming:latest image. And my docker version output is the following (MacOS, but the issue is identical on Windows):

Client: Docker Engine - Community
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:47:43 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:55:00 2018
  OS/Arch:          linux/amd64
  Experimental:     true

mmanna-sapfgl · 2019-04-17T23:33:35Z

@kozlovic Okay got it to pass through connection issue. The reason it kept failing was because of the volume mapping where it kept the client and server .dat files. Once I deleted them and restarted the streaming server, it went past the connection issue. +1 for that.

kozlovic · 2019-04-17T23:47:46Z

I still see the discover_prefix in the last conf file that you have attached. Anyway, if you can connect it means that you have either removed it or change the test program to use that discover prefix.

Note that if you run a test that sends and reads messages from a channel, the next time you restart the test, if you use the same channel, there will be more messages added to the channel, while the subscriber will read at the beginning of the channel, which will cause perf to drop because server has to read data back from disk. You can use -io in the bench program to ignore old messages: the subscription will be created with start position "new only".

kozlovic · 2019-05-31T01:18:44Z

Received email notification from "kSpacer201" linked to this issue. I don't see the post in here, so copying for reference:

the command option -c can only to test-cluster , it doesn't auto-create if the cluster-name was others.

kSpacer201, not sure what you are stating/asking here. Could you elaborate?

reidlai · 2020-04-19T08:26:24Z

i also encountered the same case when i change to use cluster mode in Kubernetes. If using single model (no cluster config with memory), everything works fine. I am now testing different situation and update here later.

kozlovic · 2020-04-20T01:10:49Z

@reidlai Few things you need to check when getting this timeout error:

Make sure you pass to the Connect() call the proper cluster ID (which has to match the one used by the servers)
In cluster mode: make sure that the configuration is correct and a leader has been elected. You should see a trace in the log stating that a node has become leader. If not, no client will be able to connect.
I think you are using SQL (based on another issue you opened): make sure that each node in the cluster is using its own SQL tables.

mmanna-sapfgl changed the title ~~nats streaming connection fails with timeout~~ nats streaming - client connection fails (timeout) Apr 17, 2019

kozlovic closed this as completed Apr 17, 2019

kozlovic reopened this Apr 17, 2019

mmanna-sapfgl closed this as completed Apr 17, 2019

mmanna-sapfgl reopened this Apr 17, 2019

mmanna-sapfgl closed this as completed Apr 17, 2019

mmanna-sapfgl mentioned this issue Apr 18, 2019

Windows - Docker Run Command Incorrect (Possibly) #243

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nats streaming - client connection fails (timeout) #242

nats streaming - client connection fails (timeout) #242

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented May 31, 2019

reidlai commented Apr 19, 2020

kozlovic commented Apr 20, 2020

nats streaming - client connection fails (timeout) #242

nats streaming - client connection fails (timeout) #242

Comments

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 • edited Loading

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 • edited Loading

kozlovic commented Apr 17, 2019

kozlovic commented Apr 17, 2019

mmanna-sapfgl commented Apr 17, 2019 • edited Loading

mmanna-sapfgl commented Apr 17, 2019

kozlovic commented Apr 17, 2019

kozlovic commented May 31, 2019

reidlai commented Apr 19, 2020

kozlovic commented Apr 20, 2020

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading

mmanna-sapfgl commented Apr 17, 2019 •

edited

Loading