Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Support to configure socket timeout #319

Merged
merged 1 commit into from
Dec 15, 2023

Conversation

banmoy
Copy link
Collaborator

@banmoy banmoy commented Dec 14, 2023

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Which issues of this PR fixes :

Fixes #

Problem Summary(Required) :

Add a configuration sink.socket.timeout-ms. It's the timeout in milliseconds that the http client waits for data or, put differently, a maximum period inactivity between two consecutive data packets. The default value -1 is same as that of apache http client which is interpreted as undefined (system default if applicable). A timeout value of zero is interpreted as an infinite timeout. You can use this option to fail the stream load from the connector side if the http client does not receive response from StarRocks before timeout. The other option sink.properties.timeout take effects on the StarRocks side, but the response to the client may delay in some unexpected cases. If you want to have a strict timeout from the connector side, you can set this option to an acceptable value.

After reaching the timeout, there will be an exception like this

java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_345]
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_345]
	at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_345]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_345]
	at com.starrocks.streamload.shade.org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]
	at com.starrocks.streamload.shade.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[starrocks-stream-load-sdk-1.0-SNAPSHOT-jar-with-dependencies.jar:?]

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr will affect users' behaviors
  • This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function

@banmoy banmoy force-pushed the socket_timeout branch 2 times, most recently from 92c3477 to 93b43a5 Compare December 14, 2023 13:48
@banmoy banmoy merged commit e532d48 into StarRocks:main Dec 15, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants