[Improve][Zeta] Disable hdfs filesystem cache of checkpoint #6718
Conversation
…cache function is disabled by default.
|
I don't have an oss environment, so I can't run oss unit tests: |
hailin0
changed the title
| @@ -186,3 +186,40 @@ seatunnel: | |||
|
|
|||
| ``` | |||
|
|
|||
| ### Enable cache | |||
|
|
|||
| When storage:type is hdfs, cache is disabled by default. If you want to enable it, set `disable.cache: false` | |||
There was a problem hiding this comment.
Could you share what risks will be caused if cache is turned on?
There was a problem hiding this comment.
Could you share what risks will be caused if cache is turned on?
I think in the scenario of seatunel, it should not be turned on at any time.
There was a problem hiding this comment.
Could you share what risks will be caused if cache is turned on?
You can take a look at the bug #6678 I proposed, which is [S3 connector], which causes occasional task failures due to the use of cache. at the same time, you can also refer to these two issues, which are bug: https://issues.apache.org/jira/browse/HADOOP-15819 and aws/aws-sdk-java#2337 about hadoop-aws and aws-sdk-java. The problems caused by turning on cache in multithreaded environment are described in detail.
|
Could you add a test case to cover this bug? |
Currently, there is no bug for connection closure in hdfs cache, so it is a precaution in advance. |
…cache function is disabled by default.
Purpose of this pull request
Does this PR introduce any user-facing change?
When using
hadoop-aws-3.1.4.jarandaws-java-sdk-bundle-1.11.271.jarto connect hdfs or s3 file systems, the default mode is cache. In multithreaded scenarios, FileSysyem objects are often closed, resulting in the closure of the connection pool. If the objects are taken from the cache, some unknown exceptions will be causedHow was this patch tested?
no
Check list
New License Guide
release-note.