Skip to content

Commit

Permalink
Merge branch 'apache:dev' into dev-cdc-heartbeat
Browse files Browse the repository at this point in the history
  • Loading branch information
@hailin0
hailin0 authored Jan 22, 2025
2 parents efc6a4f + 2bfb97e commit c67bcfe
Show file tree
Hide file tree
Showing 95 changed files with 2,880 additions and 390 deletions.
39 changes: 39 additions & 0 deletions docs/en/connector-v2/Config-Encryption-Decryption.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,42 @@ If you want to customize the encryption method and the configuration of the encr
5. Package it to jar and add jar to `${SEATUNNEL_HOME}/lib`
6. Change the option `shade.identifier` to the value that you defined in `ConfigShade#getIdentifier`of you config file, please enjoy it \^_\^

### How to encrypt and decrypt with customized params

If you want to encrypt and decrypt with customized params, you can follow the steps below:
1. Add a configuration named `shade.properties` in the env part of the configuration file, the value of this configuration is in the form of key-value pairs (the type of the key must be a string), as shown below:

```hocon
env {
shade.properties = {
suffix = "666"
}
}
```

2. Override the `ConfigShade` interface's `open` method, as shown below:

```java
public static class ConfigShadeWithProps implements ConfigShade {

private String suffix;
private String identifier = "withProps";

@Override
public void open(Map<String, Object> props) {
this.suffix = String.valueOf(props.get("suffix"));
}
}
```
3. Use the parameters passed in the open method in the encryption and decryption methods, as shown below:

```java
public String encrypt(String content) {
return content + suffix;
}

public String decrypt(String content) {
return content.substring(0, content.length() - suffix.length());
}
```
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/CosFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| encoding | string | no | "UTF-8" | Only used when file_format_type is json,text,csv,xml. |
Expand Down
2 changes: 1 addition & 1 deletion docs/en/connector-v2/sink/Doris.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The internal implementation of Doris sink connector is cached and imported by st
| table | String | Yes | - | The table name of `Doris` table, use `${table_name}` to represent the upstream table name |
| table.identifier | String | Yes | - | The name of `Doris` table, it will deprecate after version 2.3.5, please use `database` and `table` instead. |
| sink.label-prefix | String | Yes | - | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of SeaTunnel. |
| sink.enable-2pc | bool | No | false | Whether to enable two-phase commit (2pc), the default is false. For two-phase commit, please refer to [here](https://doris.apache.org/docs/dev/sql-manual/sql-statements/Data-Manipulation-Statements/Load/STREAM-LOAD/). |
| sink.enable-2pc | bool | No | false | Whether to enable two-phase commit (2pc), the default is false. For two-phase commit, please refer to [here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc). |
| sink.enable-delete | bool | No | - | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/) |
| sink.check-interval | int | No | 10000 | check exception with the interval while loading |
| sink.max-retries | int | No | 3 | the max retry times if writing records to database failed |
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/FtpFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/HdfsFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ Output data to hdfs file
| xml_row_tag | string | no | RECORD | Only used when file_format is xml, specifies the tag name of the data rows within the XML file |
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml, specifies Whether to process data using the tag attribute format. |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/LocalFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/ObsFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ It only supports hadoop version **2.9.X+**.
| is_enable_transaction | boolean | no | true | [Tips](#is_enable_transaction) |
| batch_size | int | no | 1000000 | [Tips](#batch_size) |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| compress_codec | string | no | none | [Tips](#compress_codec) |
| common-options | object | no | - | [Tips](#common_options) |
| max_rows_in_memory | int | no | - | When File Format is Excel,The maximum number of data items that can be cached in the memory.Only used when file_format is excel. |
Expand Down
1 change: 1 addition & 0 deletions docs/en/connector-v2/sink/OssFile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ If write to `csv`, `text` file type, All column will be string.
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
| single_file_mode | boolean | no | false | Each parallelism will only output one file. When this parameter is turned on, batch_size will not take effect. The output file name does not have a file block suffix. |
| create_empty_file_when_no_data | boolean | no | false | When there is no data synchronization upstream, the corresponding data files are still generated. |
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
Expand Down
Loading

0 comments on commit c67bcfe

Please sign in to comment.