Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [CDCSOURCE] sync mysql to paimon with oss catalog error #4150

Open
2 of 3 tasks
NCUZK opened this issue Jan 15, 2025 · 2 comments
Open
2 of 3 tasks

[Bug] [CDCSOURCE] sync mysql to paimon with oss catalog error #4150

NCUZK opened this issue Jan 15, 2025 · 2 comments
Assignees
Labels
Bug Something isn't working

Comments

@NCUZK
Copy link

NCUZK commented Jan 15, 2025

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

dinky task error log:
2025-01-15 15:03:58.229 INFO org.dinky.service.task.FlinkSqlTask(69): Initializing Flink job config... 2025-01-15 15:03:58.234 INFO org.dinky.trans.AbstractOperation(86): Start build CDCSOURCE Task... 2025-01-15 15:03:58.238 INFO org.dinky.trans.AbstractOperation(184): A total of 1 tables were detected... 2025-01-15 15:03:58.238 INFO org.dinky.trans.AbstractOperation(186): 1: mysql-database.mysql-table 2025-01-15 15:03:58.238 INFO org.dinky.trans.AbstractOperation(195): Set parallelism: 1 2025-01-15 15:03:58.238 INFO org.dinky.trans.AbstractOperation(199): Set checkpoint: 10000 2025-01-15 15:03:58.238 INFO org.dinky.trans.AbstractOperation(202): Build mysql-cdc successful... 2025-01-15 15:03:58.239 INFO org.dinky.cdc.AbstractSinkBuilder(190): Build deserialize successful... 2025-01-15 15:03:58.239 INFO org.dinky.cdc.AbstractSinkBuilder(203): Build partitionBy successful... 2025-01-15 15:03:58.240 INFO org.dinky.cdc.AbstractSinkBuilder(205): Build shunt successful... 2025-01-15 15:03:58.240 ERROR org.dinky.cdc.AbstractSinkBuilder(152): Build mysql-database.mysql-table cdc sync failed... 2025-01-15 15:03:58.240 ERROR org.dinky.cdc.AbstractSinkBuilder(153): org.apache.flink.table.api.ValidationException: Variable character string length must be between 1 and 2147483647 (both inclusive). at org.apache.flink.table.types.logical.VarCharType.<init>(VarCharType.java:75) at org.apache.flink.table.types.logical.VarCharType.<init>(VarCharType.java:83) at org.dinky.cdc.convert.DataTypeConverter.getLogicalType(DataTypeConverter.java:113) at org.dinky.cdc.AbstractSinkBuilder.buildColumn(AbstractSinkBuilder.java:327) at org.dinky.cdc.sql.AbstractSqlSinkBuilder.lambda$addTableSinkForTags$0(AbstractSqlSinkBuilder.java:143) at java.base/java.util.LinkedHashMap.forEach(Unknown Source) at org.dinky.cdc.sql.AbstractSqlSinkBuilder.addTableSinkForTags(AbstractSqlSinkBuilder.java:135) at org.dinky.cdc.sql.AbstractSqlSinkBuilder.buildPipeline(AbstractSqlSinkBuilder.java:206) at org.dinky.cdc.AbstractSinkBuilder.build(AbstractSinkBuilder.java:293) at org.dinky.trans.ddl.CreateCDCSourceOperation.execute(CreateCDCSourceOperation.java:203) at org.dinky.interceptor.FlinkInterceptor.build(FlinkInterceptor.java:53) at org.dinky.executor.Executor.pretreatExecute(Executor.java:247) at org.dinky.executor.Executor.executeSql(Executor.java:260) at org.dinky.job.runner.JobPipelineRunner.run(JobPipelineRunner.java:76) at org.dinky.job.JobManager.executeSql(JobManager.java:289) at org.dinky.service.task.FlinkSqlTask.execute(FlinkSqlTask.java:70) at org.dinky.service.impl.TaskServiceImpl.executeJob(TaskServiceImpl.java:211) at org.dinky.service.impl.TaskServiceImpl.executeJob(TaskServiceImpl.java:202) at org.dinky.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$22087f7c.invoke(<generated>)

What you expected to happen

Task is running normally

How to reproduce

run this dinky job
`
SET 'table.exec.sink.upsert-materialize' = 'NONE';

EXECUTE CDCSOURCE demo
WITH
(
'connector' = 'mysql-cdc',
'hostname' = 'mysql-host.com',
'port' = '3306',
'username' = 'mysql-username',
'password' = 'mysql-password',
'checkpoint' = '10000',
'parallelism' = '1',
'database-name' = 'mysql-database',
'table-name' = 'mysql-database.mysql-table',
'server-time-zone' = 'Asia/Shanghai',
'scan.startup.mode' = 'initial',
'server-id' = '100-110',
'sink.connector' = 'paimon',
'sink.database' = 'test',
'sink.metastore' = 'filesystem',
'sink.path' = 'oss://paimon/pre/',
'sink.warehouse' = 'oss://paimon/pre/',
'sink.fs.oss.endpoint' = 'oss-endpoint',
'sink.fs.oss.accessKeyId' = 'xxxx',
'sink.fs.oss.accessKeySecret' = 'xxx'
);
`

Anything else

No response

Version

1.2.0

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@NCUZK NCUZK added Bug Something isn't working Waiting for reply Waiting for reply labels Jan 15, 2025
Copy link

Hello @NCUZK, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @NCUZK, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。

@aiwenmo aiwenmo removed the Waiting for reply Waiting for reply label Jan 15, 2025
@aiwenmo
Copy link
Contributor

aiwenmo commented Jan 15, 2025

Please check the length of the string field in MySQL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants