Debezium server is failing to create a table because of an invalid table identifier

[mevadadhruv]

Hello,

I'm reaching out to discuss an issue we're facing while setting up the Debezium-Iceberg integration in our architecture. We followed the steps outlined in Chris Cranford (Naros)'s blog post (https://debezium.io/blog/2022/09/30/debezium-oracle-series-part-1/) and ensured that the Oracle database user has been granted the necessary privileges as described in the Debezium documentation.

However, we've encountered an error that we're struggling to resolve. The error message in the logs indicates that the Debezium server is failing to create a table due to an invalid table identifier: apacheiceberg.debeziumcdc_aws_DNT_TEST. This issue arises during the processing of change data capture (CDC) events using the Iceberg sink connector.

Attached is a screenshot showing the exact error message for reference.

We would greatly appreciate any guidance or assistance you can provide to help us resolve this issue.

Thank you for your support.

Best regards,
Dhruv Mevada

[ismailsimsek]

@mevadadhruv as mentioned before please check that Glue catalog naming rules are supporting the naming you are using.

please test with Skip Name Validation config

#365 (comment)

the details of glue catalog here https://iceberg.apache.org/docs/nightly/aws/?h=glue#glue-catalog

[mevadadhruv]

also i was checking this issue in the debezium community side where Vojtech Juranek suggested us the "SMT". from the iceberg side it'll also help right ?

[GOVINDARAMTEKKAR97]

Hi @ismailsimsek , we are successfully able to get data to a3 bucket through debezium and iceberg.
When we create a table in lower case its work but for uppercase its showing errorand unable to create them.
Please let me know how should I bypass this thing or need to add any parameter in application.properties
please share the details or examples if possible.

Regards,
Govinda

[ismailsimsek]

you could add this dependency to your project and use this SMT https://github.com/jcustenborder/kafka-connect-transform-common
examples how to use it are available int the project.

additional details https://stackoverflow.com/a/60860628/2679740

[mevadadhruv]

hi @ismailsimsek,
Thanks for giving quick reply everytime, and also thanks for this helps now the thing is recent solution for our error SMT it's not feasible as we are hoping because the link which you share it totally related to kafka, which we are not using, we are using debezium iceberg consumer from github repository without kafka. now let me share our whole things like root level pom.xml file, application.properties file, error logs in file so you can guide easily. also after checking your solution we have also add this jar files
connect-utils-0.7.173.jar, freemarker-2.3.31.jar, kafka-connect-transform-common-0.1.0.58.jar, ws-commons-util-1.0.1.jar , xml-apis-1.4.01.jar, connect-utils-jackson-0.7.173.jar, junit-3.8.1.jar, value-2.8.2.jar, xercesImpl-2.12.1.jar.

application_properties.txt
errorlog.txt
pom_xml.txt

[mevadadhruv]

hi @ismailsimsek ,
Please, do let me know if you are finding difficulties to understand this !
Thank you,
Dhruv

[mevadadhruv]

Good Morning, @ismailsimsek

[mevadadhruv]

@ismailsimsek ?

[ismailsimsek]

this option is now implemented in 0.6.0.Final release

[mevadadhruv]

Hi @ismailsimsek,

Thank you so much for addressing our issue and providing a release that resolved the problem. We really appreciate your support.

Currently, we are able to retrieve data from our source, but it's only working for a single table. After reviewing the Debezium documentation, we understand that Debezium should be capable of capturing data from multiple tables simultaneously.

However, we're now encountering an issue where we can't achieve this. We've attempted several different scenarios to resolve the problem, but unfortunately, we haven't been successful.

Could you please assist us with this? Below, I'm sharing our application.properties file and the logs we're receiving for your reference.

Thank you in advance for your help!

debe-logs.txt
app_pro.txt