From 1e42b2e5aefed0056409f10136d2b3cf691a6416 Mon Sep 17 00:00:00 2001 From: Chris Cranford Date: Mon, 18 Nov 2024 09:03:26 -0500 Subject: [PATCH] 3.0.2.Final announcement --- _data/releases/3.0/3.0.2.Final.yml | 2 +- ...4-11-18-debezium-3-0-2-final-released.adoc | 184 ++++++++++++++++++ 2 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 _posts/2024-11-18-debezium-3-0-2-final-released.adoc diff --git a/_data/releases/3.0/3.0.2.Final.yml b/_data/releases/3.0/3.0.2.Final.yml index f68098a40c..484d4eb685 100644 --- a/_data/releases/3.0/3.0.2.Final.yml +++ b/_data/releases/3.0/3.0.2.Final.yml @@ -2,4 +2,4 @@ date: 2024-11-15 version: "3.0.2.Final" stable: true summary: Vitess connector can cand schema change events; Debezium Operator can enable Debezium Server REST API; Oracle Connector suppoert NLS time format; Offset storage in k8s config map; Improved reliability of read-only incremental snapshots; RabbitMQ sink no longer skips data in case of failure -#announcement_url: +announcement_url: /blog/2024/11/18/debezium-3-0-2-final-released/ diff --git a/_posts/2024-11-18-debezium-3-0-2-final-released.adoc b/_posts/2024-11-18-debezium-3-0-2-final-released.adoc new file mode 100644 index 0000000000..8576b8d24e --- /dev/null +++ b/_posts/2024-11-18-debezium-3-0-2-final-released.adoc @@ -0,0 +1,184 @@ +--- +layout: post +title: Debezium 3.0.2Final Released +date: 2024-11-18 +tags: [ releases, mongodb, mysql, mariadb, postgres, sqlserver, cassandra, oracle, db2, vitess, outbox, spanner, jdbc, informix, ibmi ] +author: ccranfor +--- + +I am excited to announce the second maintenance release for the Debezium 3 release stream, **3.0.2.Final**. +This maintenance release introduces a number of features, let's take a moment and dive into the highlights. + +++++++ + +[id="new-features-and-improvements"] +== New features and improvements + +Debezium 3.0.2.Final introduces a couple of improvements and features, lets take a look at each individually. + +=== Core + +==== Perform blocking snapshots with never snapshot mode + +The Debezium blocking snapshot process is designed to execute the _initial_ snapshot based on the signal provided, selectively emitting the historical data for one or more tables. +When this was paired with the _never_ `snapshot mode`, this lead to unexpected behavior. + +In this release, we modified the connector offsets to track the configured `snapshot.mode`, allowing the blocking snapshot to succeed and perform the _initial_ snapshot when signaled, even if the `snapshot.mode` is configured to _never_ perform a snapshot. +This allows users to safely use this feature with this configuration (https://issues.redhat.com/browse/DBZ-7903[DBZ-7903]). + +[WARNING] +==== +Due to the connector offset storage change, once the connector is upgraded to 3.0.2.Final or later, the connector cannot be downgraded to 3.0.1.Final or earlier. +==== + +=== MongoDB + +==== RowsChanged JMX metric type changed + +In previous builds of the MongoDB connector, the `RowsChanged` JMX metric was being exposed as a `java.util.Map`, which contradicted the same JMX metric exposed on the relational connectors, which is `TabularData`. +This has been fixed in 3.0.2.Final so that the JMX metric is exposed identical to its relational connector counterparts (https://issues.redhat.com/browse/DBZ-8359[DBZ-8359]). + +[NOTE] +==== +Any existing JMX pipelines for MongoDB may need to be adjusted. +Please be sure to review your JMX pipelines before applying to your production systems if you use `RowsChanged`. +==== + +=== Oracle + +=== Higher precision with timestamps + +Debezium for Oracle has traditionally emitted column timestamp values with millisecond precision, influenced by the NLS session properties set on the mining session connection. +We have improved this behavior by setting the precision to nanosecond (aka `FF9`) precision (https://issues.redhat.com/browse/DBZ-8379[DBZ-8379]). + +[NOTE] +==== +The emitted field type is based on the column's data type, so field emitted data types remain unchanged. +What will change is cases where columns have micro or nanosecond-based values, where these were previously zero, they'll now have non-zero values. +==== + +==== Warn or skip DML exceptions + +The `event.processing.failure.handling.mode` can be configured to _fail_, _warn_, or _skip_ specific connector error conditions to improve connector reliability to various data issues. +For the Oracle connector, this configuration is historically used to control how to behave when an error is encountering with DDL failures. + +In this release, the `event.processing.failure.handling.mode` extends to DML exceptions, which means that if there was an issue with the Oracle connector parsing your _insert_, _update_, or _delete_ operations, you can safely configure the connector to _fail_, _warn_, or _skip_ the DML event based on your needs (https://issues.redhat.com/browse/DBZ-8208[DBZ-8208]). + +[NOTE] +==== +The default behavior is to always _fail_ when an event fails to be handled by the connector. +By adjusting this to _warn_ or _skip_, you will introduce data loss as the event will not be emitted by the connector, and you'll need to address this manually. +==== + +=== Vitess + +==== Performance improvements + +In earlier builds of the Debezium for Vitess connector, the connector used a regular expression-based filter system that would match all tables based on a prefix with varying suffixes, and later would be excluded due to configuration. +This has the potential to waste CPU and create a hotspot due to creating a set of intermediate objects for the event that would later to be filtered. + +In this release, we've improved the way the Vitess connector processes this use case by applying the filtration earlier in the event processing chain. +This should reduce the number of intermediate objects created and improve the overall performance of the connector. +For keyspaces that have the same prefix and differing suffixes, this should provide better overall performance than older builds (https://issues.redhat.com/browse/DBZ-8354[DBZ-8354]). + +=== Debezium Operator + +==== Enabling Debezium Server REST endpoint + +The Debezium Server API REST-ful endpoint can now be automatically enabled through a Debezium Server deployment on Kubernetes using the Debezium Operator. +Within the `spec` section of the deployment descriptor, you can include the `runtime` section to toggle the API endpoint (https://issues.redhat.com/browse/DBZ-8234[DBZ-8234]). + +.An example YAML configuration +[source,yaml] +---- +apiVersion: debezium.io/v1alpha1 +kind: DebeziumServer +metadata: + name: my-debezium +spec: + image: quay.io/debezium/server:3.0.2.Final + quarkus: + config: + log.console.json: false + kubernetes-config.enabled: true + kubernetes-config.secrets: postgresql-credentials + runtime: + api: + enabled: true + sink: + type: kafka + config: + producer.bootstrap.servers: dbz-kafka-kafka-bootstrap:9092 + producer.key.serializer: org.apache.kafka.common.serialization.StringSerializer + producer.value.serializer: org.apache.kafka.common.serialization.StringSerializer + source: + class: io.debezium.connector.postgresql.PostgresConnector + offset: + memory: { } + schemaHistory: + memory: { } + config: + database.hostname: postgresql + database.port: 5432 + database.user: ${POSTGRES_USER} + database.password: ${POSTGRES_PASSWORD} + database.dbname: ${POSTGRES_DB} + topic.prefix: inventory + schema.include.list: inventory +---- + +By default, the Debezium Server API endpoint is disabled, but can be enabled by setting the `spec.runtime.api.enabled` with a value of `true`, as shown above. + +[id="other-fixes"] +== Other fixes + +In total there were https://issues.redhat.com/issues/?jql=project%20%3D%20DBZ%20and%20fixVersion%20%20in%20(3.0.2.Final)[46 issues] resolved in Debezium 3.0.2.Final. +The list of changes can also be found in our https://debezium.io/releases/3.0[release notes]. + +Here are some noteworthy changes: + +* Clarify signal data collection should be unique per connector https://issues.redhat.com/browse/DBZ-6837[DBZ-6837] +* Race condition in stop-snapshot signal https://issues.redhat.com/browse/DBZ-8303[DBZ-8303] +* Debezium shifts binlog offset despite RabbitMQ Timeout and unconfirmed messages https://issues.redhat.com/browse/DBZ-8307[DBZ-8307] +* Use DebeziumSinkRecord instead of Kafka Connect's SinkRecord inside Debezium sink connectors https://issues.redhat.com/browse/DBZ-8346[DBZ-8346] +* Implement new config map offset store in DS https://issues.redhat.com/browse/DBZ-8351[DBZ-8351] +* Debezium server with eventhubs sink type and eventhubs emulator connection string fails https://issues.redhat.com/browse/DBZ-8357[DBZ-8357] +* Filter for snapshot using signal doesn't seem to work https://issues.redhat.com/browse/DBZ-8358[DBZ-8358] +* JDBC storage module does not use quay.io images https://issues.redhat.com/browse/DBZ-8362[DBZ-8362] +* Failure on offset store call to configure/start is logged at DEBUG level https://issues.redhat.com/browse/DBZ-8364[DBZ-8364] +* Object name is not in the list of S3 schema history fields https://issues.redhat.com/browse/DBZ-8366[DBZ-8366] +* Faulty "Failed to load mandatory config" error message https://issues.redhat.com/browse/DBZ-8367[DBZ-8367] +* Upgrade protobuf dependencies to avoid potential vulnerability https://issues.redhat.com/browse/DBZ-8371[DBZ-8371] +* Add transform page to provide a single place to list the already configured transform plus UI to add a new transform https://issues.redhat.com/browse/DBZ-8374[DBZ-8374] +* Upgrade Kafka to 3.8.1 https://issues.redhat.com/browse/DBZ-8385[DBZ-8385] +* Tests in IncrementalSnapshotIT may fail randomly https://issues.redhat.com/browse/DBZ-8386[DBZ-8386] +* Add Transform Edit and delete support. https://issues.redhat.com/browse/DBZ-8388[DBZ-8388] +* Log SCN existence check may throw ORA-01291 if a recent checkpoint occurred https://issues.redhat.com/browse/DBZ-8389[DBZ-8389] +* ExtractNewRecordState transform: NPE when processing non-envelope records https://issues.redhat.com/browse/DBZ-8393[DBZ-8393] +* Oracle LogMiner metric OldestScnAgeInMilliseconds can be negative https://issues.redhat.com/browse/DBZ-8395[DBZ-8395] +* SqlServerConnectorIT.restartInTheMiddleOfTxAfterCompletedTx fails randomly https://issues.redhat.com/browse/DBZ-8396[DBZ-8396] +* ExtractNewDocumentStateTestIT fails randomly https://issues.redhat.com/browse/DBZ-8397[DBZ-8397] +* BlockingSnapshotIT fails on Oracle https://issues.redhat.com/browse/DBZ-8398[DBZ-8398] +* Oracle OBJECT_ID lookup and cause high CPU and latency in Hybrid mining mode https://issues.redhat.com/browse/DBZ-8399[DBZ-8399] +* Upgrade Kafka to 3.9.0 https://issues.redhat.com/browse/DBZ-8400[DBZ-8400] +* Protobuf plugin does not compile for PostgreSQL 17 on Debian https://issues.redhat.com/browse/DBZ-8403[DBZ-8403] +* Update Quarkus Outbox Extension to Quarkus 3.16.3 https://issues.redhat.com/browse/DBZ-8409[DBZ-8409] + +A big thank you to all the contributors from the community who worked diligently on this release: +https://github.com/ani-sha[Anisha Mohanty], +https://github.com/dasvh[dario], +https://github.com/Naros[Chris Cranford], +https://github.com/enzo-cappa[Enzo Cappa], +https://github.com/jcechace[Jakub Cechacek], +https://github.com/jpechane[Jiri Pechanec], +https://github.com/kavyaramaiah1991[Kavya Ramaiah], +https://github.com/nrkljo[Lars M. Johansson], +https://github.com/mfvitale[Mario Fiore Vitale], +https://github.com/martinvlk[Martin Vlk], +https://github.com/paumr[P. Aum], +https://github.com/rk3rn3r[René Kerner], +https://github.com/stn1slv[Stanislav Deviatov], +https://github.com/smiklosovic[Stefan Miklosovic], +https://github.com/twthorn[Thomas Thornton], +https://github.com/vjuranek[Vojtech Juranek], and +Yevhenii Lopatenko! \ No newline at end of file