-
Notifications
You must be signed in to change notification settings - Fork 188
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
132 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| --- | ||
| layout: post | ||
| title: Debezium 2.7.0.Alpha2 Released | ||
| date: 2024-05-13 | ||
| tags: [ releases, mongodb, mysql, mariadb, postgres, sqlserver, cassandra, oracle, db2, vitess, outbox, spanner, jdbc, informix, ibmi ] | ||
| author: ccranfor | ||
| --- | ||
|
|
||
| The old saying is "April showers bring May flowers"; however, in this case it seems a new Debezium release has sprouted packed with many new features. | ||
| We're pleased to announce the release of Debezium **2.7.0.Alpha2**, the next pre-release in the Debezium 2.7 stream, is now available for testing. | ||
|
|
||
| This release includes new ROW_ID serialization for the Oracle connector, PostgreSQL array support for the JDBC sink connector, NATs authentication with Debezium Server, performance improvements with Oracle LogMiner and large tables, and more. | ||
| Let's walk through the highlights of this release and discuss these and more in-depth... | ||
|
|
||
| +++<!-- more -->+++ | ||
|
|
||
| [id="new-features-and-improvements"] | ||
| == New features and improvements | ||
|
|
||
| Debezium 2.7.0.Alpha2 also introduces many improvements and features, lets take a look at each individually. | ||
|
|
||
| === Oracle ROW_ID included in change events | ||
|
|
||
| While `ROW_ID` is not unique across all rows of a table for the table's lifespan, it can be used in certain situations when the lifecycle of the table and rows are managed in a very strict way. | ||
| At the community's request, we've added a new `row_id` field to the Oracle connector's change event source information block (https://issues.redhat.com/browse/DBZ-4332[DBZ-4332]). | ||
| This new field will be populated with the `ROW_ID` value under the following conditions: | ||
|
|
||
| * Only populated from streaming events for inserts, updates, and deletes. | ||
| * Snapshot evnets will not contain a `row_id` value. | ||
| * Only provided by the LogMiner and XStream streaming adapters, OpenLogReplicator is not supported. | ||
|
|
||
| Any event that does not match the criteria will not include a `row_id` field as its marked as _optional_. | ||
|
|
||
| === PostreSQL Arrays with the JDBC sink | ||
|
|
||
| The JDBC sink connector supports the use of mapping source columns to Kafka ARRAY-based payload field types. | ||
| With Debezium 2.7, you can now serialize ARRAY-based fields to a target PostgreSQL database, with no change in configuration. | ||
| The new support should be completely transparent (https://issues.redhat.com/browse/DBZ-7752[DBZ-7752]). | ||
|
|
||
| === Oracle flush table with custom schema names | ||
|
|
||
| In prior versions of Debezium, the Oracle connector was strictly designed to create the LogMiner flush table in the default tablespace of the connector user account. | ||
| This wasn't always useful in situations where the user's default tablespace may not be the ideal destination and the DBA would prefer that table to exist in a separate tablespace. | ||
|
|
||
| Previously, users would need to modify the user account or use a new user with the correct tablespace to have the table created in the right tablespace location. | ||
| With Debezium 2.7, this is no longer required, and you can safely include the name of the target schema/tablespace in the configuration (https://issues.redhat.com/browse/DBZ-7819[DBZ-7819]). | ||
|
|
||
| .Example using a custom schema name | ||
| [source] | ||
| ---- | ||
| log.mining.flush.table.name=THE_OTHER_SCHEMA.LOG_MINING_FLUSH_TABLE | ||
| ---- | ||
|
|
||
| The schema name is optional and if not supplied, the connector continues to use the same legacy behavior of creating the flush table and checking for its existence in the user's default tablespace. | ||
|
|
||
| === NATS authentication with JWT/seed | ||
|
|
||
| The Debezium Server NATs streaming sink adapter was improved, supporting JWT/seed based authentication (https://issues.redhat.com/browse/DBZ-7829[DBZ-7829]). | ||
| To get started using the JWT/seed-based authentication, supply the following necessary values in the configuration: | ||
|
|
||
| .JWT authentication example | ||
| [source] | ||
| ---- | ||
| debezium.sink.nats-jetstream.auth.jwt=<your_jwt_token> | ||
| ---- | ||
|
|
||
| .Seed authentication example | ||
| [source] | ||
| ---- | ||
| debezium.sink.nats-jetstream.auth.seed=<your_nkey_seed> | ||
| ---- | ||
|
|
||
| For this and more, please see the NATS documentation for details about JWT and NKey seed based authentication. | ||
|
|
||
| === Oracle query filter with large numbers of tables | ||
|
|
||
| The Debezium Oracle connector can support thousands of tables in a single connector deployment with ease; however, you may have found you wanted to customize the query filter using the `IN` mode. | ||
| This mode is used in situations where you may have a high volume of changes for other tables and you want to filter that dataset out at the database level before the changes are passed to Debezium for processing. | ||
|
|
||
| In earlier versions, users may have noticed that setting `log.mining.query.filter.mode` with a value of `in` and where your table include list contained more than 1000 elements generated a SQL error. | ||
| Oracle does not permit more than 1000 elements within an in-clause; however, Debezium 2.7 addresses this limitation by using a disjunction between multiple buckets of 1000 item in-clause lists (https://issues.redhat.com/browse/DBZ-7847[DBZ-7847]). | ||
|
|
||
| [id="other-changes"] | ||
| == Other changes | ||
|
|
||
| Altogether, https://issues.redhat.com/issues/?jql=project%20%3D%20DBZ%20AND%20fixVersion%20%3D%202.7.0.Alpha2%20ORDER%20BY%20issuetype%20DESC[27 issues] were fixed in this release. | ||
| Here are a list of some additional noteworthy changes: | ||
|
|
||
| * Log exception details early in case MySQL keep-alive causes deadlock on shutdown https://issues.redhat.com/browse/DBZ-7570[DBZ-7570] | ||
| * Extend mongodb system tests with ssl option https://issues.redhat.com/browse/DBZ-7605[DBZ-7605] | ||
| * > io.debezium.text.ParsingException : SQL Contains Partition https://issues.redhat.com/browse/DBZ-7805[DBZ-7805] | ||
| * Ad-hoc blocking snapshot not working through file channeling without inserting a row in the database. https://issues.redhat.com/browse/DBZ-7806[DBZ-7806] | ||
| * Postgres: Potential data loss on connector restart https://issues.redhat.com/browse/DBZ-7816[DBZ-7816] | ||
| * DEBEZIUM_VERSION is wrongly set to 2.6.0.Alpha1 https://issues.redhat.com/browse/DBZ-7827[DBZ-7827] | ||
| * Sql Server incorrectly applying quoted snapshot statement overrides https://issues.redhat.com/browse/DBZ-7828[DBZ-7828] | ||
| * Debezium JDBC Sink not handle order correctly https://issues.redhat.com/browse/DBZ-7830[DBZ-7830] | ||
| * Bump Outbox Extension to Quarkus 3.10.0 https://issues.redhat.com/browse/DBZ-7842[DBZ-7842] | ||
| * Support Oracle DDL Alter Audit Policy https://issues.redhat.com/browse/DBZ-7864[DBZ-7864] | ||
| * Support Oracle DDL Create Audit Policy https://issues.redhat.com/browse/DBZ-7865[DBZ-7865] | ||
|
|
||
| A huge thank you to all contributors from the community who worked on this release: | ||
| https://github.com/acristu[Alexandru Cristu], | ||
| https://github.com/adasari[Anil Dasari], | ||
| https://github.com/bpaquet[Bertrand Paquet], | ||
| https://github.com/cjho0316[Choi Jang Ho], | ||
| https://github.com/Naros[Chris Cranford], | ||
| https://github.com/calmera[Daan Gerits], | ||
| https://github.com/DLT1412[Duc Le Tu], | ||
| https://github.com/ilyasahsan123[Ilyas Ahsan], | ||
| https://github.com/ismailsimsek[Ismail Simsek], | ||
| https://github.com/jpechane[Jiri Pechanec], | ||
| https://github.com/zeldanerd24[Kevin Rothenberger], | ||
| https://github.com/mfvitale[Mario Fiore Vitale], | ||
| https://github.com/MartinMedek[Martin Medek], | ||
| https://github.com/PradeepNain[Pradeep Nain], | ||
| https://github.com/roldanbob[Robert Roldan], and | ||
| https://github.com/leoloel[leoloel]! | ||
|
|
||
| [id="whats-next"] | ||
| == What's next? | ||
|
|
||
| We have our team face-to-face next week, and it's going to be absolutely fantastic since its the first time we get to meet in person since the Covid pandemic. | ||
| We're going to use this time to discuss all the community feedback we're received throughout Debezium 2, reflect on what worked, and put together action plans for what didn't. | ||
|
|
||
| The main focus for our meeting is to develop an action plan for Debezium 3.0 and beyond, assign tasks and priorities across the team so that as we focus on Debezium 3 next quarter, we can make this next major release an easy replacement for the community while also delivering a new, refreshing, feature-rich major version. | ||
| We will be updating the roadmap and deliverables when we're back, so be sure to stay tuned to our link:/docs/roadmap[road map]. | ||
|
|
||
| In terms of Debezium 2.7, we're halfway through the quarter, and we're about to turn our focus on the last half where we will address any bugs, regressions, and polish new features. | ||
| If you have the chance to test-drive the pre-releases, we strongly encourage you to do so and file bug reports. | ||
|
|
||
| Until next time... |