You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
kafka_skip has a couple issues when it comes to delete events. Consider the scenario:
System A creates instance of Model A and emits event to topicA.
System B consumes event and creates same instance, setting kafka_skip=True so producers listening to changes on Model A don't re-send the event to topicA (Debezium source, or python).
Instance of ModelA is deleted in System B.
Because kafka_skip=True was set to the instance in 2., the producers will filter the deletion event to topicA when it shouldn't.
For the producer, if we simply stop filtering on kafka_skip in the case of deletes (either in the debezium config or python producer code). This instead just leads to duplicated message issues in the following scenario:
System B creates instance of Model A and emits event to topicA.
System A consumes event and creates the same instance.
System B then deletes the instance, emitting a deletion event to topicA.
System A consumes the event and deletes its instance.
Since there is no filtering for deletes, another delete event is fired to topicA.
Since instances are already deleted, there should be no further errors. The only problem is deletion events are duplicated for each additional system listening to the topic.
kafka_skip was intended as a marker to not cause re-send events when consuming create/update operations. However this marker is not set during deletes, if it's even possible.
Having duplicated message flow isn't great since it will make debugging message flow harder.
The text was updated successfully, but these errors were encountered:
kafka_skip
has a couple issues when it comes to delete events. Consider the scenario:topicA
.kafka_skip=True
so producers listening to changes on Model A don't re-send the event totopicA
(Debezium source, or python).kafka_skip=True
was set to the instance in2.
, the producers will filter the deletion event totopicA
when it shouldn't.For the producer, if we simply stop filtering on
kafka_skip
in the case of deletes (either in the debezium config or python producer code). This instead just leads to duplicated message issues in the following scenario:topicA
.topicA
.topicA
.kafka_skip
was intended as a marker to not cause re-send events when consuming create/update operations. However this marker is not set during deletes, if it's even possible.Having duplicated message flow isn't great since it will make debugging message flow harder.
The text was updated successfully, but these errors were encountered: