You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hermes management creates special zookeeper nodes indicating new kafka offset for each consumer.
Proper hermes consumers are notified and start the retransmission operation:
a) seek to the specific offset
b) remove node created in point 2.
c) and start consuming from that offset on a kafka partition.
Hermes consumer process handling the partition dies and spawn once again before committing the older offset (ensued from retransmission).
The new hermes consumer starts consuming from the last committed offset on partition - which is the newest offset committed before the retransmission even happened.
Hermes consumer ignores the events related to retransmission.
In this case the events related to retransmission are ignored even if hermes management returned 200 to the user. To avoid this issue, we need to check if there is any pending retransmission before committing the offset. If there is, we should retry the retransmission. Once the retransmission is done we remove nodes with retransmission offsets for consumers.
The text was updated successfully, but these errors were encountered:
faderskd
changed the title
Offset committing should take into consideration pending retransmissions.
Offset committing should take into consideration pending retransmissions
Nov 14, 2022
Consider the following situation:
a) seek to the specific offset
b) remove node created in point 2.
c) and start consuming from that offset on a kafka partition.
In this case the events related to retransmission are ignored even if hermes management returned 200 to the user. To avoid this issue, we need to check if there is any pending retransmission before committing the offset. If there is, we should retry the retransmission. Once the retransmission is done we remove nodes with retransmission offsets for consumers.
The text was updated successfully, but these errors were encountered: