OpenSRP serverVersions generated inconsistently on reveal-stage

[gstuder-ona]

During the reveal-stage events Beam ETL process, the following read from OpenSRP was observed:

2021-02-22 07:39:55,032 [unbounded-source-system-consumer-0] INFO com.onaio.beam.etl.OpenSrpRestSource  - Fetching next batch: serverVersion=1599116541924
2021-02-22 07:39:55,050 [unbounded-source-system-consumer-0] INFO com.onaio.beam.etl.OpenSrpRestSource  - Fetched new batch: size=1
2021-02-22 07:39:55,051 [unbounded-source-system-consumer-0] INFO com.onaio.beam.etl.OpenSrpRestSource  - Fetching next batch: serverVersion=1613979578956

This indicates that getAll() was called with version 1599116541924 and a batch was returned with a single value with version 1613979578955. When you call getAll() today, you get back at least 100 events before that server version:

https://reveal-stage.smartregister.org/opensrp/rest/event/getAll?serverVersion=1599116541923&limit=100

id  serverVersion created_at event_date
f8632d51-9258-4255-8967-4fbaf2c4445a - 1599116541957 - 2021-02-22T17:08:32.364+02:00 2021-02-22T02:00:00.000+02:00
fd702668-d739-4bc0-92f0-94f0153b394f - 1599116541958 - 2021-02-22T17:08:32.418+02:00 2021-02-22T02:00:00.000+02:00
72704e1e-a741-49ec-aca7-152eb2db5761 - 1599116541959 - 2021-02-22T17:10:59.901+02:00 2021-02-21T23:00:00.000+02:00
9a7971e5-8539-40fb-bb46-4b579c4a8f66 - 1599116541960 - 2021-02-22T17:52:56.493+02:00 2021-02-22T00:00:00.000+02:00
...
43125155-a42a-4c2e-8926-b07d944728b3 - 1599116542120 - 2021-03-03T09:52:43.784+02:00 2021-03-03T02:00:00.000+02:00

Note the final event in this batch was created in March.

When you look at the event with the weird 1613979578955 server version:

https://reveal-stage.smartregister.org/opensrp/rest/event/getAll?serverVersion=1613979578954&limit=2

b3b35cd4-24c7-48a9-96d2-788bbeff379d - 1613979578955 - 2021-02-22T09:37:29.047+02:00 2021-02-22T00:00:00.000+02:00
936a4957-c6e4-4bc0-8d40-2e587454d35c - 1613980119096 - 2021-02-22T09:46:36.603+02:00 2021-02-21T23:00:00.000+02:00

You can see that the later serverVersion event (1613979578955) was created on February 22 (and was visible to the ETL on February 22), and the intermediate server version events (1599116542120) were not visible to the ETL.

It's possible this is a Beam ETL bug, however the NiFi ETL raw_events table also does not contain entries for these intermediate events (and generally both ETLs seem to be missing a lot of events in this range). Was there a server clock reset or something going on?