Questions about locking, client-side caching and node failure

[iqram-firebolt]

Hi @rueian, kudos on the work you did for redis/rueidis. I'm excitedly looking forward to use client-side caching with sub-microsecond read latency. I have a few quick questions.

• The Redlock Algorithm suggests to use a random delay in order to try to desynchronize multiple clients trying to acquire the lock. Looks like rueidislock doesn't do that. Was it intentional not to implement it?

• Client-side caching in Redis suggests flushing local cache when client loses connection with server. Looks like it's not implemented in redis/rueidis. Is my understanding correct? I'm worried about holding stale data indefinitely in the case where

  1. Primary node goes down.
  2. Elasticache is promoting a replica to primary. It takes a few seconds and while it does that all writes to Redis are failing but writing to the main persistent database will succeed.
  3. Replica becomes primary but now it holds stale data for the items where writes were failing. So does all the local caches.

  Is there a good workaround for this other than setting a short TTL on both server and client side?

Thanks again,
Iqram

[rueian]

Hi @iqram-firebolt,

For your first question, rueidislock doesn't use random delay not only because I found that the suggestion:

so ideally the client should try to send the SET commands to the N instances at the same time using multiplexing (https://redis.io/docs/manual/patterns/distributed-locks/#retry-on-failure)

still frequently results in conditions where nobody wins EVEN with random delay. Instead, rueidislock sends the SET commands one by one.

But also I want rueidislock to be faster to acquire locks after failures. Therefore, it retries immediately after receiving client-side caching invalidations.

---

For your second question, rueidis do flush the local cache when losing connection with servers here:

rueidis/pipe.go

Line 323 in 8410f48

p.cache.Close(ErrDoCacheAborted)

In terms of your Redis and Database consistency concerns, the consistency is typically handled by Change Data Capture solutions, such as https://github.com/rueian/pgcapture.

[iqram-firebolt]

Perfect! Thank you @rueian.