GetOrSetAsync for a cacheKey hangs for an infinite time

[AntonSmolkov]

What does the bug do?

GetOrSetAsync call for particular cacheKey sometimes hangs for an infinite time.

I think the reason of the behaviour is this TaskCompletionSource which can be created, for intance, when 2 threads getting value for a first time in parallel (cache miss), and thread A goes to this branch of code , thread B goes to this branch of code, sleeps for a while, and then calls KeyLock.WaitAsync(cacheKey) when the KeyLock is already released by thread A.

Then, Thread B waits for the TaskCompletionSource just created by itself, and from the thread perspective it looks like infinite call of GetOrSetAsync method, and only another thread calling GetOrSetAsync with the same cacheKey can release Thread B from this block.
Not sure, but it looks like even this another thread call won't help if it happens after cacheEntry (created by thread A) is expired.

How can it be reproduced?

In my case it could only be reproduced under (relatively) heavy load.

Environment (if applicable)

• NuGet Package Version: 0.14.0
• .NET Runtime Version: .NET 8
• Operating System: Linux

[AntonSmolkov]

dotnet-dump analyze => dumpasync of my stuck application seems to confirm the assumption

...
...
...
STACK 47
00007f82a6e7d0f8 00007f85935ebab8 ( ) System.Threading.Tasks.Task<CacheTower.ICacheEntry>
  00007f82a6e7d140 00007f8593608078 (7) CacheTower.CacheStack+<RefreshValueAsync>d__19<Zyfra.Udl.Cmn.Models.DataSources.DsConfigExtDto>
    00007f82a6e7c060 00007f85935e34a8 (1) CacheTower.CacheStack+<GetOrSetAsync>d__17<Zyfra.Udl.Cmn.Models.DataSources.DsConfigExtDto>
      00007f82a6e7c130 00007f85935e4878 (0) Zyfra.Udl.DataSources.Providers.PropertyConfigProvider+<GetPropertyConfigFromCacheAsync>d__16
        00007f82a6e7d018 00007f85935ec040 (4) Zyfra.Udl.Rtd.Core.RabbitMqSubscriptionManager+<AddConsumerToPropertySubscriptionAsync>d__35
          00007f82a68c99c8 00007f8592c5ea28 (0) Zyfra.Udl.Rtd.Core.RabbitMqSubscriptionManager+<HandleRtdCommandsAsync>d__33
            00007f82a68c7c28 00007f85927fbb30 ( ) System.Threading.Tasks.UnwrapPromise<System.Threading.Tasks.VoidTaskResult>
...
...
...

[adamtash]

We are experiencing the same issue in a similar environment

[AntonSmolkov]

We are experiencing the same issue in a similar environment

We ended up switching to FusionCache, satisfied so far