Memory leak or creeping memory usage over time

[danieldaquino]

notepush: c67a0b5e06b8c0cef351be8efeb0f0fbdd0e0c55

Notepush was observed in production to use higher and higher amounts of memory after a long uptime. We are probably not managing memory well enough and it is using increasing amounts of memory over time.

[jb55]

Daniel D’Aquino ***@***.***> writes:

notepush: `c67a0b5e06b8c0cef351be8efeb0f0fbdd0e0c55` Notepush was observed in production to use higher and higher amounts of memory after a long uptime. We are probably not managing memory well enough and it is using increasing amounts of memory over time.

I wonder if this is the note cache? Maybe nostrdb can help here.

[danieldaquino]

Daniel D’Aquino @.***> writes:
notepush: c67a0b5e06b8c0cef351be8efeb0f0fbdd0e0c55

Notepush was observed in production to use higher and higher amounts of memory after a long uptime. We are probably not managing memory well enough and it is using increasing amounts of memory over time.
I wonder if this is the note cache? Maybe nostrdb can help here.

Yes, it seems to be the caching logic. I took a quick look at it, and I believe the culprit is our "lazy expiration" logic

Expired cache items stay in memory because no one reads them (so they never lazily clear), and once a user tries to read them, the expired cache item gets replaced with a new one. In other words, the amount of cache items can only stay stable or grow over time. Extrapolating this to infinity, memory exhaustion is basically guaranteed to happen at some point.

I think the fix might be to simply run a cache clearing function periodically instead of lazily.