Memory leak when spawning isolates that use HTTP

[davidmartos96]

Dart version: Dart SDK version: 3.6.0 (stable) (Thu Dec 5 07:46:24 2024 -0800) on "linux_x64"

On our production shelf server, deployed on Cloud Run, we've discovered a memory leak. The leak happens when using Isolate.run and inside the isolate callback we fetch data from an external endpoint using the http package.

Here is a reproduction repository: https://github.com/davidmartos96/dart-isolate-http-leak

The RSS memory will rapidly grow with the cli.dart and stress_server.dart scripts provided. The scripts also provide a way to disable the isolate to see the difference.

Cloud Run visualization

[mraleph]

Are you sure it is a memory leak and not a higher peak memory usage? Each time you invoke Isolate.run you need to create a fresh isolate - fresh isolate means fresh static state, etc. If you run many of these concurrently it takes time for GC to reclaim all that memory. It's only a leak if RSS does not drop back eventually.

[davidmartos96]

@mraleph Thank you for the response.

We've been trying to debug a leak on our Dart server the last weeks, where memory was starting to grow slowly but steadily. We've used a 1GB RAM configuration, and it never stopped growing until reaching the max memory and Cloud Run starting a new instance because of OOM.

We will still need to wait a couple of days to see if a recent deploy without Isolate.run fully stops the leak. Otherwise it could be something else. The server serves around 1.5 requests/second, so it's not a big server where we would expect 1 GB memory.

One thing we've been wondering while debugging these past weeks, is the implications of setting the VM flag --old_gen_heap_size=500 in the 1 GB Cloud Run machine. Would that, assuming a "non leaking" long running program like a server, make the Garbage collector try to stay below that memory, and possibly do more aggressive collects if needed?

[davidmartos96]

The endpoint on our server where we use Isolate.run is only called on some occasions, the reproduction stress script is not a representation of how our server was processing these requests, but it shows the memory growth without garbage collecting. On my local machine it kept growing past 3 GB, even when constraining the old heap size flag.

[davidmartos96]

As a follow up, since we removed the Isolate.run from that particular endpoint route, the memory usage has stabilized on the 20% of the 1 GB machine. It even drops to the 10% on some occasions. This API route is called once every 10 seconds approximately, that's why we are suspicious of some kind of memory leak.
Here is a comparison visualization with the Cloud Run charts. This is regular usage on the production server, not using the stress tools in the reproduction repository.