Troubleshooting slow upload speeds

[lejsekt]

Hi,
I'd like to ask for advice on troubleshooting slow file upload speeds. It takes almost 4 minutes before a 650 MB file gets fully buffered which is roughly 20 Mb/s. This seems rather slow given the server bandwidth should be at 1 Gb/s and my ISP limit should be at 100 Mb/s.

My setup is the following:

• Yarp as a front-facing reverse proxy with TLS termination.
• C# ASP.NET Core backend that accepts multi-part from request using the default IFormCollection.
• File uploaded from a web browser.

Both Yarp and the backend are running as docker containers. I see a tmp file in the /tmp dir of the backend container growing in size during the upload. docker stats report cca 25-40% CPU usage during the upload which seems a bit too much as well.

Any ideas on how to troubleshoot this?

Thank you

[karelz]

Some ideas from team discussion:

• Do a local test to check for server issues - upload with / without YARP, make sure that upload is buffered to fast disk.
• Test bandwidth between each 2 nodes with a simple HelloWorld style app to make sure the bandwidth is really usable.

[MihaZupan]

Also just to confirm, what version of .NET & YARP are you running on?

[lejsekt]

Thank you for your quick responses!

I'm on .NET 6 and Yarp 2.0.1. The server has NVMe SSD, so that shouldn't be the bottleneck. I'll test the uploads with or without the proxy.

[lejsekt]

Interestingly, I see a significant speed improvement, if I force the request to HTTP 1.1. That's when the proxy is used.

The proxy is handling HTTPS and was set to Http1AndHttp2 protocols, but the destination is only using HTTP. So I guess the connection to the proxy used HTTP 2, while the request to the final destination had to use HTTP 1.1.

Do you think this could be the cause of the issue?

[Tratcher]

HTTP/2 does have differences that may be relevant.

• Were you only testing one upload at time or concurrent ones? HTTP/2 manages concurrency differently.
• Is there much latency on the connection? HTTP/2 manages its own data window sizes and acks. These can be adjusted in KestrelServerOptions: https://github.com/dotnet/aspnetcore/blob/7eec26ec37f0277f13340ed959e2bec778b647fd/src/Servers/Kestrel/Core/src/Http2Limits.cs#L111-L134