-
|
Hello, I have an application composed of 3 services behind an API Gateway developed with YARP. Both the APIGateway and 2 of the services are containerised and deployed on Kubernetes (AKS), but the third service is deployed on two Windows virtual machines because of its dependencies. This service runs on IIS and the tasks it runs require a lot of CPU and memory usage and take quite a long time, some of them more than 5 minutes.
The point is that with low traffic to the VMs everything works properly, but when I run load tests the APIGateway returns a lot of errors (0, 502, 503 and 504). Looking at the logs it is the APIGateway that generates these errors because many of them in the VMs run without any problem. For example, this is the trace of an error when the APIGateway returns a 0: I have tried to run the same load tests, but directly against the VMs, and they end up fine, some occasional errors when there are many concurrent requests, but nothing to worry about. So I think the problem is in the APIGateway with YARP. The version of Yarp.ReverseProxy is 2.1.0 with .Net 7. And this is the configuration (I have changed the names of the services): Any suggestions or where to start researching? Thank you very much |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
I have done a test that I think may provide some useful information. I have created an endpoint that does nothing except a delay of 6 minutes. public async Task<ActionResult<ImportPartResponseDto>> ImportPartModel([CoddedNotNull] ImportPartRequestDto importPartRequestDto)
{
logger.LogInformation("Starting test endpoint");
await Task.Delay(TimeSpan.FromMinutes(6));
logger.LogInformation("Test endpoint ended");
return NoContent();
}I have run against this endpoint the same tests (time and Virtual Users) that against other endpoints that generates a lot of errors and the result has been that there have been hardly any errors, as when I launch them directly against the VMs. The impression is that when the VMs are close to collapse the API Gateway generates errors.... |
Beta Was this translation helpful? Give feedback.
-
Is this the whole exception (no other messages / inner exceptions)? Judging based just on the stack trace, YARP decided to cancel the request before receiving the response headers. This generally means that some sort of timeout was hit, or the client talking to the proxy canceled the request (e.g. disconnected). |
Beta Was this translation helpful? Give feedback.
-
|
Hello, first of all thank you for your reply.
The trace I have put is all I have, or at least all I have found in AppInsights and log files.
I don't know exactly what you mean by
In the application I don't have any timeout, in fact when I run the tests directly against the virtual machines I don't have so many errors. |
Beta Was this translation helpful? Give feedback.
-
|
For YARP logs, you could expect to see something like where If you were doing it from code, you can see the await next();
if (context.Features.Get<IForwarderErrorFeature>() is { } errorFeature)
{
ForwarderError error = errorFeature.Error;
Exception exception = errorFeature.Exception;
}
Have you looked at |
Beta Was this translation helpful? Give feedback.
For YARP logs, you could expect to see something like
where
RequestCanceledis theForwarderErrorenum.If you were doing it from code, you can see the
ForwarderErrorin a middleware like so: