Vector CPU Utilisation

[prayerx]

Hi, im trying to understand limit of vector. I read scaling documents and i should be easily see 10MB/s for each CPU core.
Im testing on my lab with a python script sending syslog and listening on vector end with socket source without transforms and sinking to blackhole.
I can reach around 70MB/s without packet drops with having 16 cpu cores, after that it starts to drop packets. When i monitor each cpu core with htop, 5 cores are working (changing which core is active) with %50 utilisation.
I'm expecting from vector to use all CPU cores to %100 (or close) before it start dropping packets. Is my expectation wrong? If not what should be the issue here?

[jszwedko]

Hi @prayerx ,

There are a few factors that can limit throughput including the concurrency model of any components involved (sources, sinks, transforms) as well as back pressure from sinks if they can't send data fast enough. In your case, you mention it is just socket source to blackhole sink. Are you sending data via UDP, TCP, or over a Unix socket? Since you said syslog traffic, I'm guessing UDP in which case there are a few relevant issues that may be limiting throughput:

• Improve Vector's throughput for UDP protocols #8518
• Excess stat calls #12451
• performance: UDP recvmmsg instead of recvfrom #12452

Importantly the socket source, for UDP traffic, is single threaded.

[prayerx]

Hi @jszwedko, thanks for your detailed answer. You guessed right, its UDP since I believed it'll perform better but maybe i'm wrong. The example threads didn't help a lot but your last sentence is very important.
I made the same test with a system with 4 CPU cores and I'm still getting same values as testing with a 16 cores system. Which alligns my utilisation problem and your last sence that UDP Socket source is single threaded.
Is there any way to check other sources and Vector's behavior about threading? I checked syslog source but there is no information about being single threaded on udp socket. It'll be very helpful to know when scaling vector for different scenarios and deciding which source to use.

[jszwedko]

Is there any way to check other sources and Vector's behavior about threading? I checked syslog source but there is no information about being single threaded on udp socket. It'll be very helpful to know when scaling vector for different scenarios and deciding which source to use.

Unfortunately there isn't much in the way of documentation on the threading model for each source. In general sources try to handle input concurrently (e.g. TCP streams in the socket source, kafka topics in the kafka source, HTTP requests in the http source, etc.), but UDP is a bit of an outlier in that everything comes into one queue and is processed sequentially.