control hub cpu load

[b-spruck]

I wonder a bit about the huge CPU load the control hub is giving us. But for a process which just redirects data packets I would not expect an excessive CPU usage, even so the amount of monitoring is significant.

Setup: ~28 IOCs to 28 boards
Each IOC queries several registers with ~1 Hz rate; the load per IOC is <1-2% CPU normally
The control hup is using 150-250 % cpu, and up to 600% if we do more than the basic monitoring.
Which was surprising.

As I didnt see any indication here that this is "known" and what to expect, is this a known behaviour?

[tswilliams]

Hi. The main measurements of CPU utilisation that were performed when the ControlHub was initially developed focussed on continual polling and continual block reads/writes. In this case, the frequency of 'continual' polling was at least ~ 1kHz, so higher than what you mentioned, but for ~ 10 board the CPU usage was of a similar order of magnitude to what you report.

[b-spruck]

My concern is how the load scales if we double the amount of hardware ... it may mean that we need 12 core machine just for the control hub.
I still wonder how the load can be so large, is this a feature of the erlang and its "guaranteed" reaction times?
Tracing the threads a bit results in a lot of calls to sched_yield (90% of the system calls).

[tswilliams]

I still wonder how the load can be so large, is this a feature of the erlang and its "guaranteed" reaction times?
Tracing the threads a bit results in a lot of calls to sched_yield (90% of the system calls).

Indeed, I think that the larger-than-expected CPU usage is related to Erlang trying to avoid large latencies. When you double the number of boards being controlled, I would not expect the CPU usage to double for the same performance - instead I'd expect that the CPU cycles are used more efficiently by the ControlHub.