Reserving CPU resource in CPU inference

[sunxiaoxia2022]

Details:

• Add property ov::hint::enable_cpu_reservation to reserve CPU resource in CPU inference
• ov::hint::enable_cpu_reservation defaults to false, user can explicitly set it to true to enable CPU reservation.
• update proc_type_table before stream scheduling in compile_model()

Tickets:

• CVS-155268
• [Performance]: How to assign model inference to specific CPUs? #27083

[wangleis]

Why we need this logic here? User can set enable_cpu_reservation to true and enable_cpu_pinning to false.

[sunxiaoxia2022]

I think if enable_cpu_reservation is true, enable_cpu_pinning must be true. Because reserving cpus means pinning tasks to fixed cpus firstly, then reserving these cpus.

[wangleis]

User still can use enable_cpu_reservation true with enable_cpu_pinning false together. Then the next model only can use remained CPU cores for latency/throughput hit.

[sunxiaoxia2022]

Ok, done.

[sunxiaoxia2022]

@wangleis Could you please clarify why we need #28117 to be part of this PR?

@dmitry-gorokhov There is two changes in #28117 for CPU reservation.

1. OV RT will identify the numa node of app thread and update proc_type_table to create inference stream on same numa node. In CPU reservation use case, user may load different models on different threads. So move identifying and updating work from loading OV core stage to compiling model stage.
2. On multiple sockets platform, if same app thread load two models on same socket with latency hint, the first model will reserve all CPU core of this socket. The logic of master branch will continue loading second model to same socket and loading fail due to no CPU resource. Update proc_type_table in compile_model() #28117 will fix this issue and load second model to another socket which has CPU resource.

I am wondering how parallel model compilation will work in that case? Like if both compilation processes tries to update proc_table. Given proc_table is singleton how do we guarantee thread safity?

@dmitry-gorokhov I added _streams_executor_mutex to guarantee the thread safety when multiple models compilation are in progress at the same time. Please have a look.

[sunxiaoxia2022]

We need to add some behavior tests that will cover cpu_reservation logic. Like we should validate that ov throws error if no available CPUs left during compilation. We also need to implement basic test that compiles to model with cpu_reservation=true and specific nthreads, and then check that compiled model parameters (like nthreads and cpu_reservation) via get_property API.

@dmitry-gorokhov These test cases are added in streams_e2e_test.cpp, please have a look, thank you!

[dmitry-gorokhov]

@wangleis Could you please clarify why we need #28117 to be part of this PR?

@dmitry-gorokhov There is two changes in #28117 for CPU reservation.

1. OV RT will identify the numa node of app thread and update proc_type_table to create inference stream on same numa node. In CPU reservation use case, user may load different models on different threads. So move identifying and updating work from loading OV core stage to compiling model stage.
2. On multiple sockets platform, if same app thread load two models on same socket with latency hint, the first model will reserve all CPU core of this socket. The logic of master branch will continue loading second model to same socket and loading fail due to no CPU resource. Update proc_type_table in compile_model() #28117 will fix this issue and load second model to another socket which has CPU resource.

I am wondering how parallel model compilation will work in that case? Like if both compilation processes tries to update proc_table. Given proc_table is singleton how do we guarantee thread safity?

@dmitry-gorokhov I added _streams_executor_mutex to guarantee the thread safety when multiple models compilation are in progress at the same time. Please have a look.

get_proc_type_table() is called in several places, not only in get_num_streams(). So how the mutex here actually helps?
Like how we can guarantee correct behavior of IStreamsExecutor::Config::get_default_num_streams() in case proc_type_table is beign updated concurantely?

[dmitry-gorokhov]

We need to add some behavior tests that will cover cpu_reservation logic. Like we should validate that ov throws error if no available CPUs left during compilation. We also need to implement basic test that compiles to model with cpu_reservation=true and specific nthreads, and then check that compiled model parameters (like nthreads and cpu_reservation) via get_property API.

@dmitry-gorokhov These test cases are added in streams_e2e_test.cpp, please have a look, thank you!

streams_e2e_test.cpp contains unit tests for proc_type_table. I was talking about behavior tests that validate whole plugin behavior.

[sunxiaoxia2022]

We need to add some behavior tests that will cover cpu_reservation logic. Like we should validate that ov throws error if no available CPUs left during compilation. We also need to implement basic test that compiles to model with cpu_reservation=true and specific nthreads, and then check that compiled model parameters (like nthreads and cpu_reservation) via get_property API.

@dmitry-gorokhov These test cases are added in streams_e2e_test.cpp, please have a look, thank you!

streams_e2e_test.cpp contains unit tests for proc_type_table. I was talking about behavior tests that validate whole plugin behavior.

@dmitry-gorokhov Ok, added test case in src/plugins/intel_cpu/tests/functional/custom/behavior/ov_executable_network/properties.cpp. Please have a look.