[VPP-1866] VPP crashes when trying to program huge number of ACLs

[vvalderrv]

When trying to program a huge number of ACLs (upwards of 50K), VPP crashes and the backtrace indicates it has run out of memory:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff5ad1801 in __GI_abort () at abort.c:79
#2  0x000055555555be5b in os_panic () at /root/vpp/src/vpp/vnet/main.c:355
#3  0x00007ffff5f487e4 in os_out_of_memory () at /root/vpp/src/vppinfra/unix-misc.c:221
#4  0x00007ffff5f51198 in clib_mem_alloc_aligned_at_offset (size=11688276, align=8, align_offset=8, os_out_of_memory_on_failure=1) at /root/vpp/src/vppinfra/mem.h:143
#5  0x00007ffff5f51499 in vec_resize_allocate_memory (v=0x7fffa7a14290, length_increment=1, data_bytes=7792288, header_bytes=8, data_align=8)

    at /root/vpp/src/vppinfra/vec.c:89
#6  0x0000555555568a91 in _vec_resize_inline (v=0x7fffa7a14290, length_increment=1, data_bytes=7792280, header_bytes=0, data_align=8) at /root/vpp/src/vppinfra/vec.h:148
#7  0x0000555555569d3d in vlib_stats_pop_heap (cm_arg=0x7fffb8fa3ae8, oldheap=0x7fffb4bda010, cindex=2, type=STAT_DIR_TYPE_COUNTER_VECTOR_COMBINED)

    at /root/vpp/src/vpp/stats/stat_segment.c:125
#8  0x00007ffff63e2ee4 in vlib_validate_combined_counter (cm=0x7fffb8fa3ae8, index=2) at /root/vpp/src/vlib/counter.c:104
#9  0x00007fffb476a4ff in validate_and_reset_acl_counters (am=0x7fffb49c5520 <acl_main>, acl_index=48099) at /root/vpp/src/plugins/acl/acl.c:355
#10 0x00007fffb476b60c in acl_add_list (count=2, rules=0x7fffb7b82fa2, acl_list_index=0x7fffb54ced78,

    tag=0x7fffb7b82f5e "net-vpp.secgroup:3ba9f6da-8940-11ea-a2da-0025b5aa0150.to-vpp") at /root/vpp/src/plugins/acl/acl.c:543
#11 0x00007fffb4772133 in vl_api_acl_add_replace_t_handler (mp=0x7fffb7b82f50) at /root/vpp/src/plugins/acl/acl.c:1957
#12 0x00007ffff7bc2a03 in msg_handler_internal (am=0x7ffff7dd3e60 <api_global_main>, the_msg=0x7fffb7b82f50, trace_it=1, do_it=1, free_it=0)

    at /root/vpp/src/vlibapi/api_shared.c:488
#13 0x00007ffff7bc32df in vl_msg_api_socket_handler (the_msg=0x7fffb7b82f50) at /root/vpp/src/vlibapi/api_shared.c:755
#14 0x00007ffff7ba16f4 in vl_socket_process_api_msg (rp=0x7fffb6bf5b10, input_v=0x7fffb7b82f40 "") at /root/vpp/src/vlibmemory/socket_api.c:199
#15 0x00007ffff7bae260 in vl_api_clnt_process (vm=0x7ffff66be680 <vlib_global_main>, node=0x7fffb54c6000, f=0x0) at /root/vpp/src/vlibmemory/vlib_api.c:394
#16 0x00007ffff641f200 in vlib_process_bootstrap (_a=140736247204816) at /root/vpp/src/vlib/main.c:1475
#17 0x00007ffff5ed4fbc in clib_calljmp () at /root/vpp/src/vppinfra/longjmp.S:123
#18 0x00007fffb6058ba0 in ?? ()
#19 0x00007ffff641f308 in vlib_process_startup (vm=0x7ffff641fcab <dispatch_suspended_process+818>, p=0x7fffb6058ca0, f=0xffffffffffffffff) at /root/vpp/src/vlib/main.c:1497
#20 0x0053d56c41112ce2 in ?? ()
#21 0x00007fffb54c6000 in ?? ()
#22 0x00007fffb6be4e78 in ?? ()
#23 0x00007fffb6be4c08 in ?? ()
#24 0x0000000000000018 in ?? ()
#25 0x00007fffb6be4e78 in ?? ()
#26 0x00007fffb54c6000 in ?? ()
#27 0x00007fffb566e4f4 in ?? ()
#28 0x0000000000000000 in ?? ()
(gdb)

 

This is a bit surprising as the system in question has lots of free memory. After discussion with Andrew, it became clear that it is the ACL plugin's private heap/memory pool which has run out of memory. This can be easily addressed by setting a higher heap/memory pool size in the ACL section of the VPP startup config file.

However, two things would be really nice:

1. a show command to display the ACL plugin's heap/memory pool size and current usage
2. Clear documentation explaining about ACL plugin's heap/memory pool and how to configure it