[VPP-1569] VPP from ARM packages doesn't start on Taishan

[vvalderrv]

testuser@huawei-1:~$ dpkg -l | grep vpp
ii libvppinfra 19.04-rc0~181-g381735028~b374 arm64 Vector Packet Processing--runtime libraries
ii vpp 19.04-rc0~181-g381735028~b374 arm64 Vector Packet Processing--executables
ii vpp-dbg 19.04-rc0~181-g381735028~b374 arm64 Vector Packet Processing--debug symbols
ii vpp-plugin-core 19.04-rc0~181-g381735028~b374 arm64 Vector Packet Processing--runtime core plugins
ii vpp-plugin-dpdk 19.04-rc0~181-g381735028~b374 arm64 Vector Packet Processing--runtime dpdk plugin

I had to remove the gid config from the default startup.conf to get a core (used config attached). This crash happens only with packages from packagecloud, which is why this is a regular (no-debug) build.

This crash is not reproducible when I disable plugins, suggesting dpdk is involved in some fashion.

vpp crashes right after starting:
testuser@huawei-1:~$ sudo /usr/bin/vpp -c /etc/vpp/startup_no_gid.conf
vlib_plugin_early_init:361: plugin path /usr/lib/aarch64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins
load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
load_one_plugin:191: Loaded plugin: cdp_plugin.so
load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))
load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet)
load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy)
load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GTPv1-U)
load_one_plugin:189: Loaded plugin: igmp_plugin.so (IGMP messaging)
load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier-locator addressing for IPv6)
load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound OAM)
load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
load_one_plugin:189: Loaded plugin: l2e_plugin.so (L2 Emulation)
load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol)
load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer)
load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC source-address filter)
load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of address and port (MAP))
load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimental))
load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation)
load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header)
load_one_plugin:189: Loaded plugin: nsim_plugin.so (network delay simulator plugin)
load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so
load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPPoE)
load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 proxy)
load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy)
load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration)
load_one_plugin:189: Loaded plugin: svs_plugin.so (Source VRF Select)
load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine)
load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (openssl based TLS Engine)
load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Plugin)
/usr/bin/vpp[4218]: clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
/usr/bin/vpp[4218]: load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
/usr/bin/vpp[4218]: dpdk: EAL init args: -c 2 -n 4 --in-memory --file-prefix vpp --master-lcore 1
Aborted

 

testuser@huawei-1:~$ sudo gdb /usr/bin/vpp core
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/vpp...Reading symbols from /usr/lib/debug/.build-id/c8/805f122c864ca91f6d96a8fd88b4139f429038.debug...done.
done.
[New LWP 4218]
[New LWP 4219]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/vpp -c /etc/vpp/startup_no_gid.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0xffff97a15010 (LWP 4218))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x0000ffff9715a8b4 in __GI_abort () at abort.c:79
#2 0x0000aaaad3eb946c in os_exit (code=code@entry=1) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vpp/vnet/main.c:359
#3 0x0000ffff973648b4 in unix_signal_handler (signum=<optimized out>, si=<optimized out>, uc=<optimized out>) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/unix/main.c:156
#4 <signal handler called>
#5 dpdk_buffer_pool_init (vm=vm@entry=0xffff97382300 <vlib_global_main>, bp=bp@entry=0xffff57313b00) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c:70
#6 0x0000ffff56590154 in dpdk_buffer_pools_create (vm=vm@entry=0xffff97382300 <vlib_global_main>) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c:392
#7 0x0000ffff565a0794 in dpdk_config (vm=<optimized out>, input=<optimized out>) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/device/init.c:1384
#8 0x0000ffff9733c624 in vlib_call_all_config_functions (vm=0xffff97382300 <vlib_global_main>, input=input@entry=0xffff577fffc8, is_early=65535, is_early@entry=0) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/init.c:146
#9 0x0000ffff973481d4 in vlib_main (vm=<optimized out>, vm@entry=0xffff97382300 <vlib_global_main>, input=input@entry=0xffff577fffc8) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/main.c:2073
#10 0x0000ffff97363964 in thread0 (arg=281473218781952) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/unix/main.c:606
#11 0x0000ffff972a5164 in clib_calljmp () from /usr/lib/aarch64-linux-gnu/libvppinfra.so.19.04
Backtrace stopped: not enough registers or memory available to unwind further
(gdb) frame 5
#5 dpdk_buffer_pool_init (vm=vm@entry=0xffff97382300 <vlib_global_main>, bp=bp@entry=0xffff57313b00) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c:70
70 /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c: No such file or directory.
(gdb) p *bp
$1 = {cacheline0 = 0xffff57313b00 "", start = 281455030697984, size = 46137344, log2_page_size = 0, index = 2 '\002', numa_node = 2, physmem_map_index = 2, data_size = 2048, n_buffers = 17160, buffers = 0xffff570d1f80, name = 0xffff5730f0ac "default-numa-2",
lock = 0xffff56ecc280, threads = 0xffff572adc00, buffer_template = {{cacheline0 = 0xffff57313b80 "", current_data = 0, current_length = 0, flags = 0, flow_id = 0, ref_count = 1 '\001', buffer_pool_index = 2 '\002', error = 0, next_buffer = 0,
current_config_index = 0, opaque = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, template_end = 0xffff57313bc0 "", cacheline1 = 0xffff57313c00 "", trace_index = 0, total_length_not_including_first_buffer = 0, opaque2 = {0 <repeats 14 times>}, cacheline2 = 0xffff57313c80 "",
pre_data = '\000' <repeats 127 times>, data = 0xffff57313d00 ""}, as_u8x16 = {

{0 <repeats 12 times>, 1, 2, 0, 0}

, {0 <repeats 16 times>}, {0 <repeats 16 times>}, {0 <repeats 16 times>}}}}
(gdb) p *vm
$2 = {cacheline0 = 0xffff97382300 <vlib_global_main> "\177\023V\005", clib_time =

{total_cpu_time = 89527167, last_cpu_time = 101065847232, clocks_per_second = 50000000, seconds_per_clock = 2e-08, round_to_units = 100000, init_cpu_time = 100976320065, last_verify_cpu_time = 101065847232, last_verify_reference_time = 1550137076.6638808, log2_clocks_per_second = 25, log2_clocks_per_frequency_verify = 26}

, cpu_time_last_node_dispatch = 0, cpu_time_main_loop_start = 0, main_loop_count = 0,
main_loop_vectors_processed = 0, main_loop_nodes_processed = 0, vector_counts_per_main_loop = {0, 0}, node_counts_per_main_loop = {0, 0}, vlib_node_runtime_perf_counter_cb = 0x0, main_loop_exit_set = 1, main_loop_exit_now = 0, main_loop_exit = {regs = {
281472149749704, 281472158292044, 8, 281473218781952, 16, 281473218202352, 281473225676568, 1073741824, 0, 0, 281472149749344, 281473218543912, 281472149749344, 0, 0, 0, 0, 0, 0, 0, 0, 0}}, main_loop_error = 0x0, name = 0xaaaaebcc08c0 "/usr/bin/vpp",
heap_base = 0xffff56e0c010, heap_aligned_base = 0xffff56e0c000, heap_size = 0, buffer_main = 0xffff57458524, physmem_main = {flags = 1, maps = 0xffff5731753c, pmalloc_main = 0xffff57311f80}, node_main =

{nodes = 0xffff57570a4c, node_by_name = 0xffff57367954, flags = 1, nodes_by_type =

{0xffff57536580, 0xffff5711e200, 0xffff5736d600, 0x0}

, pending_interrupt_node_runtime_indices = 0x0, pending_interrupt_lock = 0x0, polling_threshold_vector_length = 0, interrupt_threshold_vector_length = 0, next_frames = 0xffff57ddf94c,
pending_frames = 0x0, timing_wheel = 0xffff58022780, signal_timed_event_data_pool = 0x0, data_from_advancing_timing_wheel = 0xffff5802580c, time_next_process_ready = 0, processes = 0xffff570f566c, current_process_index = 0, suspended_process_frames = 0x0,
recycled_event_data_vectors = 0x0, input_node_counts_by_state = {0, 11, 12}, frame_size_hash = 0xffff5736d75c, frame_sizes = 0x0, time_last_runtime_stats_clear = 0, node_registrations = 0xffff55c244b8 <vmxnet3_input_node>}, cli_main =

{commands = 0xffff57fcd63c, command_index_by_path = 0xffff57fe8354, parse_rules = 0x0, parse_rule_index_by_name = 0x0, parse_rule_data = 0x0, cli_command_registrations = 0xffff55c24438 <show_vmxnet3_command>}

, trace_main =

{trace_buffer_pool = 0x0, last_main_loop_count = 0, filter_node_index = 0, filter_flag = 0, filter_count = 0, trace_enable = 0, nodes = 0x0, verbose = 0}

, dispatch_pcap_main =

{lock = 0x0, file_name = 0x0, n_packets_to_capture = 0, packet_type = PCAP_PACKET_TYPE_null, n_packets_captured = 0, flags = 0, file_descriptor = 0, n_pcap_data_written = 0, pcap_data = 0x0, packets_read = 0x0, timestamps = 0x0, min_packet_bytes = 0, max_packet_bytes = 0}

, dispatch_pcap_enable = 0, pcap_buffer = 0x0, error_main =

{counters = 0xfffb53a37ccc, counters_last_clear = 0x0, error_strings_heap = 0xffff5757e180}

, os_punt_frame = 0x0, mc_stream_index = 0, procs_waiting_for_mc_stream_join = 0x0, elog_main = {n_total_events = 0, n_total_events_disable_limit = 4294967295, dummy_event = {{time_cycles = 0, time = 0}, type = 0, track = 0,
data = '\000' <repeats 19 times>}, event_ring_size = 131072, event_ring = 0xffff57800080, event_types = 0x0, event_type_by_format = 0x0, string_table = 0xffff57362bdc "null-node", tracks = 0xffff5706040c, default_track =

{name = 0xffff972da920 "default", track_index_plus_one = 1}

, cpu_timer =

{total_cpu_time = 0, last_cpu_time = 100976395167, clocks_per_second = 50000000, seconds_per_clock = 2e-08, round_to_units = 0, init_cpu_time = 100976395167, last_verify_cpu_time = 100976395167, last_verify_reference_time = 1550137074.8748384, log2_clocks_per_second = 25, log2_clocks_per_frequency_verify = 25}

, init_time = {cpu = 100976395558, os_nsec = 59251966874846384}, serialize_time = {cpu = 0, os_nsec = 0}, lock = 0x0, nsec_per_cpu_clock = 0,
events = 0x0}, elog_trace_api_messages = 0, elog_trace_cli_commands = 0, elog_trace_graph_dispatch = 0, elog_trace_graph_circuit = 0, elog_trace_graph_circuit_node_index = 0, node_call_elog_event_types = 0xffff575f5ebc,
node_return_elog_event_types = 0xffff57c000bc, error_elog_event_types = 0xffff57c0968c, random_seed = 100981582619, random_buffer = {ctx = {{memory =

{2620497467346831672, 3244870759197145983, 16844733809704711351, 15162463060836654054, 5540841756506599954, 9919435146587345174, 3030489230478709167, 1438290055871884815, 643337747533210493, 105795752944511344, 13771037136446541908, 16386983901594699599, 8499979750853252571, 15039116162850586550, 3403607821893519355, 15502314087502703790}

, a = 0, b = 0, c = 0}, {
memory =

{13950185659467432195, 4935278134164438092, 1829907392312249798, 14931966196221913508, 9235915728095897059, 2387164126153048165, 8582152180194635212, 13557094843362847433, 15386456528893374202, 14395740752707628947, 6645327715385977895, 14901116945941576631, 11516234572391798138, 16433667478103746546, 3715748946614034106, 11736420597923239612}

, a = 0, b = 0, c = 0}}, buffer = 0x0, n_cached_bytes = 0, {cached_bytes = "\000\000\000\000\000\000\000", cached_word = 0}},
init_functions_called = 0xffff57ee40b4, thread_index = 0, cpu_id = 0, numa_node = 0, init_function_registrations = 0xffff55c246c0 <_vlib_init_function.35314>, worker_init_function_registrations = 0xffff97991950 <_vlib_init_function.38576>,
main_loop_enter_function_registrations = 0xffff979913d8 <_vlib_init_function.38053>, main_loop_exit_function_registrations = 0xaaaad3f6a758 <_vlib_init_function.33510>, api_init_function_registrations = 0xffff55dff640 <_vlib_init_function.40989>,
config_function_registrations = 0xffff55ef7db8 <_vlib_config_function_snat_config>, queue_signal_pending = 0, api_queue_nonempty = 0, queue_signal_callback = 0xffff979b8530 <memclnt_queue_callback>, argv = 0xaaaaebcc08e0, worker_thread_main_loop_callback = 0x0,
parked_at_barrier = 0, elog_post_mortem_dump = 0, need_vlib_worker_thread_node_runtime_update = 1, barrier_epoch = 0, barrier_no_close_before = 0, pending_rpc_requests = 0xffff5800b70c, processing_rpc_requests = 0xffff5802584c, pending_rpc_lock = 0x0}
(gdb) down
#4 <signal handler called>
(gdb) down
#3 0x0000ffff973648b4 in unix_signal_handler (signum=<optimized out>, si=<optimized out>, uc=<optimized out>) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/unix/main.c:156
156 /w/workspace/vpp-arm-merge-master-ubuntu1804/src/vlib/unix/main.c: No such file or directory.
(gdb) up
#4 <signal handler called>
(gdb)
#5 dpdk_buffer_pool_init (vm=vm@entry=0xffff97382300 <vlib_global_main>, bp=bp@entry=0xffff57313b00) at /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c:70
70 /w/workspace/vpp-arm-merge-master-ubuntu1804/src/plugins/dpdk/buffer.c: No such file or directory.
(gdb) info locals
buffer_mem_start = 281454938423296
mp = 0x0
nmp = 0x0
priv = {mbuf_data_room_size = 2176, mbuf_priv_size = 256}
iova_mode = <optimized out>
bi = <optimized out>
name = <optimized out>
elt_size = <optimized out>
(gdb) q