Crashes on route creation

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa49ab393ea in rte_stack_pop (s=0x0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/stack/rte_stack.h:140

warning: 140    ../lib/stack/rte_stack.h: No such file or directory
[Current thread is 1 (LWP 9)]
#0  0x00007fa49ab393ea in rte_stack_pop (s=0x0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/stack/rte_stack.h:140
#1  stack_dequeue (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../drivers/mempool/stack/rte_mempool_stack.c:59
#2  0x00007fa49d316650 in rte_mempool_ops_dequeue_bulk (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/mempool/rte_mempool.h:793
#3  rte_mempool_do_generic_get (mp=<optimized out>, obj_table=0x7fa4985dc9e8, n=<optimized out>, cache=<optimized out>) at ../lib/mempool/rte_mempool.h:1608
#4  rte_mempool_generic_get (mp=<optimized out>, obj_table=<optimized out>, n=<optimized out>, cache=<optimized out>) at ../lib/mempool/rte_mempool.h:1661
#5  rte_mempool_get_bulk (mp=0x11c669b5c0, obj_table=0x7fa4985dc9e8, n=1) at ../lib/mempool/rte_mempool.h:1696
#6  rte_mempool_get (mp=0x11c669b5c0, obj_p=0x7fa4985dc9e8) at ../lib/mempool/rte_mempool.h:1722
#7  node_alloc (rib=rib@entry=0x11c7894f40) at ../lib/rib/rte_rib.c:91
#8  0x00007fa49d316ac5 in rte_rib_insert (rib=rib@entry=0x11c7894f40, ip=ip@entry=184156216, depth=depth@entry=32 ' ') at ../lib/rib/rte_rib.c:274
#9  0x000055b64732aa3c in dp_add_route (port=0x55b65a638490, vni=<optimized out>, t_vni=207, ip=184156216, t_ip6=t_ip6@entry=0x7fa4985dcae0, depth=<optimized out>) at ../src/dp_lpm.c:101
#10 0x000055b6472e0c4b in dp_grpc_add_route (port=<optimized out>, vni=<optimized out>, t_vni=<optimized out>, addr=0x7fa4985dcba8, t_ip6=0x7fa4985dcae0, depth=<optimized out>)
    at ../src/grpc/dp_grpc_impl.c:43
#11 dp_process_create_route (responder=0x7fa4985dcba0) at ../src/grpc/dp_grpc_impl.c:578
#12 dp_process_request (m=<optimized out>) at ../src/grpc/dp_grpc_impl.c:999
#13 0x000055b64731a8fd in handle_nongraph_queues () at ../src/nodes/rx_periodic_node.c:52
#14 rx_periodic_node_process (graph=0x11ddd14100, node=0x11ddd22f80, objs=0x11ddcd0100, nb_objs=<optimized out>) at ../src/nodes/rx_periodic_node.c:74
#15 0x000055b6473320ca in __rte_node_process (graph=0x11ddd14100, node=0x11ddd22f80) at /usr/local/include/rte_graph_worker_common.h:210
#16 rte_graph_walk_rtc (graph=<optimized out>) at /usr/local/include/rte_graph_model_rtc.h:42
#17 rte_graph_walk (graph=0x11ddd14100) at /usr/local/include/rte_graph_worker.h:38
#18 graph_main_loop (arg=<optimized out>) at ../src/dpdk_layer.c:171
#19 0x00007fa49ce4967e in eal_thread_loop (arg=<optimized out>) at ../lib/eal/common/eal_common_thread.c:199
#20 0x00007fa49ce59829 in eal_worker_thread_loop (arg=<optimized out>) at ../lib/eal/linux/eal.c:517
#21 0x00007fa49c451b7b in ?? ()
#22 0x0000000000000000 in ?? ()

We are running stress tests and the types of errors are piling up, this may even be connected to #621

It is starting to look like some memory corruption which is the simply manifesting in these "random" crashes and leads us on wild goose chases.

@FlorinPeter is implementing an AI environment that will look through dumps and run dpservice under debug.

Any fixes found this way will be linked to this issue.

Start with the stack trace in src/dp_lpm.c, src/grpc/dp_grpc_impl.c, and lib/rib/rte_rib.c, then inspect node_alloc and the mempool path in lib/mempool/rte_mempool_stack.c. Reproduce the failure with the stress tests and use the reported dumps to determine whether route creation or earlier memory corruption is responsible. Done means the root cause is identified and route creation remains stable under stress.