Bug #6006
closedTask #5965: tracking: Improving DPDK capture interface and docs
dpdk: query eth stats only by the first worker
Description
When Suricata is under heavy load (and imissed counter is increasing heavily) it leads to buffer-overflow-like behavior when the counter suddenly has very high (unrealistic) values. Example of such behavior can be seen in the counters below when for 4 seconds packets in the imissed
counter are increasing by 500k per second and then the counter is suddenly doubled.
After further investigation, I concluded that the function rte_eth_stats_get
is not multi-thread safe. (as the behavior was neither reproduced when Suricata was not using this function at all nor when only the first worker used the function).
At the same time, the result of this function is not used anywhere for non-first workers. Therefore, suggested fix is to enable the usage of the stats retrieval to the first worker only.
$ clear && cat /tmp/stats.log | grep -E "Date|capture" Date: 4/18/2023 -- 12:09:53 (uptime: 0d, 00h 00m 08s) Date: 4/18/2023 -- 12:09:55 (uptime: 0d, 00h 00m 10s) capture.packets | Total | 4319079841 capture.rx_errors | Total | 4312173989 capture.dpdk.imissed | Total | 4312173989 Date: 4/18/2023 -- 12:09:57 (uptime: 0d, 00h 00m 12s) capture.packets | Total | 4351338655 capture.rx_errors | Total | 4334083449 capture.dpdk.imissed | Total | 4334083449 Date: 4/18/2023 -- 12:09:59 (uptime: 0d, 00h 00m 14s) capture.packets | Total | 4351338655 capture.rx_errors | Total | 4334083449 capture.dpdk.imissed | Total | 4334083449 Date: 4/18/2023 -- 12:10:01 (uptime: 0d, 00h 00m 16s) capture.packets | Total | 8678269587 capture.rx_errors | Total | 8650362891 capture.dpdk.imissed | Total | 8650362891 Date: 4/18/2023 -- 12:10:03 (uptime: 0d, 00h 00m 18s) capture.packets | Total | 8710930232 capture.rx_errors | Total | 8672352621 capture.dpdk.imissed | Total | 8672352621 Date: 4/18/2023 -- 12:10:05 (uptime: 0d, 00h 00m 20s) capture.packets | Total | 8710930232 capture.rx_errors | Total | 8672352621 capture.dpdk.imissed | Total | 8672352621 Date: 4/18/2023 -- 12:10:07 (uptime: 0d, 00h 00m 22s) capture.packets | Total | 17334045626 capture.rx_errors | Total | 17284806839 capture.dpdk.imissed | Total | 17284806839 Date: 4/18/2023 -- 12:10:09 (uptime: 0d, 00h 00m 24s) capture.packets | Total | 17366979962 capture.rx_errors | Total | 17307296380 capture.dpdk.imissed | Total | 17307296380