Suricata

We have observed on several of our sensors that Suricata has stopped analyzing traffic, kernel packets goes to zero, but the sensor still receives packets.

Issues observed on:

• Centos 7 and Centos Stream 8
• Suricata 5.0.7 and 6.0.4 (with custom patches to SMB and a proprietary app-layer)
• Intel and Netronome NICs
• AF-packet v3
• With and without XDP
• No hardware bypass

We started to look into whether any stats continued to update after the kernel packets decreased to zero. One of the stats that was identified was app_layer.expectations, which is also increased by the FTP decoder.

<see image graph>

Blue line: Kernel packets
Yellow line: app_layer.expectations

We are not sure if the increase and update on expectations is just a symptom of something else with the FTP decoder, but we have not observed the issues after we disabled the FTP decoder.

On Suricata 5.0.7, we observed an drastic increase in the memory used by Suricata, which with our resources never really stopped (128 GB - 256 GB RAM). This made us assume some sort of memory leak, but we did not observe the same degree of increase for Suricata 6.0.4 when the same issues appeared, and the increase stopped at a "moderate" amount of some tens of gigabytes. We thought it could be related to #5204, and tried to update our config for ippair. It seems this had no impact on the issue, as we still observed this type of behavior.

Observations from a perf report:

We have a perf report from a sensor that has entered the "kernel packets stuck at zero" state. The image below shows what is observed for two workers.

<see perf image>