Bug #2954
openStrange interaction with afpacket - high CPU usage and no packet processing
Description
Greetings from the Security Onion community!
As you may know, Security Onion uses Suricata, netsniff-ng, and Bro. We recently started defaulting our Suricata configuration to use afpacket instead of pfring. I just noticed a strange interaction with Suricata and afpacket resulting in high CPU usage and no packet processing.
Here is what I know so far:
- If Suricata is the only sniffing process running (no netsniff-ng), then it processes traffic with normal CPU usage.
- If netsniff-ng is running and Suricata starts in afpacket mode, Suricata spikes to 100% CPU usage and stays there for a long period of time. The log reports that Suricata is initialized, but it is not processing any traffic.
- If I then try to restart Suricata immediately, it again spikes to 100% CPU usage and does not process traffic.
- After waiting a certain amount of time, I can restart Suricata and it will process traffic with normal CPU usage.
- If I start Suricata first without netsniff-ng running, Suricata will process traffic with normal CPU usage. If I then start netsniff-ng, Suricata will immediately spike to 100% CPU usage.
- netsniff-ng consumes traffic via tpacket v3.
- Bro consumes traffic from afpacket and is unaffected by all of this.
- Suricata running in pfring mode is unaffected by all of this.
- Duplicated this issue on multiple different physical boxes with Intel NICs, have not been able to duplicate in a VM yet.
- Duplicated this issue with Suricata configured for 1, 2, and 3 afpacket workers.
- Ubuntu 16.04
- Linux 4.15.0-48-generic
It almost feels as if netsniff is creating some kind of lock or changing afpacket processing in such a way as to cause Suricata to spin. After a certain amount of time, that reverts and Suricata is able to process traffic properly. Bro is unaffected by all of this, so I wonder if Suricata can workaround this condition in the same way.
Thoughts?
Thanks in advance!
Files
Updated by Victor Julien over 5 years ago
Could you attach the af-packet section from your yaml, the suricata start commandline, the netsniff-ng commandline and netsniff and suricata console (stdout/stderr) output?
Updated by Doug Burks over 5 years ago
It looks like adding the --no-hwtimestamp option to netsniff makes this issue go away. We'll update our scripts to use this option, so this is no longer a high priority issue for us.
However, I am curious why netsniff's default use of hardware timestamps caused an issue with Suricata (but not with Bro)?
If you're still interested in the information you requested, I can send that tomorrow.
Thanks!
Updated by Doug Burks over 5 years ago
- File suricata-yaml-afpacket.txt suricata-yaml-afpacket.txt added
- File suricata-output.txt suricata-output.txt added
- File suricata-cmdline.txt suricata-cmdline.txt added
- File netsniff-output.txt netsniff-output.txt added
- File netsniff-cmdline.txt netsniff-cmdline.txt added
I've attached the requested information.
I had been working on this issue for 2 days before creating this ticket. As it turns out, the key point from my description above was "Duplicated this issue on multiple different physical boxes with Intel NICs, have not been able to duplicate in a VM yet." Right after submitting this ticket, I started focusing on the difference between an affected physical box and an unaffected VM. I noticed on an affected physical box that the netsniff log showed "HW timestamping enabled" whereas on an unaffected VM it did not show that. So when I tried adding the --no-hwtimestamp option to netsniff-ng, Suricata was no longer affected.
Updated by Doug Burks over 5 years ago
Updated by Andreas Herz over 5 years ago
- Assignee set to Community Ticket
- Target version set to TBD
Updated by Victor Julien over 5 years ago
- Related to Feature #1954: runtime option/flag to disable hardware timestamp support added