Bug #2220
openWhen running on a single-CPU machine, pcap processing takes a long time
Description
I was attempting to batch-process a list of PCAPs, but I noticed the speed was very low on one of my worker machines. It is a virtual machine with only a single CPU. Increasing the number of CPUs resolves the issue.
I started Suricata in socket mode:suricata --unix-socket=/tmp/suricata/socket --pidfile=/tmp/suricata/pid -S /tmp/suricata/rules
With the rules file being empty, or containing a single simple rule.
Then, I submit a lot of PCAPs to process using a very simple Python script, and interrupting it using Ctrl+C after a while:
import suricatasc
s = suricatasc.SuricataSC("/tmp/suricata/socket")
s.connect()
while True:
s.send_command("pcap-file", {"filename": "empty.pcap", "output-dir": "/tmp/suricata/"})
On a machine with 1 CPU and 1 core, it takes almost exactly 1 second to process a single PCAP file, even if it is empty. The script above will be successful in sending two commands to the socket, so after 30s of running this script, 60 pcaps will be submitted (and 30 will still need processing).
On a machine with more CPU threads, the above works very fast.
This seems to be an issue with the threading library or so. Since my testing showed the processing taking 1 second per PCAP, I suspected a random sleep somewhere, but couldn't find it.
This issue occurs with at least Suricata 3.0, 3.2 and 4.0.