Suricata

I was attempting to batch-process a list of PCAPs, (see also issue #2220), I noticed that while submitting PCAPs to Suricata, processing seemed very fast, but after this was done, processing was limited to 5 PCAPs/second.

I started Suricata in socket mode:
suricata --unix-socket=/tmp/suricata/socket --pidfile=/tmp/suricata/pid -S /tmp/suricata/rules

With the rules file being empty, or containing a single simple rule. Then, I submit a lot of PCAPs to process using a very simple Python script, and interrupting it using Ctrl+C after a while:

import suricatasc
s = suricatasc.SuricataSC("/tmp/suricata/socket")
s.connect()
while True:
    s.send_command("pcap-file", {"filename": "empty.pcap", "output-dir": "/tmp/suricata/"})

Processing of Suricata slowed down to at most 5 PCAPs/second after Ctrl+C'ing the above script. This seems to be due to the fact that after each PCAP processing, Suricata reads messages for 0.2s from the Socket interface. After not having received any messages, it continues processing. This seems to be caused by the function UnixMain of unix-manager.c, where the 200ms timeout is specified.

I feel the polling of the socket should probably be done in a separate thread. This would accomplish two things:

• PCAP batch processing is not slowing down when there are no messages served on the socket
• Clients sending messages on the socket do not have to wait for Suricata when it is processing a large PCAP

Note that the issue is resolved by, after submitting all PCAPs, continuing to send messages on the socket.

This issue occurs with at least Suricata 3.2 and 4.0.