Suricata

Definitely worth looking at. It's probably failing now because of the defrag engine needing that buffer for copying the reassembled packet into.

Defrag should only need to be modified with respect to storing the reassembled, so a pretty minor fix there.

But will this work? This buffer that is passed to us is managed by libpcap, and will be reclaimed by libpcap at its will. Is there a way to tell libpcap not to reclaim a buffer until we're ready to let it go?

I did a simple test app that made a copy of the packet and stored a pointer to it. The both contained the same data only for about 32 packets (just a very default settings pcap app). Increasing the pcap buffer size may push this out a bit, but its still not under our control.

If you look at pcap_next_ex, which is kind of a wrapper around pcap_dispatch so you can avoid using the callback yourself, it places the packet in a buffer you provided. So you still have the memcpy, but this time inside libpcap rather than in your app. This is required because once the callback has returned you can't rely on the contents of the packet buffer passed to you inside the callback.

I have to admit I haven't kept fully up to date on the latest libpcap developments so there may be away around this.

This is the same problem we encounter with DAG integration -- we have to memcpy the packets from the DAG buffers as the DAG card writes to the same ring buffer.

We do have control though on when the DAG card can write over packets but this information would have to be propagated through the engine in some fashion, you would still require copying pkts from the buffer for IP/TCP reassembly, but you may avoid an initial copy.

In the past with more specialized applications I've been able to keep packets in the buffer and then copy them out when we've needed to, e..g IMA (Inverse Multiplexing over ATM) lends itself nicely to this -- this is a relatively simple application though compared with Suricata.

Could be interesting work to see if this is applicable -- Will what sort of HW did you do your testing on?

"If you look at pcap_next_ex, which is kind of a wrapper around pcap_dispatch so you can avoid using the callback yourself, it places the packet in a buffer you provided. So you still have the memcpy, but this time inside libpcap rather than in your app."

Does this buy us anything performance wise?

PF_RING now supports this and I think work is being done to support this for netfilter_queue as well.

After a first read, it seems we can use the zero copy under the same assumption as with af_packet: all treatment must be made within the callback thread. This means only 'single' and 'workers' mode can benefit from this. I preparing a patch to implement this.

Maybe you can address #415 as well, so that we don't have to limit ourselves to single and workers? Or does pcap not have a "release packet" function for 0copy?

pcap does not have such a function but I start working on #415 for other capture mode.

This can be closed as #415 implement a release data mechanism.