Packet Acquisition API¶
Introduction¶
To add a new capture mode, you need to add two things to suricata:- Code to realize the capture
- Dedicated running modes
We will use AF_PACKET as example for the rest of the document.
- src/source-af-packet.c
- src/source-af-packet.h
- src/runmode-af-packet.c
- src/runmode-af-packet.h
Adding the code to build¶
To add the files to the build system, src/Makefile.am need to be edited. The alphabetically sorted variable suricata_SOURCES must be updated
to contain the neme of the new files:
suricata_SOURCES = ... runmode-af-packet.c runmode-af-packet.h \ ... source-af-packet.c source-af-packet.h \
It must be possible to disable or enable the build from configure. To do so, configure.ac must be updated and should at least have a new AC_ARG_ENABLE line:
# AF_PACKET support
AC_ARG_ENABLE(af-packet,
AS_HELP_STRING([--enable-af-packet], [Enable AF_PACKET support [default=yes]]),
,[enable_af_packet=yes])
This defined a
HAVE_AF_PACKET define in config.h so the dedicated capture code must be surrounded by the following construct:#ifdef HAVE_AF_PACKET ... #endif
If some checks need to be done (library check, header check, version check), they can be added via an AS_IF construct:
AS_IF([test "x$enable_af_packet" = "xyes"], [
AC_CHECK_DECL([TPACKET_V2],
AC_DEFINE([HAVE_AF_PACKET],[1],[AF_PACKET support is available]),
[enable_af_packet="no"],
[[#include <sys/socket.h>
#include <linux/if_packet.h>]])
AC_CHECK_DECL([PACKET_FANOUT],
AC_DEFINE([HAVE_PACKET_FANOUT],[1],[Packet fanout support is available]),
[],
[[#include <linux/if_packet.h>]])
])
A last thing to add in
configure.ac is the indication that the capture module is build or not:
echo "
Suricata Configuration:
AF_PACKET support: ${enable_af_packet}
This message is displayed at the end of configure run.
Declare the capture mode¶
We need to declare to suricata that it has a new capture mode. This is done by filling in the tmm_modules array which contains all arrays. First thing to do is to declare the new elements in the list of modules. This is done in source:src/tm-threads-common.h by updating the TmmId enum. Usually two items are defined:- A receive item: TMM_RECEIVEAFP
- A decode item: TMM_DECODEAFP
Once done, the setup is made by defining two functions in the source:src/source-af-packet.c
void TmModuleReceiveAFPRegister (void) {
tmm_modules[TMM_RECEIVEAFP].name = "ReceiveAFP";
tmm_modules[TMM_RECEIVEAFP].ThreadInit = ReceiveAFPThreadInit;
tmm_modules[TMM_RECEIVEAFP].Func = NULL;
tmm_modules[TMM_RECEIVEAFP].PktAcqLoop = ReceiveAFPLoop;
tmm_modules[TMM_RECEIVEAFP].ThreadExitPrintStats = ReceiveAFPThreadExitStats;
tmm_modules[TMM_RECEIVEAFP].ThreadDeinit = NULL;
tmm_modules[TMM_RECEIVEAFP].RegisterTests = NULL;
tmm_modules[TMM_RECEIVEAFP].cap_flags = SC_CAP_NET_RAW;
tmm_modules[TMM_RECEIVEAFP].flags = TM_FLAG_RECEIVE_TM;
}
void TmModuleDecodeAFPRegister (void) {
tmm_modules[TMM_DECODEAFP].name = "DecodeAFP";
tmm_modules[TMM_DECODEAFP].ThreadInit = DecodeAFPThreadInit;
tmm_modules[TMM_DECODEAFP].Func = DecodeAFP;
tmm_modules[TMM_DECODEAFP].ThreadExitPrintStats = NULL;
tmm_modules[TMM_DECODEAFP].ThreadDeinit = NULL;
tmm_modules[TMM_DECODEAFP].RegisterTests = NULL;
tmm_modules[TMM_DECODEAFP].cap_flags = 0;
tmm_modules[TMM_DECODEAFP].flags = TM_FLAG_DECODE_TM;
}
These functions are exported in the corresponding header and are called inside source:src/suricata.c
/* af-packet */
TmModuleReceiveAFPRegister();
TmModuleDecodeAFPRegister();
It is common that the support for the capture can be disable. In this case, alternate registration function are written to emit a warning when an non-built-in support is called.
Anatomy of the code¶
The architecture of the code is defined in the ReceiveAFP registration.
void TmModuleReceiveAFPRegister (void) {
tmm_modules[TMM_RECEIVEAFP].name = "ReceiveAFP";
tmm_modules[TMM_RECEIVEAFP].ThreadInit = ReceiveAFPThreadInit;
tmm_modules[TMM_RECEIVEAFP].Func = NULL;
tmm_modules[TMM_RECEIVEAFP].PktAcqLoop = ReceiveAFPLoop;
tmm_modules[TMM_RECEIVEAFP].ThreadExitPrintStats = ReceiveAFPThreadExitStats;
tmm_modules[TMM_RECEIVEAFP].ThreadDeinit = NULL;
tmm_modules[TMM_RECEIVEAFP].RegisterTests = NULL;
tmm_modules[TMM_RECEIVEAFP].cap_flags = SC_CAP_NET_RAW;
tmm_modules[TMM_RECEIVEAFP].flags = TM_FLAG_RECEIVE_TM;
}
A set of functions needs to be defined:
- ThreadInit: the code called once when the capture thread start
- ThreadExitPrintStats: function which is in charge of printing statistics of the run to stdout
- ThreadDeinit: the code called once when the capture thread end
- PktAcqLoop: this is the 'I do the work' function. It reads data from the capture system, create packet and send them to the processing system
- RegisterTests: to define unittest
- cap_flags: define the capabilities needed for the capture thread
- flags: always set to TM_FLAG_RECEIVE_TM for a receive module
Init and Deinit phase¶
The init function has the following type:
TmEcode ReceiveAFPThreadInit(ThreadVars *tv, void *initdata, void **data)
tv is the ThreadVars attached to the thread the capture will run into. initdata is a configuration structure which is declared and filled in in the running mode code. So the main part of the code is to get element from the configuration
and to take the corresponding action. It will often be a good idea to store most of the setting into the output of the init
function wich is data.
AFPIfaceConfig *afpconfig = initdata;
...
ptv->buffer_size = afpconfig->buffer_size;
ptv->ring_size = afpconfig->ring_size;
...
*data = (void *)ptv;
SCReturnInt(TM_ECODE_OK);
ptv is a pointer to structure which is allocated by the Init function and that will be received as argument in all subsequent calls of the module. In AF_PACKET code this is a AFPThreadVars structure and this lead to the following code:
TmEcode ReceiveAFPLoop(ThreadVars *tv, void *data, void *slot)
{
SCEnter();
uint16_t packet_q_len = 0;
AFPThreadVars *ptv = (AFPThreadVars *)data;
This structure must contain everything needed to be able to get packet as well as all parameters needed to reopen a capture if there is a capture failure. For AF_PACKET we have among other fields:
typedef struct AFPThreadVars_
{
/* thread specific socket */
int socket;
/* data link type for the thread */
int datalink;
int cooked;
/* If possible livedevice the capture thread is rattached to. */
LiveDevice *livedev;
/* counters */
uint32_t pkts;
uint64_t bytes;
uint32_t errs;
/* Needed for loop processing */
ThreadVars *tv;
TmSlot *slot;
/* specific AF_PACKET params needed for reconnection */
int cluster_id;
int cluster_type;
} AFPThreadVars;
The capture loop¶
The capture loop, ReceiveAFPLoop in AF_PACKET case, is the main function. It has two phase:- Capture start: open the capture socket. This is good to done that here as it will limit the delay between opening of the capture and the start of the processing
- Packet processing loop: a loop where packet are fetch and send to the processing
while (1) {
/* Start by checking the state of our interface */
if (unlikely(ptv->afp_state == AFP_STATE_DOWN)) {
int dbreak = 0;
/* Try to restart the capture */
...
/* reconnect unsuccesful if local var dbreak is 1 */
if (dbreak == 1)
break;
}
/* make sure we have at least one packet in the packet pool, to prevent
* us from alloc'ing packets at line rate */
do {
packet_q_len = PacketPoolSize();
if (unlikely(packet_q_len == 0)) {
PacketPoolWait();
}
} while (packet_q_len == 0);
/* Wait for data from capture */
r = poll(&fds, 1, POLL_TIMEOUT);
/* if suricata_ctl_flags is not zero we have to leave */
if (suricata_ctl_flags != 0) {
break;
}
/* Read the data, a loop on packet to read is possible here depending on
* cature mode */
/* First we need to get a packet */
Packet *p = PacketGetFromQueueOrAlloc();
if (p == NULL) {
SCReturnInt(AFP_FAILURE);
}
PKT_SET_SRC(p, PKT_SRC_WIRE);
/* Copy the data to the packet, here h.raw for h.len. It is also possible
to use PacketSetData to do zero copy. In this case, see Packet Release
Mechanism below. */
if (PacketCopyData(p, (unsigned char*)h.raw, h.len) == -1) {
TmqhOutputPacketpool(ptv->tv, p);
SCReturnInt(AFP_FAILURE);
}
/* Do the packet processing by calling TmThreadsSlotProcessPkt, this will, depending on the running mode,
pass the packet to the treatment functions or push it to a packet pool. So processing time can vary.
*/
if (TmThreadsSlotProcessPkt(ptv->tv, ptv->slot, p) != TM_ECODE_OK) {
TmqhOutputPacketpool(ptv->tv, p);
SCReturnInt(AFP_FAILURE);
}
/* Realize counter sync if needed */
SCPerfSyncCountersIfSignalled(tv, 0);
}
Packet release mechanism¶
In workers mode, all the processing is done in the loop function by a single call to TmThreadsSlotProcessPkt() function.
Thus the code can be something like the following:
/* packet data is pointed by h.raw and length of data is h.len */
/* Set data on packet by making Packet field point to capture provided data */
if (PacketSetData(p, (unsigned char*)h.raw, h.len) == -1) {
TmqhOutputPacketpool(ptv->tv, p);
SCReturnInt(AFP_FAILURE);
}
/* Process the packet */
TmThreadsSlotProcessPkt(ptv->tv, ptv->slot, p);
/* You free function here */
FREE_CAPTURE_DATA(h.ra, h.len);
But this is not the case of all running mode, as TmThreadsSlotProcessPkt will return as soon as a packet pool is used
in the processing. To fix this problem, it is possible to use the Packet release mechanism.
Basically, the code will setup the packet to have a function called when Suricata will get rid of packet data. This is done
by telling Suricata to use a callback function pointed by p->ReleaseData. if the function is not NULL, it will be called during the freeing all resssources linked with the packet.
p->ReleaseData = AFPReleaseDataFromRing;
The callback function has the following declaration:
TmEcode AFPReleaseDataFromRing(ThreadVars *t, Packet *p)
The ThreadVars pointer is not the one of the capture thread, so you may need to setup the necessary things (like pointer to the capture data to be freed) inside the Packet structure. This can be done by adding a new element in the union inside Packet declaration:
union {
/* nfq stuff */
#ifdef NFQ
NFQPacketVars nfq_v;
#endif /* NFQ */
#ifdef IPFW
IPFWPacketVars ipfw_v;
#endif /* IPFW */
#ifdef AF_PACKET
AFPPacketVars afp_v;
#endif
/** libpcap vars: shared by Pcap Live mode and Pcap File mode */
PcapPacketVars pcap_v;
};
Decoding packet¶
This part of the code has mainly for role to be a wrapper to existing decoding function:
TmEcode DecodeAFP(ThreadVars *tv, Packet *p, void *data, PacketQueue *pq, PacketQueue *postpq)
{
SCEnter();
DecodeThreadVars *dtv = (DecodeThreadVars *)data;
/* update counters, this counter are generic for capture threads */
SCPerfCounterIncr(dtv->counter_pkts, tv->sc_perf_pca);
SCPerfCounterIncr(dtv->counter_pkts_per_sec, tv->sc_perf_pca);
SCPerfCounterAddUI64(dtv->counter_bytes, tv->sc_perf_pca, GET_PKT_LEN(p));
SCPerfCounterAddUI64(dtv->counter_avg_pkt_size, tv->sc_perf_pca, GET_PKT_LEN(p));
SCPerfCounterSetUI64(dtv->counter_max_pkt_size, tv->sc_perf_pca, GET_PKT_LEN(p));
/* call the decoder, the real work begin */
switch(p->datalink) {
case LINKTYPE_LINUX_SLL:
DecodeSll(tv, dtv, p, GET_PKT_DATA(p), GET_PKT_LEN(p), pq);
break;
case LINKTYPE_ETHERNET:
DecodeEthernet(tv, dtv, p,GET_PKT_DATA(p), GET_PKT_LEN(p), pq);
break;
case LINKTYPE_PPP:
DecodePPP(tv, dtv, p, GET_PKT_DATA(p), GET_PKT_LEN(p), pq);
break;
case LINKTYPE_RAW:
DecodeRaw(tv, dtv, p, GET_PKT_DATA(p), GET_PKT_LEN(p), pq);
break;
default:
SCLogError(SC_ERR_DATALINK_UNIMPLEMENTED, "Error: datalink type %" PRId32 " not yet supported in module DecodeAFP", p->datalink);
break;
}
SCReturnInt(TM_ECODE_OK);
}
Counters¶
Performance counters must be implemented to ease information fetching and performance tracking. There is two types of counters:- Suricata Performance counters: basicall per-threads value
- Capture performance counters: atomic integer shared between the capture threads
A capture thread should at least have a kernel accept and drop counters (suricata perf counter), here's how to init them:
#ifdef PACKET_STATISTICS
ptv->capture_kernel_packets = SCPerfTVRegisterCounter("capture.kernel_packets",
ptv->tv,
SC_PERF_TYPE_UINT64,
"NULL");
ptv->capture_kernel_drops = SCPerfTVRegisterCounter("capture.kernel_drops",
ptv->tv,
SC_PERF_TYPE_UINT64,
"NULL");
#endif
And it must increments the Live device counter:
(void) SC_ATOMIC_ADD(ptv->livedev->drop, kstats.tp_drops);
It is recommanded to do the increment operation in a dedicated function which is called when packet are treated and which is also called in the ExitStats function. For example:
static inline void AFPDumpCounters(AFPThreadVars *ptv)
{
#ifdef PACKET_STATISTICS
struct tpacket_stats kstats;
socklen_t len = sizeof (struct tpacket_stats);
/* AF_PACKET feching stats */
if (getsockopt(ptv->socket, SOL_PACKET, PACKET_STATISTICS,
&kstats, &len) > -1) {
/* Counter increment */
SCPerfCounterAddUI64(ptv->capture_kernel_packets, ptv->tv->sc_perf_pca, kstats.tp_packets);
SCPerfCounterAddUI64(ptv->capture_kernel_drops, ptv->tv->sc_perf_pca, kstats.tp_drops);
(void) SC_ATOMIC_ADD(ptv->livedev->drop, kstats.tp_drops);
}
#endif
}
Running modes¶
Once the capture is ready, we need to connect it to Suricata. This is done by writing a runmode file with uses all the features proposed by source:src/util-runmodes.c
The active part of the code is the registration function which will add the dedicated running mode to Suricata. This is a simple function which defined the default mode and add the running mode to the running mode list by calling RunModeRegisterNewRunMode
void RunModeIdsAFPRegister(void)
{
default_mode_auto = "autofp";
RunModeRegisterNewRunMode(RUNMODE_AFP_DEV, "workers",
"Multi threaded af-packet mode",
RunModeIdsAFPWorkers);
...
}
A call to RunModeIdsAFPRegister must be added to RunModeRegisterRunModes.
The running mode is then defined by calling a RunModeSetLiveCapture funtion. For example, for the workers running mode, the capture module need to call RunModeSetLiveCaptureWorkers. For example, a slightly simplified version of RunModeIdsAFPWorkers is the following:
int RunModeIdsAFPWorkers(DetectEngineCtx *de_ctx)
{
SCEnter();
#ifdef HAVE_AF_PACKET
int ret;
char *live_dev = NULL;
/* Standard initializatoin function */
RunModeInitialize();
/* Call this if capture is live */
TimeModeSetLive();
/* Try to get the interface given on command line, no value is handled by NULL */
(void)ConfGet("af-packet.live-interface", &live_dev);
/* Setup and start the running mode */
ret = RunModeSetLiveCaptureWorkers(de_ctx,
ParseAFPConfig,
AFPConfigGeThreadsCount,
"ReceiveAFP",
"DecodeAFP", "AFPacket",
live_dev);
if (ret != 0) {
SCLogError(SC_ERR_RUNMODE, "Unable to start runmode");
exit(EXIT_FAILURE);
}
SCLogInfo("RunModeIdsAFPWorkers initialised");
#endif
SCReturnInt(0);
}
So it is easy to implement a running mode as soon as, the callback functions (here ParseAFPConfig and AFPConfigGeThreadsCount) are coded.
First function is in charge of creating and filling a configuration for a given interface. Its declaration is the following:
void *ParseAFPConfig(const char *iface)
Given a interface name, it returns a complete configuration structure. This structure will be used as reference in the ThreadInit function of the capture module. So it must contain all information fetch from the YAML and all that's needed for a correct start of the capture thread.
The second callback function is the easiest one AFPConfigGeThreadsCount which get the number of capture threads to start for a configuration.
The configuration MUST be per-interface as this enable the user to have a setup with multiple interfaces:
af-packet:
- interface: eth0
threads: 1
cluster-id: 99
- interface: eth1
threads: 2
cluster-id: 98
cluster-type: cluster_flow
defrag: yes
In the configuration parsing function, you can access to the configuration in one interface with the following code:
/* Find initial node */
af_packet_node = ConfGetNode("af-packet");
if (af_packet_node == NULL) {
SCLogInfo("Unable to find af-packet config using default value");
return aconf;
}
if_root = ConfNodeLookupKeyValue(af_packet_node, "interface", iface);
if (if_root == NULL) {
SCLogInfo("Unable to find af-packet config for "
"interface %s, using default value",
iface);
return aconf;
}
Once done, you can access to configuration element by using a code similar to the following:
if (ConfGetChildValue(if_root, "copy-mode", ©modestr) == 1) {
One thing that SHOULD be read is the configuration parsing callback is the BPF filter.
if (ConfGet("bpf-filter", &bpf_filter) == 1) {
if (strlen(bpf_filter) > 0) {
aconf->bpf_filter = bpf_filter;
SCLogInfo("Going to use command-line provided bpf filter '%s'",
aconf->bpf_filter);
}
}
A per-interface value for the BPF SHOULD also be provided and could be get as follow:
if (ConfGetChildValue(if_root, "bpf-filter", &bpf_filter) == 1) {
One of the difficulties is that the configuration can be used by multiple threads at init. So if you want to to free it once everything is started, you need to implement a reference counter. Look at AFPDerefConfig and DerefFunc in the code for an example.
Adding support for cuda inside the Packet Acquisition API.¶
Enabling one to send packets to the gpu requires some minor code additions inside the packet acquisition source file.
1. Including the following header files.
#ifdef __SC_CUDA_SUPPORT__ #include "util-cuda.h" #include "util-cuda-buffer.h" #include "util-mpm-ac.h" #include "util-cuda-handlers.h" #include "detect-engine.h" #include "detect-engine-mpm.h" #include "util-cuda-vars.h" #endif /* __SC_CUDA_SUPPORT__ */
2. Init the cuda stage so that we can buffer the packets. Insert the following code inside the Decoder's Thread Init function -
#ifdef __SC_CUDA_SUPPORT__
if (CudaThreadVarsInit(&dtv->cuda_vars) < 0)
SCReturnInt(TM_ECODE_FAILED);
#endif
3. Finally the code that sends the packet itself. This needs to be done inside the decoder function itself, after the decoding on the packet is done -
#ifdef __SC_CUDA_SUPPORT__
if (dtv->cuda_vars.mpm_is_cuda)
CudaBufferPacket(&dtv->cuda_vars, p);
#endif
You can use source-pcap.c as a reference for the above changes.