Project

General

Profile

Suricata » Suricata Developers Guide »

Log API

This document describes the Log API's in Suricata 2.0. It was merged through the following pull request: https://github.com/inliniac/suricata/pull/797, which should also be helpful in understanding the changes.

At it's core, each log module is a Thread_Modules.

Common

All the API calls below get a pointer to the ThreadVars structure and more interestingly to a void pointer 'thread_data'. This should be should be casted to the type that is set up by the ThreadInit function of the module.

Registration

All modules are registered in the same way. Per output type there are 2 registration functions. A 'regular' and one for 'sub-modules'. You can register the same module as both, or even more times.

    /* register as separate module */
    OutputRegisterTxModule("JsonHttpLog", "http-json-log", OutputHttpLogInit,
            ALPROTO_HTTP, JsonHttpLogger);

    /* also register as child of eve-log */
    OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "eve-log.http", OutputHttpLogInitSub,
            ALPROTO_HTTP, JsonHttpLogger);

The declarations are:

void OutputRegisterTxModule(const char *name, const char *conf_name,
    OutputCtx *(*InitFunc)(ConfNode *), AppProto alproto,
    TxLogger TxLogFunc);

  • name: the module name, used interally
  • conf_name: the configuration name, this is how the user interacts with it

In the example above, http-json-log is used to enable the log output:

outputs:
  http-json-log:
    enabled: yes
    filename: http.json

  • InitFunc: a function pointer to set up a OutputCtx. This output ctx is shared between all threads.
  • alproto: the app layer protocol this Tx module applies to (specific to tx logger)
  • TxLogFunc: the log function called on transactions (specific to tx logger)

The sub-module registration is almost the same, except for 3 differences:

void OutputRegisterTxSubModule(const char *parent_name, const char *name,
    const char *conf_name, OutputCtx *(*InitFunc)(ConfNode *, OutputCtx *parent_ctx),
    AppProto alproto, TxLogger TxLogFunc);

It adds the name of a parent and the conf_name and InitFunc are different:

  • parent_name: name of the parent, this is the conf_name of the parent
  • conf_name: this is: <parent conf name>.<module conf name>
  • InitFunc: a function pointer to set up a OutputCtx. This output ctx is shared between all threads. This is different from the 'regular' module in that it takes 2 arguments. It gets the OutputCtx pointer from the parent as an argument, so that it's possible to share this among all children.

Regular vs Sub Modules

Log modules can be registered in 2 separate ways. As a normal module and as a sub-module. The sub-module is enabled by it's parent and shares the output with it's parent. So sub modules can be used to have multiple log modules all output to the same file.

An example of this is the 'eve-log' module:

  # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: no   # force logging magic on all logged files
            force-md5: no     # force logging of md5 checksums
        - drop

Here, the types are log modules registered as sub-modules of 'eve-log'. For example, the http log is registered as:
    OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "eve-log.http", OutputHttpLogInitSub,
            ALPROTO_HTTP, JsonHttpLogger);

Packet Logger

The packet logger API has 2 callbacks. A log condition functiona and a logger. The condition is called for every packet. If it returns TRUE, the logger is also called for a packet. Otherwise, the logger is not invoked.

The logger looks like this:

/** packet logger function pointer type */
typedef int (*PacketLogger)(ThreadVars *, void *thread_data, const Packet *);

In addition to the thread data, the logger gets a const pointer to the packet structure. It is not supposed to modify the packet, hence the const.

The condition also gets a const pointer to the packet:

/** packet logger condition function pointer type,
 *  must return true for packets that should be logged
 */
typedef int (*PacketLogCondition)(ThreadVars *, const Packet *);

An example of a condition function is:

int AlertFastLogCondition(ThreadVars *tv, const Packet *p)
{
    return (p->alerts.cnt ? TRUE : FALSE);
}

This function returns TRUE if there are alerts stored in the packet.

Tx Logger

The transaction logger is an event based logger. It runs on top of the AppLayer API and is only applied to protocols that are 'Tx Aware', like HTTP and DNS. Each transaction is tracked by Suricata, and the parser has defined conditions indicating when the transaction is considered 'complete'. When a Tx is complete, the Tx Logger API calls are invoked.

There is a single logger function:

typedef int (*TxLogger)(ThreadVars *, void *thread_data, const Packet *, Flow *f, void *state, void *tx, uint64_t tx_id);

In addition to the thread vars, this function has access to the packet triggering this logger (usually a TCP ACK packet triggering reassembly), the LOCKED flow, the app layer state, the transaction and the transaction id.

This function is also called if according to the protocol the transaction is not yet complete (e.g. a HTTP request w/o response), but the flow is closed anyway. This can be for various reasons, like TCP Resets, Flow timeout, etc.

As the flow is locked while this function runs, accessing the flow, state and tx is thread safe.

Registration

There are 2 registration functions.

    /* register as separate module */
    OutputRegisterTxModule("JsonHttpLog", "http-json-log", OutputHttpLogInit,
            ALPROTO_HTTP, JsonHttpLogger);

The Tx Logger needs an alproto and a logger pointer.

The sub-module registration is similar, but takes a parent name.

    /* also register as child of eve-log */
    OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "eve-log.http", OutputHttpLogInitSub,
            ALPROTO_HTTP, JsonHttpLogger);

File Logger

The file logger will be able to log files tracked in HTTP. As more than one file can be transferred per transaction, a separate API call was introduced.

typedef int (*FileLogger)(ThreadVars *, void *thread_data, const Packet *, const File *);

This logger is called once per file, when it's 'closed'. This means that Suricata either saw the entire file, or the file was truncated.

Registration

There are 2 registration functions:

    OutputRegisterFileModule(MODULE_NAME, "file-log", LogFileLogInitCtx,
            LogFileLogger);

The LogFileLogger here is the callback of the type FileLogger as outlined above.

For sub-modules:

    /* register as child of eve-log */
    OutputRegisterFileSubModule("eve-log", "JsonFileLog", "eve-log.files",
            OutputFileLogInitSub, JsonFileLogger);

Filedata Logger

The file data logger logs actual data chunks. It's meant to store the contents of a file somehow.

typedef int (*FiledataLogger)(ThreadVars *, void *thread_data, const Packet *, const File *, const FileData *, uint8_t flags);

The flags field is used to indicate to the callback if the file is opened or closed. The OUTPUT_FILEDATA_FLAG_OPEN flag is only set in the first call for a specific file. OUTPUT_FILEDATA_FLAG_CLOSE is only set in the last call for a specific file. With the latter flag, it is possible that the FileData pointer is NULL.

TODO explain more