In the process of asking for legal advice on this.

Another lib: http://research.wand.net.nz/software/libprotoident.php

Any update on the legal issue? the lib from wand looks not really up to date

I'll try to put and add more information about different protocol inspection projects, for now:

• libprotoident
• nDPI

Generalities (contact page, main development site, etc etc)¶

libprotoident¶

Description:¶

(from the project page)

Libprotoident is a library that performs application layer protocol identification for flows. Unlike many techniques that require capturing the entire packet payload, only the first four bytes of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident.

• project page: http://research.wand.net.nz/software/libprotoident.php
• source code: https://github.com/wanduow/libprotoident
• blog and news: https://secure.wand.net.nz/projects/details/libprotoident
• wiki: https://secure.wand.net.nz/trac/libprotoident/

nDPI¶

Description¶

(from the project page):

nDPI is a ntop-maintained superset of the popular OpenDPI library. Released under the LGPL license, its goal is to extend the original library by adding new protocols that are otherwise available only on the paid version of OpenDPI. In addition to Unix platforms, we also support Windows, in order to provide you a cross-platform DPI experience. Furthermore, we have modified nDPI do be more suitable for traffic monitoring applications, by disabling specific features that slow down the DPI engine while being them un-necessary for network traffic monitoring.

• project page: http://www.ntop.org/products/deep-packet-inspection/ndpi/
• source code: https://github.com/ntop/nDPI
• blog and news: http://ntop.org

Specificities¶

libprotoident¶

• Language: C++

• Dependencies: wand libtrace, from http://research.wand.net.nz/software/libtrace.php

• Protocol Identification:
  • Payload Matches (mostly), pattern matching on the first four bytes of the payload on each direction of the traffic
  • Payload Size
  • Port Number, used in case of ambiguity and only for well known ports
  • IP Matching, very few cases
  • protocols are checked in order depending on the confidence the author have on the rules and the popularity of the protocol

• Multi thread support: no, but can be added in the user application

• Flow-aware: no, but can use the wand libflowmanager, from http://research.wand.net.nz/software/libflowmanager.php. Only needed for building the tools

• Test Pcap: none

nDPI¶

• Language: C

• Dependencies: none except for the build environment

• Protocol Identification:
  • Payload Matches
  • Payload Size
  • Port Number
  • IP Matching
  • ability to specify custom ports for protocol in specific environment through a configuration file

• Multi thread support: yes

• Flow-aware: yes (embedded)

• Test Pcap: few in the code base

Community support¶

• libprotoident
  The source code is openly available on GitHub, at the moment there is only one author active.

• nDPI
  The source code is openly available on GitHub.

Licence¶

• libprotoident
  GPLv2.

• nDPI
  LGPLv3. This is an issue if you want to include its source into suricat, according to http://www.gnu.org/licenses/gpl-faq.html#AllCompatibility

Supported Protocols¶

• libprotoident
  A list is here (updated?) : https://secure.wand.net.nz/trac/libprotoident/wiki/SupportedProtocols

• nDPI
  A list is present on the main project page: http://www.ntop.org/products/deep-packet-inspection/ndpi/

Interesting Papers¶

• T. Bujlow et al, Jan. 2014 - Extended Independent Comparison of Popular Deep Packet Inspection (DPI) Tools for Traffic Classification (http://www.ac.upc.edu/app/research-reports/html/RR/2014/1.pdf)
  Very long (more than 400 pages) comparison of the most popular and public available DPI engine. A shorter article version from the same author also exists (Nov. 2014, http://tomasz.bujlow.com/publications/2014_journal_elsevier_comnet_independent_comparison.pdf)

Important statement about the libprotoident and its future from his author can be found here: https://github.com/wanduow/libprotoident/issues/12

Does anyone have some real experience with those projects and could share that knowledge?
Might be worth to take a look at it but seems like a more time consuming task :)

The libprotoident comes out with a few examples, in the tools folder of the code. From the README https://github.com/wanduow/libprotoident

• lpi_protoident

  Description:
  This tool attempts to identify each individual flow within the provided
  trace. Identification only occurs when the flow has concluded or
  expired, so it is not very effective for real-time applications.

• lpi_live (DEPRECATED)

  Description:
  This tool reports byte and packet counts (both inbound and outbound)
  for each identified protocol in real-time. Identification of a flow
  occurs as soon as possible, so that the statistics reported are as
  up-to-date as possible.
  lpi_live has been deprecated and will not be built by default. The code
  is still available in our git repository, but we will not update or
  support this tool anymore. Instead, please use the lpicollector
  (https://github.com/wanduow/lpicollector) for real-time flow analysis
  with libprotoident.
  In combination with the included lpi.py example client, lpicollector
  can produce output similar to that produced by lpi_live.

Libprotoident calls (to external libs) and mappings to suricata's (tentative)¶

from libtrace:¶

• types

libtrace_t | NOT NEEDED
libtrace_tcp_t | combination of PKT_IS_TCP(p) && (p)->tcph
libtrace_udp_t | combination of PKT_IS_UDP(p) && (p)->udph
libtrace_packet_t | analogous to the Packet struct
libtrace_filter_t | use the suricata's buildin BPF code

• calls

trace_get_layer3 | IP_GET_IPPROTO(p)
trace_get_direction | FlowGetPacketDirection(f, p) == TOSERVER ? 1 : 0; NOTE: they are reversed
trace_get_payload | (Packet *)p->payload
trace_get_payload_length | (Packet *)p->payload_len
trace_read_packet | NOT NEEDED, use suricata source modules
trace_get_seconds | XXX
trace_create | NOT NEEDED, use suricata source modules
trace_destroy | NOT NEEDED, use suricata source modules
trace_create_filter | use suricata's BPF filtering
trace_perror | NOT NEEDED
trace_is_err | NOT NEEDED

from libflowmanager¶

NEEDED | lfm_match_packet_to_flow | FlowReference(&p->flow, f);
NEEDED | lfm_update_flow_expiry_timeout | this is done via the flow-timeouts values
NOT NEEDED | lfm_expire_next_flow
NOT NEEDED | lfm_set_config_option

Do we have any progress with this feature?

It seems that the GitHub repository of libprotoident is updated frequently (at least in the last months) and this would make Suricata another step ahead of Snort :)

Since from 6th August 2016 libprotoident is GNUv3 licensed [1], putting it in the same situation as nDPI

There is an open ticket [2] but it got no reply since Mar 9, 2017

[1] https://github.com/wanduow/libprotoident/commit/de8e2ca6d6eb04526912dbc433b0c6003b9f65e1#diff-7116ef0705885343c9e1b2171a06be0e
[2] https://github.com/wanduow/libprotoident/issues/20

@Jason Ish could this be a use case for a plugin ?

Philippe Antoine wrote in #note-15:

@Jason Ish could this be a use case for a plugin ?

Not sure.. This sounds like something that's pretty core? Did you have an idea in mind how it might make sense as a plugin?

just adding a +1 to this feature for nDPI. When Victor was speaking during the "State of Suricata" about plugins, nDPI was the first one that came to my mind. I've talked to a lot of users who desire this type of feature within suricata.