Suricata

Been seeing this consistently on some of our higher traffic sensors since our major upgrade from Suricata 3.0.2 to 4.1.2 release and I also confirmed the issue persists on today's 4.1.4 release. Output below is from our compiled debug version. Happy to provide the crash file / core dump to a developer if needed.

Platform: Ubuntu 16.04 LTS
Uname: Linux sensor 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Configure:

./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --enable-geoip --enable-luajit --enable-pie --enable-hiredis --enable-gccprotect --disable-gccmarch-native --enable-debug CFLAGS="-ggdb -O0" && make && make install

Build Info and Crash Logs

user@sensor:~$ suricata --build-info

This is Suricata version 4.1.4 RELEASE
Features: DEBUG PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 5.4.0 20160609, C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  liblzma support:                         yes
  hiredis support:                         yes
  hiredis async with libevent:             yes
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          yes

  Rust support:                            yes (default)
  Rust strict mode:                        no
  Rust debug mode:                         no
  Rust compiler:                           rustc 1.31.0
  Rust cargo:                              cargo 1.31.0

  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    yes
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: yes
  CFLAGS                                   -ggdb -O0 -I${srcdir}/../rust/gen/c-headers
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

user@sensor:~$ sudo journalctl -u suricata -ef | grep -A20 -B20 -i segv
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Entering ... >>
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Returning ... <<
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "tag" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "threshold" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "host" name "bit" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "flow" name "expectation" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "flow" name "tag" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "expectation" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "threshold" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - type "ippair" name "bit" size "8" 
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Parent is waiting for child to be ready
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Daemon: Parent waiting for child to be ready...
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - failed to lookup configuration parameter 'daemon-directory'
Apr 30 19:01:37 sensor suricata[27656]: 30/4/2019 -- 19:01:37 - <Debug> - Child is ready, parent exiting
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Main process exited, code=killed, status=11/SEGV
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Control process exited, code=exited status=1
Apr 30 19:02:53 sensor kill[27690]: Usage:
Apr 30 19:02:53 sensor kill[27690]:  kill [options] <pid> [...]
Apr 30 19:02:53 sensor kill[27690]: Options:
Apr 30 19:02:53 sensor kill[27690]:  <pid> [...]            send signal to every <pid> listed
Apr 30 19:02:53 sensor kill[27690]:  -<signal>, -s, --signal <signal>
Apr 30 19:02:53 sensor kill[27690]:                         specify the <signal> to be sent
Apr 30 19:02:53 sensor kill[27690]:  -l, --list=[<signal>]  list all signal names, or convert one to a name
Apr 30 19:02:53 sensor kill[27690]:  -L, --table            list all signal names in a nice table
Apr 30 19:02:53 sensor kill[27690]:  -h, --help     display this help and exit
Apr 30 19:02:53 sensor kill[27690]:  -V, --version  output version information and exit
Apr 30 19:02:53 sensor kill[27690]: For more details see kill(1).
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Unit entered failed state.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Triggering OnFailure= dependencies.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Failed with result 'exit-code'.
Apr 30 19:02:53 sensor systemd[1]: suricata.service: Service hold-off time over, scheduling restart.
Apr 30 19:02:53 sensor systemd[1]: Stopped Suricata Daemon.
Apr 30 19:02:53 sensor systemd[1]: Starting Suricata Daemon...
Apr 30 19:02:53 sensor genconf_suricata[27699]: Generating Variable File
Apr 30 19:02:53 sensor genconf_suricata[27699]:         Writing to /etc/suricata/suricata_vars.yaml

user@sensor:~$ dmesg | grep seg
[70850.864982] W#01-br0[27147]: segfault at 40 ip 000055a99aa623f7 sp 00007fd16e63d5d8 error 4 in suricata-4.1.4-prod[55a99a9c6000+4b7000]
[70989.001085] W#02-br0[27582]: segfault at 40 ip 000055d9aca963f7 sp 00007f71e67545d8 error 4 in suricata-4.1.4-prod[55d9ac9fa000+4b7000]
[71083.258640] W#02-br0[27672]: segfault at 40 ip 000055ce734ba68c sp 00007f3d26afe560 error 4 in suricata-4.1.4-dbg[55ce73368000+bae000]
[71178.483007] W#01-br0[27715]: segfault at 40 ip 000056092ef8168c sp 00007fb9e4ee4560 error 4 in suricata-4.1.4-dbg[56092ee2f000+bae000]
[71224.431899] W#01-br0[27997]: segfault at 40 ip 000055ab74e8968c sp 00007f3b3854d560 error 4 in suricata-4.1.4-dbg[55ab74d37000+bae000]
[71427.648879] W#01-br0[28119]: segfault at 40 ip 000055c70935c68c sp 00007fea065ee560 error 4 in suricata-4.1.4-dbg[55c70920a000+bae000]

user@sensor:~$ tail /var/log/apport.log -n1
ERROR: apport (pid 28453) Tue Apr 30 19:09:42 2019: apport: report /var/crash/_usr_bin_suricata-4.1.4-dbg.0.crash already exists and unseen, doing nothing to avoid disk usage DoS