Actions
Bug #2718
openpkts/drops counters discrepancy
Affected Versions:
Effort:
Difficulty:
Label:
Description
Running git with AFPv3 (info below at the end) and seeing discrepancy in total and dropped packet counters on the same Suricata run between suricata.log/stdout and stats.log/eve stats.json
Seeing this on 2 different high speed Suricata deployments.
------------------------------------------------------------------------------------ Date: 11/29/2018 -- 22:53:39 (uptime: 0d, 20h 58m 32s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 34526257089 capture.kernel_drops | Total | 364864207 decoder.pkts | Total | 34161381126 decoder.bytes | Total | 27521454249680 decoder.invalid | Total | 125528 decoder.ipv4 | Total | 33874924190 decoder.ipv6 | Total | 299143889 decoder.ethernet | Total | 34161381126 [1841] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#15-eth3) Kernel: Packets 1748200153, dropped 23108264 [1841] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#15-eth3) Files extracted 0 [1842] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#16-eth3) Kernel: Packets 13, dropped 0 [1842] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#16-eth3) Files extracted 0 [1843] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#01-eth0) Kernel: Packets 1235440, dropped 119484 [1843] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#01-eth0) Files extracted 0 [1701] 29/11/2018 -- 22:53:39 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 3052 [1701] 29/11/2018 -- 22:53:40 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [1701] 29/11/2018 -- 22:53:42 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1622144 bytes, maximum: 16777216 [1701] 29/11/2018 -- 22:53:42 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth2': pkts: 8251080725, drop: 42475597 (0.51%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth3': pkts: 26275081393, drop: 322269126 (1.23%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth0': pkts: 1235440, drop: 119484 (9.67%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache root@suricata:~#
Above -
Total counter(s, combined) has a mismatch
Drop counter(s , combined) is good - exactly the same
[74190] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#33-enp59s0) Kernel: Packets 1063135836, dropped 1969930 [74191] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#34-enp59s0) Kernel: Packets 1069756460, dropped 4021948 [74192] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#35-enp59s0) Kernel: Packets 1128621774, dropped 1065953 [74193] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#36-enp59s0) Kernel: Packets 1143644762, dropped 1572917 [74194] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#37-enp59s0) Kernel: Packets 1029388098, dropped 110874 [74195] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 1049514375, dropped 237793 [74196] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 1109232251, dropped 2193864 [74197] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 1125246426, dropped 1889525 [73968] 29/11/2018 -- 23:06:28 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 16159124 [73968] 29/11/2018 -- 23:06:37 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [73968] 29/11/2018 -- 23:06:45 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 33554400 bytes, maximum: 33554432 [73968] 29/11/2018 -- 23:06:48 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [73968] 29/11/2018 -- 23:06:48 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0': pkts: 44861666289, drop: 106594312 (0.24%), invalid chksum: 0 [73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache ------------------------------------------------------------------------------------ Date: 11/29/2018 -- 23:06:25 (uptime: 0d, 06h 24m 52s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 44852339658 capture.kernel_drops | Total | 103678933 decoder.pkts | Total | 44748644246 decoder.bytes | Total | 42596317608384 decoder.invalid | Total | 23390 decoder.ipv4 | Total | 39668212206 decoder.ipv6 | Total | 6252968947 decoder.ethernet | Total | 44748644246 decoder.tcp | Total | 34949571497
Here both counters differ (in millions) - total and dropped.
Also even in a short run as well - where it seems there is even bigger diff.
[41642] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 39079572, dropped 229553 [41643] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 41426017, dropped 306604 [41644] 29/11/2018 -- 23:38:47 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 35466864, dropped 215202 [41420] 29/11/2018 -- 23:38:47 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 440925 [41420] 29/11/2018 -- 23:38:51 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [41420] 29/11/2018 -- 23:38:53 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1746992 bytes, maximum: 33554432 [41420] 29/11/2018 -- 23:38:55 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [41420] 29/11/2018 -- 23:38:55 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0': pkts: 1563131426, drop: 10799950 (0.69%), invalid chksum: 0 [41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache ------------------------------------------------------------------------------------ Date: 11/29/2018 -- 23:38:44 (uptime: 0d, 00h 18m 29s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 1548885068 capture.kernel_drops | Total | 1766298 decoder.pkts | Total | 1547105082 decoder.bytes | Total | 1535747270554 decoder.invalid | Total | 1584 decoder.ipv4 | Total | 1424313847 decoder.ipv6 | Total | 181882866 decoder.ethernet | Total | 1547105082
Some more info
This is Suricata version 4.1.0-dev (rev 683be948)
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.28, linked against LibHTP v0.5.28
Suricata Configuration:
AF_PACKET support: yes
eBPF support: yes
XDP support: yes
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
liblzma support: no
hiredis support: no
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: yes, through luajit
libluajit: yes
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
Rust support: yes (default)
Rust strict mode: yes
Rust debug mode: no
Rust compiler: rustc 1.24.1
Rust cargo: cargo 0.25.0
Suricatasc install: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: yes
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
Host: x86_64-pc-linux-gnu
Compiler: clang-6.0 (exec name) / clang (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
PCAP_CFLAGS -I/usr/include
SECCFLAGS
Updated by Victor Julien almost 6 years ago
One thing that is less than ideal is that afpacket has 2 places it updates counters: the Stats API and the Livedev. Both are updated from the same source, but it remains a duplicate effort.
Updated by Andreas Herz over 5 years ago
- Assignee set to OISF Dev
- Target version set to TBD
Updated by Andreas Herz over 5 years ago
What would be a proper solution for this?
Actions