Actions
Bug #2718
openpkts/drops counters discrepancy
Affected Versions:
Effort:
Difficulty:
Label:
Description
Running git with AFPv3 (info below at the end) and seeing discrepancy in total and dropped packet counters on the same Suricata run between suricata.log/stdout and stats.log/eve stats.json
Seeing this on 2 different high speed Suricata deployments.
------------------------------------------------------------------------------------ Date: 11/29/2018 -- 22:53:39 (uptime: 0d, 20h 58m 32s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 34526257089 capture.kernel_drops | Total | 364864207 decoder.pkts | Total | 34161381126 decoder.bytes | Total | 27521454249680 decoder.invalid | Total | 125528 decoder.ipv4 | Total | 33874924190 decoder.ipv6 | Total | 299143889 decoder.ethernet | Total | 34161381126 [1841] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#15-eth3) Kernel: Packets 1748200153, dropped 23108264 [1841] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#15-eth3) Files extracted 0 [1842] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#16-eth3) Kernel: Packets 13, dropped 0 [1842] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#16-eth3) Files extracted 0 [1843] 29/11/2018 -- 22:53:39 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#01-eth0) Kernel: Packets 1235440, dropped 119484 [1843] 29/11/2018 -- 22:53:39 - (log-filestore.c:568) <Info> (LogFilestoreLogExitPrintStats) -- (W#01-eth0) Files extracted 0 [1701] 29/11/2018 -- 22:53:39 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 3052 [1701] 29/11/2018 -- 22:53:40 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [1701] 29/11/2018 -- 22:53:42 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1622144 bytes, maximum: 16777216 [1701] 29/11/2018 -- 22:53:42 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth2': pkts: 8251080725, drop: 42475597 (0.51%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth3': pkts: 26275081393, drop: 322269126 (1.23%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:42 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'eth0': pkts: 1235440, drop: 119484 (9.67%), invalid chksum: 0 [1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [1701] 29/11/2018 -- 22:53:43 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache root@suricata:~#
Above -
Total counter(s, combined) has a mismatch
Drop counter(s , combined) is good - exactly the same
[74190] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#33-enp59s0) Kernel: Packets 1063135836, dropped 1969930 [74191] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#34-enp59s0) Kernel: Packets 1069756460, dropped 4021948 [74192] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#35-enp59s0) Kernel: Packets 1128621774, dropped 1065953 [74193] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#36-enp59s0) Kernel: Packets 1143644762, dropped 1572917 [74194] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#37-enp59s0) Kernel: Packets 1029388098, dropped 110874 [74195] 29/11/2018 -- 23:06:27 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 1049514375, dropped 237793 [74196] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 1109232251, dropped 2193864 [74197] 29/11/2018 -- 23:06:28 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 1125246426, dropped 1889525 [73968] 29/11/2018 -- 23:06:28 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 16159124 [73968] 29/11/2018 -- 23:06:37 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [73968] 29/11/2018 -- 23:06:45 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 33554400 bytes, maximum: 33554432 [73968] 29/11/2018 -- 23:06:48 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [73968] 29/11/2018 -- 23:06:48 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0': pkts: 44861666289, drop: 106594312 (0.24%), invalid chksum: 0 [73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [73968] 29/11/2018 -- 23:06:48 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache ------------------------------------------------------------------------------------ Date: 11/29/2018 -- 23:06:25 (uptime: 0d, 06h 24m 52s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 44852339658 capture.kernel_drops | Total | 103678933 decoder.pkts | Total | 44748644246 decoder.bytes | Total | 42596317608384 decoder.invalid | Total | 23390 decoder.ipv4 | Total | 39668212206 decoder.ipv6 | Total | 6252968947 decoder.ethernet | Total | 44748644246 decoder.tcp | Total | 34949571497
Here both counters differ (in millions) - total and dropped.
Also even in a short run as well - where it seems there is even bigger diff.
[41642] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#38-enp59s0) Kernel: Packets 39079572, dropped 229553 [41643] 29/11/2018 -- 23:38:46 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#39-enp59s0) Kernel: Packets 41426017, dropped 306604 [41644] 29/11/2018 -- 23:38:47 - (source-af-packet.c:2655) <Perf> (ReceiveAFPThreadExitStats) -- (W#40-enp59s0) Kernel: Packets 35466864, dropped 215202 [41420] 29/11/2018 -- 23:38:47 - (counters.c:830) <Info> (StatsLogSummary) -- Alerts: 440925 [41420] 29/11/2018 -- 23:38:51 - (ippair.c:290) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216 [41420] 29/11/2018 -- 23:38:53 - (host.c:294) <Perf> (HostPrintStats) -- host memory usage: 1746992 bytes, maximum: 33554432 [41420] 29/11/2018 -- 23:38:55 - (detect-engine-build.c:1733) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [41420] 29/11/2018 -- 23:38:55 - (util-device.c:329) <Notice> (LiveDeviceListClean) -- Stats for 'enp59s0': pkts: 1563131426, drop: 10799950 (0.69%), invalid chksum: 0 [41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1070) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch [41420] 29/11/2018 -- 23:38:55 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache ------------------------------------------------------------------------------------ Date: 11/29/2018 -- 23:38:44 (uptime: 0d, 00h 18m 29s) ------------------------------------------------------------------------------------ Counter | TM Name | Value ------------------------------------------------------------------------------------ capture.kernel_packets | Total | 1548885068 capture.kernel_drops | Total | 1766298 decoder.pkts | Total | 1547105082 decoder.bytes | Total | 1535747270554 decoder.invalid | Total | 1584 decoder.ipv4 | Total | 1424313847 decoder.ipv6 | Total | 181882866 decoder.ethernet | Total | 1547105082
Some more info
This is Suricata version 4.1.0-dev (rev 683be948) Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC RUST SIMD support: SSE_4_2 SSE_4_1 SSE_3 Atomic intrisics: 1 2 4 8 16 byte(s) 64-bits, Little-endian architecture GCC version 4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final), C version 199901 compiled with _FORTIFY_SOURCE=0 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.28, linked against LibHTP v0.5.28 Suricata Configuration: AF_PACKET support: yes eBPF support: yes XDP support: yes PF_RING support: no NFQueue support: no NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes liblzma support: no hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: yes, through luajit libluajit: yes libgeoip: yes Non-bundled htp: no Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: yes Rust support: yes (default) Rust strict mode: yes Rust debug mode: no Rust compiler: rustc 1.24.1 Rust cargo: cargo 0.25.0 Suricatasc install: yes Profiling enabled: no Profiling locks enabled: no Development settings: Coccinelle / spatch: yes Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr --sysconfdir /etc --localstatedir /var Host: x86_64-pc-linux-gnu Compiler: clang-6.0 (exec name) / clang (real) GCC Protect enabled: no GCC march native enabled: yes GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers PCAP_CFLAGS -I/usr/include SECCFLAGS
Updated by Victor Julien almost 6 years ago
One thing that is less than ideal is that afpacket has 2 places it updates counters: the Stats API and the Livedev. Both are updated from the same source, but it remains a duplicate effort.
Updated by Andreas Herz over 5 years ago
- Assignee set to OISF Dev
- Target version set to TBD
Updated by Andreas Herz over 5 years ago
What would be a proper solution for this?
Actions