Bug #2099
closedaf_packet: In IPS mode some some packets are not copied
Description
Kernel 4.9.24
Suricata 3.2.x or 4.0dev, does not matter.
1 test alert signature.
Testing with TCPReplay's sample PCAP: https://s3.amazonaws.com/tcpreplay-pcap-files/bigFlows.pcap
Sending 39580750 to eno50 interface (PCAP replayed 50 times), speed capped @ 1.15 Mpps
af-packet:
- interface: eno49
threads: 10
cluster-id: 99
cluster-type: cluster_flow
defrag: no
use-mmap: yes
mmap-locked: yes
tpacket-v3: yes
ring-size: 524288
block-size: 524288
checksum-checks: no
copy-mode: ips
copy-iface: eno50
- interface: eno50
threads: 10
cluster-id: 98
cluster-type: cluster_flow
defrag: no
use-mmap: yes
mmap-locked: yes
tpacket-v3: yes
ring-size: 524288
block-size: 524288
checksum-checks: no
copy-mode: ips
copy-iface: eno49
No drops on input in kernel or Suricata, but only 39 230 764 (349 986 packets gone) are sent to eno49 interface (checked by ethtool -S eno49 | grep tx_packets and sniffer connected to eno49).
This count is always the same for the same PCAP file.
Maybe Suricata drops some packets internally and does not copy them to output interface according to some internal logic?
Thanks in advance!
Resulting stats.log:
------------------------------------------------------------------------------------
Date: 4/23/2017 -- 12:11:42 (uptime: 0d, 00h 01m 39s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
capture.kernel_packets | Total | 39580750
decoder.pkts | Total | 39580750
decoder.bytes | Total | 17770889200
decoder.ipv4 | Total | 39558950
decoder.ipv6 | Total | 21950
decoder.ethernet | Total | 39580750
decoder.tcp | Total | 31694700
decoder.udp | Total | 7656750
decoder.icmpv4 | Total | 213300
decoder.icmpv6 | Total | 1700
decoder.teredo | Total | 150
decoder.avg_pkt_size | Total | 448
decoder.max_pkt_size | Total | 1514
tcp.sessions | Total | 19918
tcp.pseudo | Total | 2190
tcp.syn | Total | 1598753
tcp.synack | Total | 300291
tcp.rst | Total | 70750
tcp.stream_depth_reached | Total | 3
tcp.reassembly_gap | Total | 2
detect.alert | Total | 86
app_layer.flow.http | Total | 2879
app_layer.tx.http | Total | 6604
app_layer.flow.tls | Total | 1448
app_layer.flow.failed_tcp | Total | 102
app_layer.flow.failed_udp | Total | 5055
flow.spare | Total | 524288
flow_mgr.flows_checked | Total | 779
flow_mgr.flows_notimeout | Total | 779
flow_mgr.rows_checked | Total | 1048576
flow_mgr.rows_skipped | Total | 1047814
flow_mgr.rows_maxlen | Total | 2
tcp.memuse | Total | 800000000
tcp.reassembly_memuse | Total | 12332832
flow.memuse | Total | 221576184
Files
Updated by Igor Novgorodov over 7 years ago
Update: If i set af_packet's copy-mode to 'tap' then no packets are lost.
Updated by Peter Manev over 7 years ago
We have investigated a similar problem with Eric and he has a patch that fixes our test case.
Would you be willing to try it out and confirm if it fixes the problem for you?
Updated by Eric Leblond over 7 years ago
- File 0001-stream-tcp-add-option-to-accept-invalid-packets.patch 0001-stream-tcp-add-option-to-accept-invalid-packets.patch added
Hi Igor. Can you try the attached patch (on top of git master) ? To activate the feature, you need to set drop-invalid: no in the stream section of the YAML.
Updated by Igor Novgorodov over 7 years ago
Thanks! i'll try the patch ASAP after i get back from vacation.
Updated by Andreas Herz over 7 years ago
- Assignee set to OISF Dev
- Target version set to TBD
Updated by Victor Julien over 7 years ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Eric Leblond
- Target version changed from TBD to 70
Updated by Igor Novgorodov over 7 years ago
Just checked out latest master from git, applied the patch.
It didn't applied cleanly, according to commit log, there was a lot of work on stream module, although only a minor fix was required.
I can confirm that patch works as expected - all packets are gone through AF_PACKET bridge node fine:
source# # tcpreplay -K --intf1=eno50 -l 50 -p 1150000 /opt/bigFlows.pcap
File Cache is enabled
Test start: 2017-05-09 18:18:49.498752 ...
Test complete: 2017-05-09 18:19:25.026170
Actual: 39580750 packets (17770889200 bytes) sent in 35.52 seconds
Rated: 500207988.2 Bps, 4001.66 Mbps, 1114103.35 pps
Flows: 40686 flows, 1145.20 fps, 39558950 flow packets, 21800 non-flow
Statistics for network device: eno50
Successful packets: 39580750
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
suricata# suricata -c /etc/suricata/suricata.yaml --af-packet [6106] 9/5/2017 -- 18:15:40 - (conf-yaml-loader.c:296) <Info> (ConfYamlParse) -- Including configuration file /etc/suricata/rules-custom.yaml at parent node rule-files. [6106] 9/5/2017 -- 18:15:40 - (suricata.c:1100) <Notice> (LogVersion) -- This is Suricata version 4.0dev (rev f18c976) [6106] 9/5/2017 -- 18:15:41 - (tm-threads.c:2178) <Notice> (TmThreadWaitOnThreadInit) -- all 20 packet processing threads, 4 management threads initialized, engine started. [6106] 9/5/2017 -- 18:18:10 - (suricata.c:2728) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [6106] 9/5/2017 -- 18:18:12 - (util-device.c:283) <Notice> (LiveDeviceListClean) -- Stats for 'eno49': pkts: 0, drop: 0 (-nan%), invalid chksum: 0 [6106] 9/5/2017 -- 18:18:12 - (util-device.c:283) <Notice> (LiveDeviceListClean) -- Stats for 'eno50': pkts: 39580750, drop: 0 (0.00%), invalid chksum: 0
sink# # netsniff-ng --in eno50 -s
Running! Hang up with ^C!
39580750 packets incoming (0 unread on exit)
39580750 packets passed filter
0 packets failed filter (out of space)
0.0000% packet droprate
49 sec, 751920 usec in total
Updated by Victor Julien about 7 years ago
- Status changed from Assigned to Closed
- Target version deleted (
70)
Erics patch was merged during 4.0 development.