Support #2548
closedsuricata flow management in 10gbs environment
Description
I have a problem in using suricata as ids mode in idc 10gbps environment.
I found finallly flow recycler cleanup flows slower than flows created. this lead to flow spare queue decrease to zero and recycle queue very large。especially when enter emergency mode,flow manager module schedule more frequent than flow recycler module。
I decreased flow-timeout conf,configed more than one flow recycler modules, but not effictive as assumed。
flow:
memcap: 2gb
hash-size: 1048576
prealloc: 1048576
emergency-recovery: 30
managers: 1 # default to one flow manager
recyclers: 6 # default to one flow recycler thread
flow-timeouts:
default:
new: 30
established: 200
closed: 5
bypassed: 100
emergency-new: 10
emergency-established: 100
emergency-closed: 0
emergency-bypassed: 50
tcp:
new: 20
established: 200
closed: 5
bypassed: 60
emergency-new: 5
emergency-established: 100
emergency-closed: 0
emergency-bypassed: 50
stream:
memcap: 2gb
checksum-validation: yes # reject wrong csums
prealloc-sessions: 65536 # sessions prealloc'd per stream thread
inline: no # auto will use inline mode in IPS mode, yes or no set it statically
bypass: yes
reassembly:
memcap: 10gb
depth: 512kb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
#randomize-chunk-range: 10
#raw: yes
segment-prealloc: 65536
#check-overlap-different-data: true
The stats as below:
------------------------------------------------------------------------------------
Date: 7/23/2018 -- 16:05:28 (uptime: 0d, 00h 59m 29s)
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
flow_mgr.closed_pruned | FM#01 | 77801406
flow_mgr.new_pruned | FM#01 | 14559621
flow_mgr.est_pruned | FM#01 | 17102653
flow_mgr.bypassed_pruned | FM#01 | 910
flow.spare | FM#01 | 142790
flow.emerg_mode_entered | FM#01 | 1
flow.tcp_reuse | FM#01 | 2266271
flow_mgr.flows_checked | FM#01 | 9386320
flow_mgr.flows_notimeout | FM#01 | 3213988
flow_mgr.flows_timeout | FM#01 | 6172332
flow_mgr.flows_timeout_inuse | FM#01 | 1446235
flow_mgr.flows_removed | FM#01 | 4726097
flow_mgr.rows_checked | FM#01 | 1048576
flow_mgr.rows_skipped | FM#01 | 4
flow_mgr.rows_empty | FM#01 | 341
flow_mgr.rows_maxlen | FM#01 | 67
tcp.memuse | Global | 1804674600
tcp.reassembly_memuse | Global | 10737418208
dns.memuse | Global | 1398589
http.memuse | Global | 2783051133
flow.memuse | Global | 2147483488
Is there any method or slolution for this problem?
Files