Actions
Bug #705
closedhttp.log missing lots of requests under high traffic load
Affected Versions:
Effort:
Difficulty:
Label:
Description
using -
suricata --build-info [23628] 10/1/2013 -- 00:49:59 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 5f4c528) [23628] 10/1/2013 -- 00:49:59 - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING [23628] 10/1/2013 -- 00:49:59 - (suricata.c:647) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture [23628] 10/1/2013 -- 00:49:59 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:655) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:658) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:661) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:664) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:667) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector [23628] 10/1/2013 -- 00:49:59 - (suricata.c:677) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2 [23628] 10/1/2013 -- 00:49:59 - (suricata.c:680) <Info> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11
root@suricata:/var/data/regit/log/suricata# time tcpdump -i eth3 -n -s 0 -w - | grep -a -o -E "Host\: .*|GET \/.*" >gets.txt tcpdump: WARNING: eth3: no IPv4 address assigned tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes ^C637676 packets captured 4955063 packets received by filter 4317303 packets dropped by kernel 1 packet dropped by interface real 0m3.942s user 0m2.824s sys 0m1.420s root@suricata:/var/data/regit/log/suricata# grep GET gets.txt |wc -l 1529
So for 3 seconds we have 1529 HTTP GET requests with tcpdump (and that is just GET)
root@suricata:/var/data/regit/log/suricata# date && wc -l http.log Thu Jan 10 00:14:16 EET 2013 518476 http.log root@suricata:/var/data/regit/log/suricata# date && wc -l http.log Thu Jan 10 00:14:18 EET 2013 518480 http.log root@suricata:/var/data/regit/log/suricata# date && wc -l http.log Thu Jan 10 00:14:19 EET 2013 518489 http.log
and for 3 sec with Suricata we get 13 http logs in http.log on the same 10G/s traffic
root@suricata:/var/data/regit/log/suricata# wc -l gets.txt 3209 gets.txt root@suricata:/var/data/regit/log/suricata# tail -8 gets.txt Host: redirector.c.youtube.com GET /v302708503/6baa/fIDnzBb3MPE.jpg HTTP/1.1 Host: cs302708.userapi.com GET /csi?v=2&s=youtube&action=results&e=901312,916623,922911,919108,920704,912806,922403,922405,929901,913605,929104,929110,920201,913302,919009,911116,926403,901451&li=1&rt=ct.281,js_head.591,js_page.863,js_foot.865,ol.1208,aft.1208 HTTP/1.1 Host: csi.gstatic.com GET /socket.io/1/?t=1357769748997 HTTP/1.1 Host: ws.9gag.com:8084 GET /vi/9d6IsU01-HE/hqdefault.jpg HTTP/1.1 root@suricata:/var/data/regit/log/suricata#
my http config part in suriacta.yaml
# a line based log of HTTP requests (no alerts)
- http-log:
enabled: yes
filename: http.log
append: yes
extended: yes # enable this for extended logging information
#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
just for general info - extended: yes/no - it has no effect on the number of logs written.
I think 1.3.1 didn't have that issue.
Actions