Suricata

Can you please describe/detail a bit more specific case scenario.

In the first part of your initial description - I am left with the impression that you have troubles downloading a large file with any browser or using wget - aka an end user download problem as opposed to file extraction. Is that correct or?

I do not encounter any download problem with wget even it also encounter timeout. However, I encounter download timeout with any browser even without md5 or filestore feature.

For example, I tried to download the following iso files with any browser, the download will be failed.

http://cdimage.kali.org/kali-2.0/kali-linux-2.0-amd64.iso
https://www.backbox.org/downloads
https://www.microsoft.com/en-hk/software-download/windows10ISO

However, Ubuntu iso can be downloaded with browser and without any problem.

Can you share your yaml?

It's interesting that you see this both with http and https links. That means that libhtp isn't a factor, as it's not used for https downloads.

My suricata.yaml is the below :

http://pastebin.com/u2An2mmM

I ran into this same issue yesterday. I compiled suricata 3.0RC2 from source to enable rule profiling and think I've narrowed it down. Here's the profiling that I did on basically just downloading a ubuntu iso (cancelled after about 100MB since the speed dropped to < 1% of my maximum bandwidth):

 --------------------------------------------------------------------------
  Date: 12/8/2015 -- 22:22:51
  --------------------------------------------------------------------------
   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
  1        2221000      1        1        656034977    13.40  60458    0        12811964    10851.09    0.00        10851.09
  2        1000003      1        1        500851016    10.23  68496    0        12701488    7312.12     0.00        7312.12
  3        2221002      1        1        203809237    4.16   19536    0        6602884     10432.50    0.00        10432.50
  4        2020865      1        3        162494234    3.32   2792     0        3372952     58199.94    0.00        58199.94
  5        2017552      1        6        115986768    2.37   19536    0        7618772     5937.08     0.00        5937.08

And the offending rules:
2221000
/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP unknown error"; flow:established; app-layer-event:http.unknown_error; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221000; rev:1;)

1000003 (custom)
/etc/suricata/rules/local.rules:alert http any any -> any any (msg:"Flash video download"; filemagic:"Flash Video"; filestore; noalert; sid:1000003; rev:1;)

2221002
/etc/suricata/rules/http-events.rules:alert http any any -> any any (msg:"SURICATA HTTP request field missing colon"; flow:established,to_server; app-layer-event:http.request_field_missing_colon; flowint:http.anomaly.count,+,1; classtype:protocol-command-decode; sid:2221002; rev:1;)

2020865
/etc/suricata/rules/emerging-current_events.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET CURRENT_EVENTS Nuclear EK Landing Apr 08 2015"; flow:established,from_server; content:"Server|3a 20|nginx"; http_header; file_data; content:"Q|22|"; fast_pattern; content:"length"; pcre:"/^\s*?\<\s*?10/Rs"; content:"replace"; within:500; pcre:"/^\s*?\x28\s*?\x22\s\x22\s*?,\s*?\x22(?:\!(?:\x22\s*?\+\s*?\x22)?)?Q(?:\x22\s*?\+\s*?\x22)?Q\x22/Rs"; classtype:trojan-activity; sid:2020865; rev:3;)

2017552
/etc/suricata/rules/emerging-current_events.rules:alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET CURRENT_EVENTS Cushion Redirection"; flow:established,to_server; content:"/index.php?"; http_uri; content:"="; distance:1; within:1; http_uri; content:!"=aHR0"; http_uri; fast_pattern; pcre:"/\/index\.php\?[a-z]=[A-Za-z0-9\/\+]*?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+?(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+(?:(?:N1cm[kw]|RpbWU)9|(?:zdXJ[ps]|0aW1l)P|c3Vy(?:aT|bD)|dGltZT)[A-Za-z0-9\/\+]+={0,2}$/U"; reference:url,malwaremustdie.blogspot.co.uk/2013/09/302-redirector-new-cushion-attempt-to.html; classtype:trojan-activity; sid:2017552; rev:6;)

Looks like those HTTP events are heavy.

Samiux, as a test, could you set these 2 libhtp settings to something much lower?

request-body-limit: 0
           response-body-limit: 0

e.g.
request-body-limit: 1mb
response-body-limit: 1mb

And see if that makes the issue go away?

Peter, do you have the same depth settings?

Could you test https://github.com/inliniac/suricata/pull/1788, think I found the issue.

Hi Victor,

I spoke too soon and pointed the finger at those rules, but after commenting them out, the issue continued.

I just compiled that pull request and it looks like it's fixed the issue. Download speed hasn't dropped and my AFPacket thread isn't pegging the cpu anymore.

This patch only works for NFQueue IPS mode but not AF_PACKET IPS mode.

Can you expand on that Samiux? I tested with AF_PACKET IPS mode.

The patch fixes the problem. The AF_PACKET IPS mode problem is due to the ET POLICY rule. When the rule is disabled, the problem solved.

Sorry, I spoke too early that the problem was fixed.

Even I disabled sid 2000419 and 2018959, Windows 10 iso stalled at 285MB. There was no alert during download.

Suricata is running af-packet mode inline (IPS) with ET Open rules.

When download kali, it failed at 1.6gb.

I applied this pull request (https://github.com/inliniac/suricata/pull/1790) and download Windows 10 via Firefox from https://www.microsoft.com/en-us/software-download/techbench with the following setting at suricata.yaml and I get success result. Some false positive drop rules are disabled, for example ET WEB_CLIENT. The download is in almost in full speed of my internet connection (10/10Mb). Meanwhile, Kali Linux 2.0 can be downloaded with lower values at libhtp section.

libhtp:

default-config:
           personality: IDS

1. Can be specified in kb, mb, gb. Just a number indicates
2. it's in bytes.
   request-body-limit: 10mb
   response-body-limit: 10mb

1. inspection limits
   request-body-minimal-inspect-size: 32kb
   request-body-inspect-window: 4kb
   response-body-minimal-inspect-size: 8192kb
   response-body-inspect-window: 1024kb