Bug #2841
closedFalse positive alerts firing after upgrade suricata 3.0 -> 4.1.0 (4.1.x)
Description
Hello, oisf, could use your input on problem.
My corporate network has been running suricata 3.0. Recently upgraded to 4.1.0 then 4.1.1 and encountered issue.
Environment is CentOS release 7.4.1708 (Core) 3.10.0-862.14.4.el7.x86_64 on ~400 Dells
suricata version: suricata-4.1.0-ESG_1.el7.centos.x86_64, suricata-debuginfo-4.1.0-ESG_1.el7.centos.x86_64
pcre version: pcre-devel-8.32-17.el7.x86_64, pcre-8.32-17.el7.x86_64
lua version: lua-5.1.4-15.el7.x86_64
In Suricata 4.1.0 (and 4.1.1) (following recent upgrade from suricata 3.0) we noticed that under certain conditions false positive alerts are firing that should not be. In short, rules looking for HTTP packets are firing on ICMP data. It appears that the issue occurs on rules with http content modifiers where another rule in the ruleset uses an alert ip prefix and any content match. The packets must include an HTTP session followed by ICMP type packets (not that the rule should not match on the http session as the pcre does not match). I’ve also tested on suricata 4.1.2 and found that this issue is there as well. I’m not sure when the issue was introduced.
Rules in my local.rules:
alert ip any any -> any any (msg:"IP Any With Content"; content:"blahblah"; classtype:bad-unknown; priority:102; sid:5002203; rev:10;) alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"Test Rule no http"; flow:to_server,established; content:"POST"; http_method; pcre:"/\/.+\/[a-zA_Z0-9]{4}\_confirm\.php/"; classtype:bad-unknown; priority:102; sid:9999991; rev:1;)
Test output:
#> docker run -it --rm --volume $PWD/data:/data sensor -c /data/suricata.yaml -r ./data/out.pcap 7/1/2019 -- 15:25:55 - <Notice> - This is Suricata version 4.1.0 RELEASE 7/1/2019 -- 15:25:55 - <Notice> - using flow hash instead of active packets 7/1/2019 -- 15:25:55 - <Warning> - [ERRCODE: SC_ERR_DEPRECATED_CONF(274)] - deprecated 'force-md5' option found. Please use 'force-hash: [md5]' instead 7/1/2019 -- 15:25:55 - <Warning> - [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2015985, gid 1: unknown rule 7/1/2019 -- 15:25:55 - <Notice> - all 7 packet processing threads, 4 management threads initialized, engine started. 7/1/2019 -- 15:25:55 - <Notice> - Signal Received. Stopping engine. 7/1/2019 -- 15:25:55 - <Notice> - Pcap-file module read 1 files, 72 packets, 10078 bytes #> cat ./data/fast.log 01/02/2019-17:57:19.054415 [**] [1:9999991:1] Test Rule no http [**] [Classification: Potentially Bad Traffic] [Priority: 102] {ICMP} 3.20.244.244:2563 -> 10.228.203.106:0
Note that the rule which his supposed to match on an HTTP session is alerting on an ICMP packet.
Eve.json alert also has http data for an icmp protocol:
{ "timestamp": "2019-01-02T17:57:19.054415+0000", "flow_id": 1426130925450708, "pcap_cnt": 22, "event_type": "alert", "src_ip": "<redacted>", "dest_ip": "<redacted>", "proto": "ICMP", "icmp_type": 3, "icmp_code": 10, "tx_id": 0, "alert": { "action": "allowed", "gid": 1, "signature_id": 9999991, "rev": 1, "signature": "Test Rule no http", "category": "Potentially Bad Traffic", "severity": 102 }, "http": { "hostname": "<redacted>", "url": "/em/event", "http_user_agent": "Wget/1.11.4 Red Hat modified", "http_method": "POST", "protocol": "HTTP/1.0", "length": 0 }, "app_proto": "http", "flow": { "pkts_toserver": 4, "pkts_toclient": 0, "bytes_toserver": 598, "bytes_toclient": 0, "start": "2019-01-02T17:56:06.194004+0000" }, "payload": "<redacted>",==", "payload_printable": "<redacted>", "stream": 0 }
Any thoughs on how to proceed on this would help.
Talked with Andreas Herz on this via oisf forum.
Files
Updated by Victor Julien over 5 years ago
- Copied from Bug #2769: False positive alerts firing after upgrade suricata 3.0 -> 4.1.0 added
Updated by Charles Dillard over 5 years ago
- File pic-for-oisf-2841.png pic-for-oisf-2841.png added
Victor Julien wrote:
Hello, oisf, could use your input on problem.
My corporate network has been running suricata 3.0. Recently upgraded to 4.1.0 then 4.1.1 and encountered issue.
Environment is CentOS release 7.4.1708 (Core) 3.10.0-862.14.4.el7.x86_64 on ~400 Dells
suricata version: suricata-4.1.0-ESG_1.el7.centos.x86_64, suricata-debuginfo-4.1.0-ESG_1.el7.centos.x86_64
pcre version: pcre-devel-8.32-17.el7.x86_64, pcre-8.32-17.el7.x86_64
lua version: lua-5.1.4-15.el7.x86_64
In Suricata 4.1.0 (and 4.1.1) (following recent upgrade from suricata 3.0) we noticed that under certain conditions false positive alerts are firing that should not be. In short, rules looking for HTTP packets are firing on ICMP data. It appears that the issue occurs on rules with http content modifiers where another rule in the ruleset uses an alert ip prefix and any content match. The packets must include an HTTP session followed by ICMP type packets (not that the rule should not match on the http session as the pcre does not match). I’ve also tested on suricata 4.1.2 and found that this issue is there as well. I’m not sure when the issue was introduced.
Rules in my local.rules:
[...]
Test output:
[...]
Note that the rule which his supposed to match on an HTTP session is alerting on an ICMP packet.Eve.json alert also has http data for an icmp protocol:
[...]
Any thoughs on how to proceed on this would help.
Talked with Andreas Herz on this via oisf forum.
Here is status on changes. Our engineers cloned git repo to test fix. This produced files for ./configure and make Produced a tar.gz file. Built rpm from this tar.gz. This was applied to a test device. According to our incident response team, the problem persists. See ICMP and HTTP.
There is an attached file that shows what incident response team is seeing.
Updated by Victor Julien over 5 years ago
Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.
Updated by Charles Dillard over 5 years ago
Victor Julien wrote:
Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.
Will get you one.
Updated by Charles Dillard over 5 years ago
Charles Dillard wrote:
Victor Julien wrote:
Ok. I have no way to reproduce this without a test case. It would be very helpful if you can provide one. Privately if you prefer.
Will get you one.
Can you give us more details on what you require for a "test case"? In the past we've provided pcap files and rules to test on.
Is it possible to set up a skype call with your team?
Updated by Victor Julien over 5 years ago
Hi Charles, a test case is what you have provided before: pcap, rules, plus config and instructions if necessary. The test case provided showed one issue, but not the icmp issue.
Can you reproduce the issue with just the pcaps and rules in this ticket? If so please give detailed instructions, because I failed to reproduce the issue with them.
Updated by Charles Dillard over 5 years ago
Victor Julien wrote:
Hi Charles, a test case is what you have provided before: pcap, rules, plus config and instructions if necessary. The test case provided showed one issue, but not the icmp issue.
Can you reproduce the issue with just the pcaps and rules in this ticket? If so please give detailed instructions, because I failed to reproduce the issue with them.
I will work on getting all that.
Updated by Victor Julien over 5 years ago
- Affected Versions 4.1.1, 4.1.2 added
Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681
Updated by Charles Dillard over 5 years ago
Victor Julien wrote:
Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681
Still working on getting authorization to send OISF the needed files.
Updated by Charles Dillard over 5 years ago
Charles Dillard wrote:
Victor Julien wrote:
Part of the issue is fixed in the 4.1.x branch now as well: https://github.com/OISF/suricata/pull/3681
Still working on getting authorization to send OISF the needed files.
Victor, hello, Hilary Curtis, who is our team lead, sent email to you asking if you or someone on your team would be available for a skype teleconference or other interactive means. Our incident response team leader would demonstrate the issue we are seeing with icmp/http alerts. This would circumvent the problem of sending out sensitive files. This is at your convenience.
Thanks for you consideration on this issue, Victor.
Updated by Victor Julien over 5 years ago
- Copied to Bug #2862: pcre related FP in HTTP inspection (4.1.x) added
Updated by Victor Julien over 5 years ago
- Target version changed from 4.1.3 to 70
I've split off #2862 from this ticket for the pcre issue that is fixed.
This ticket will remain open pending a test case for the icmp issue (private comms about this in progress).
Updated by Duane Howard over 5 years ago
fwiw, I'm seeing this behavior as well, but it seems extremely difficult to reproduce.
For some reason when I run with a pcap and just the rule that triggered I get no alerts, but when I run with my full rule-set I get an alert (on the same pcap). I'm struggling to figure out how to narrow this to an easy to reproduce test case.
$ suricata -r out2.pcap -l /tmp/surilog --runmode=single -S test.rules <snip> 21/5/2019 -- 12:58:52 - <Info> - Alerts: 0
$ suricata -r out2.pcap -l /tmp/surilog --runmode=single <snip> 20/5/2019 -- 20:31:56 - <Notice> - Pcap-file module read 1 files, 50 packets, 6324 bytes 20/5/2019 -- 20:31:56 - <Info> - Alerts: 1
Updated by Peter Manev over 5 years ago
Is it possible to narrow it down to a rules file/set of rules?
Updated by Victor Julien about 5 years ago
- Status changed from Assigned to Closed
- Target version changed from 70 to 4.1.5