Suricata

Hi.

I noticed a problem when trying to threshold filestore rules. Thresholding made the alerts stop as expected but the file still got extracted and stored which was unexpected. As a workaround I decided to use flowbits and have one rule that sets a flowbit and the other rule that checks the flowbit then alerts and stores the file. I use the flowbit setting rule in my threshold.conf. But using flowbits this way doesn't work. Suricata stores a 0-byte file when I use flowbits for suppression.

My expectation is be that using a suppression on a file-store rule should stop both the alert and the file from being stored.

I first reported this issue in the Help forums and Andreas was able to reproduce the problem. https://forum.suricata.io/t/file-store-creates-empty-files/372

My testing to reproduce this issue with filestore v2 are below and pcap attached.

Thanks.

FileStore config in suricata-v2-test.yaml:

- file-store:
version: 2
enabled: yes
stream-depth: 10mb
dir: filestore-test

Test 1 - One filestore rule with thresholding.
Thresholding works (no alert is generated) but the file is extracted and stored.

Rule:
alert http !$HOME_NET any -> [$HOME_NET,!$GUEST_NET] any (msg:"FILEMAGIC Windows executable"; filemagic:"for MS Windows"; filestore; sid:7000001; rev:1;)

Threshold suppression set to on saddr.
# cat threshold.conf
suppress gen_id 1, sig_id 7000001, track by_src, ip 72.21.81.240/32`

~$ suricata -r /tmp/test.pcap -c suricata-v2-test.yaml --runmode single -k none -S file-extraction.rules -l /tmp
...
1/7/2020 -- 17:59:30 - <Info> - 1 rule files processed. 1 rules successfully loaded, 0 rules failed
1/7/2020 -- 17:59:30 - <Info> - Threshold config parsed: 1 rule(s) found
1/7/2020 -- 17:59:30 - <Info> - 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only
1/7/2020 -- 17:59:30 - <Info> - Checking file or directory /tmp/test.pcap
1/7/2020 -- 17:59:30 - <Info> - /tmp/test.pcap: Plain file, not a directory
1/7/2020 -- 17:59:30 - <Info> - Argument /tmp/test.pcap was a file
1/7/2020 -- 17:59:30 - <Info> - using magic-file /usr/share/file/magic
1/7/2020 -- 17:59:30 - <Info> - Using unix socket file '/var/run/suricata//suricata-command.socket'
1/7/2020 -- 17:59:30 - <Warning> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - Unix socket: UNIX socket bind(/var/run/suricata//suricata-command.socket) error: Address already in use
1/7/2020 -- 17:59:30 - <Warning> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - Unable to create unix command socket
1/7/2020 -- 17:59:30 - <Notice> - all 1 packet processing threads, 4 management threads initialized, engine started.
1/7/2020 -- 17:59:30 - <Info> - Starting file run for /tmp/test.pcap
1/7/2020 -- 17:59:30 - <Info> - pcap file /tmp/test.pcap end of file reached (pcap err code 0)
1/7/2020 -- 17:59:30 - <Notice> - Signal Received. Stopping engine.
1/7/2020 -- 17:59:32 - <Info> - time elapsed 1.570s
1/7/2020 -- 17:59:32 - <Notice> - Pcap-file module read 1 files, 2415 packets, 3066567 bytes
1/7/2020 -- 17:59:32 - <Info> - Alerts: 0
1/7/2020 -- 17:59:33 - <Info> - cleaning up signature grouping structure... complete

Output
eve-log: {"timestamp":"2020-06-27T02:29:46.268118+0000","flow_id":2040636389374308,"pcap_cnt":2264,"event_type":"fileinfo","vlan":503,"src_ip":"72.21.81.240","src_port":80,"dest_ip":"172.22.137.67","dest_port":64793,"proto":"TCP","http":{"hostname":"au.download.windowsupdate.com","url":"\/d\/msdownload\/update\/software\/defu\/2020\/06\/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe","http_user_agent":"Microsoft-Delivery-Optimization\/10.0","http_content_type":"application\/octet-stream","http_method":"GET","protocol":"HTTP\/1.1","status":206,"length":1048576},"app_proto":"http","fileinfo":{"filename":"\/d\/msdownload\/update\/software\/defu\/2020\/06\/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe","magic":"PE32+ executable (GUI) x86-64, for MS Windows","gaps":false,"state":"CLOSED","sha256":"8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034","stored":true,"file_id":1,"size":1048576,"tx_id":0}}

Filesystem:
$ ls -al /tmp/filestore-test/8b/8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034
-rw-r----- 1 user primarygroup 1048576 Jul 1 17:59 /tmp/filestore-test/8b/8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034

Test 2 - Two rules, one sets flow bit, other alerts and stores file.
An alert is generated and a 0-byte file is stored.

# cat file-extraction.rules.with.flowbits
alert http !$HOME_NET any -> [$HOME_NET,!$GUEST_NET] any (msg:"FILEMAGIC Windows executable"; filemagic:"for MS Windows"; filestore; flowbits:isset,file_extraction_store; sid:7000001; rev:1;)
alert http !$HOME_NET any -> [$HOME_NET,!$GUEST_NET] any (msg:"FILEMAGIC Windows executable flowbit set"; filemagic:"for MS Windows"; flowbits:noalert; flowbits:set,file_extraction_store; sid:7000002; rev:1;)

Threshold suppression set to on saddr and sid 7000002.
# cat threshold.conf
suppress gen_id 1, sig_id 7000002, track by_src, ip 72.21.81.240/32

$ suricata -r /tmp/test.pcap -c suricata-v2-test.yaml --runmode single -k none -S file-extraction.rules.with.flowbits -l /tmp
...
1/7/2020 -- 18:02:53 - <Info> - 1 rule files processed. 2 rules successfully loaded, 0 rules failed
1/7/2020 -- 18:02:53 - <Info> - Threshold config parsed: 1 rule(s) found
1/7/2020 -- 18:02:53 - <Info> - 2 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 2 inspect application layer, 0 are decoder event only
1/7/2020 -- 18:02:53 - <Info> - Checking file or directory /tmp/test.pcap
1/7/2020 -- 18:02:53 - <Info> - /tmp/test.pcap: Plain file, not a directory
1/7/2020 -- 18:02:53 - <Info> - Argument /tmp/test.pcap was a file
1/7/2020 -- 18:02:53 - <Info> - using magic-file /usr/share/file/magic
...
1/7/2020 -- 18:02:53 - <Notice> - all 1 packet processing threads, 4 management threads initialized, engine started.
1/7/2020 -- 18:02:53 - <Info> - Starting file run for /tmp/test.pcap
1/7/2020 -- 18:02:53 - <Info> - pcap file /tmp/test.pcap end of file reached (pcap err code 0)
1/7/2020 -- 18:02:53 - <Notice> - Signal Received. Stopping engine.
1/7/2020 -- 18:02:54 - <Info> - time elapsed 1.537s
1/7/2020 -- 18:02:56 - <Notice> - Pcap-file module read 1 files, 2415 packets, 3066567 bytes
1/7/2020 -- 18:02:56 - <Info> - Alerts: 1
1/7/2020 -- 18:02:56 - <Info> - cleaning up signature grouping structure... complete

Output:
eve-log:
... {"timestamp":"2020-06-27T02:29:46.268118+0000","flow_id":542092235028836,"pcap_cnt":2264,"event_type":"fileinfo","vlan":503,"src_ip":"72.21.81.240","src_port":80,"dest_ip":"172.22.137.67","dest_port":64793,"proto":"TCP","http":{"hostname":"au.download.windowsupdate.com","url":"\/d\/msdownload\/update\/software\/defu\/2020\/06\/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe","http_user_agent":"Microsoft-Delivery-Optimization\/10.0","http_content_type":"application\/octet-stream","http_method":"GET","protocol":"HTTP\/1.1","status":206,"length":1048576},"app_proto":"http","fileinfo":{"filename":"\/d\/msdownload\/update\/software\/defu\/2020\/06\/am_delta_57b2ef591610c4fb36d241b1c7fa48866541c834.exe","magic":"PE32+ executable (GUI) x86-64, for MS Windows","gaps":false,"state":"CLOSED","sha256":"8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034","stored":true,"file_id":1,"size":1048576,"tx_id":0}}

Filesystem:
$ ls -al
/tmp/filestore-test/8b:
-rw-r----- 1 user primarygroup 0 Jul 1 17:51 8b5bd2520d8a4dfbececc17de400de09aa566e4177fedb7c2e63fcd0a8aca034