Bug #2395
closedFile_data inspection depth while inspecting base64 decoded data
Added by Bryant Smith about 7 years ago. Updated over 5 years ago.
Description
I had noticed that when needing to inspect SMTP traffic that suricata can use file_data to inspect the base64 encoded attachments. This doesn't seem to work correctly all of the time. It seems that anything that is small seems to work but if the attachment is large I can't inspect deep into the payload or sometimes not even at the beginning of the payload. I've attached a sample pcap and simple rule that looks for two things. MZ at the beginning of the file_data payload and CreateFont which shows up further in. I've tried adjusting the various settings for libhtp but ended up with the same results.
/opt/suricata/bin/suricata -V
This is Suricata version 4.0.0 RELEASE
--------------------------------------------------------------------------
Date: 12/26/2017 -- 07:18:17. Sorted by: number of matches.
--------------------------------------------------------------------------
Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
-------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
1 123456 1 1 26488 100.00 1 0 26488 26488.00 0.00 26488.00Files
| sample.rules (212 Bytes) sample.rules | Bryant Smith, 12/26/2017 08:11 AM | ||
| export.pcap (157 KB) export.pcap | Bryant Smith, 12/26/2017 08:11 AM | ||
| test-mail-attach.pcap (5.69 MB) test-mail-attach.pcap | Gabriel Somlo, 09/24/2018 05:03 PM | ||
| etc_suricata_suricata.yaml (72.8 KB) etc_suricata_suricata.yaml | Gabriel Somlo, 08/19/2019 11:16 PM | ||
| etc_suricata_classification.config (4.16 KB) etc_suricata_classification.config | Gabriel Somlo, 08/19/2019 11:16 PM | ||
| var_lib_suricata_rules_suricata.rules (315 Bytes) var_lib_suricata_rules_suricata.rules | Gabriel Somlo, 08/19/2019 11:16 PM | ||
| suricata.rules (436 Bytes) suricata.rules | Gabriel Somlo, 09/22/2019 01:00 PM |
Updated by Francis Trudeau about 7 years ago
Your unmodified rule fires for me here:
This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - This is Suricata version 4.0.0 RELEASE
27/12/2017 -- 09:49:02 - <Notice> - Ring buffer initialized with 5 files.
27/12/2017 -- 09:49:02 - <Notice> - all 5 packet processing threads, 4 management threads initialized, engine started.
27/12/2017 -- 09:49:02 - <Notice> - Signal Received. Stopping engine.
27/12/2017 -- 09:49:02 - <Notice> - Pcap-file module read 89 packets, 159659 bytes
12/19/2017-13:01:34.001543 [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25
Here is the SMTP section of my yaml:
smtp:
enabled: yes
# Configure SMTP-MIME Decoder
mime:
# Decode MIME messages from SMTP transactions
# (may be resource intensive)
# This field supercedes all others because it turns the entire
# process on or off
decode-mime: yes
# Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
decode-base64: yes
decode-quoted-printable: yes
# Maximum bytes per header data value stored in the data structure
# (default is 2000)
header-value-depth: 2000
# Extract URLs and save in state data structure
extract-urls: yes
# Set to yes to compute the md5 of the mail body. You will then
# be able to journalize it.
body-md5: yes
# Configure inspected-tracker for file_data keyword
inspected-tracker:
content-limit: 100000
content-inspect-min-size: 32768
content-inspect-window: 4096
CreateFontIndirectW is at offset 11432, so I imagine your limits are lower than mine. See if you can tweak those to get it to fire.
Bryant Smith wrote:
I had noticed that when needing to inspect SMTP traffic that suricata can use file_data to inspect the base64 encoded attachments. This doesn't seem to work correctly all of the time. It seems that anything that is small seems to work but if the attachment is large I can't inspect deep into the payload or sometimes not even at the beginning of the payload. I've attached a sample pcap and simple rule that looks for two things. MZ at the beginning of the file_data payload and CreateFont which shows up further in. I've tried adjusting the various settings for libhtp but ended up with the same results.
Updated by Bryant Smith about 7 years ago
Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.
smtp:
enabled: yes
# Configure SMTP-MIME Decoder
mime:
# Decode MIME messages from SMTP transactions
# (may be resource intensive)
# This field supercedes all others because it turns the entire
# process on or off
decode-mime: yes
# Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
decode-base64: yes
decode-quoted-printable: yes
# Maximum bytes per header data value stored in the data structure
# (default is 2000)
header-value-depth: 2000
# Extract URLs and save in state data structure
extract-urls: yes
# Set to yes to compute the md5 of the mail body. You will then
# be able to journalize it.
body-md5: no
# Configure inspected-tracker for file_data keyword
inspected-tracker:
content-limit: 100000
content-inspect-min-size: 32768
content-inspect-window: 4096
Updated by Francis Trudeau about 7 years ago
Bryant Smith wrote:
Here is what I have for those values. Most of what is in my yaml file is default. Would there be something else affecting it? I've tried adjusting the libhtp values from kb to mb but that didn't seem to change anything.
[...]
In the yaml you sent me it doesn't look like you are loading rules. Uncomment this:
# - test.rules
On line 52.
If that alone doesn't work, add this to your rules file (it appears you are using /opt/suricata/etc/suricata/rules/test.rules):
alert tcp any any -> any any (msg:"FILE store all"; filestore; flowbits:noalert; sid:303; rev:1;)
If I turn off that filestore rule it doesn't fire locally. I am not sure if that's expected behavior.
Updated by Bryant Smith about 7 years ago
I am loading rule but I don't use the yaml file to load them. I use the -S option in a small script to run pcaps and different rule files. I have multiple rule files that I test with and this seemed to be the easiest way to run it.
/opt/suricata/bin/suricata -c /opt/suricata/etc/suricata/suricata.yaml -k none -r $PCAP -S $FILE --runmode=single
Updated by Bryant Smith almost 7 years ago
So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?
12/19/2017-13:01:34.001543 [**] [1:123456:1] MZ and CreateFont [**] [Classification: (null)] [Priority: 3] {TCP} 10.1.1.9:59724 -> 10.2.2.5:25
--------------------------------------------------------------------------
Date: 12/28/2017 -- 07:24:33. Sorted by: number of matches.
--------------------------------------------------------------------------
Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
-------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- --------------
1 123456 1 1 45674 10.09 1 1 45674 45674.00 45674.00 0.00
2 303 1 1 407015 89.91 166 0 51319 2451.90 0.00 2451.90
Updated by Francis Trudeau almost 7 years ago
That I do not know. I'll see what I can find out.
Bryant Smith wrote:
So I added your filestore signature to the rule file I was using and I got an alert from it. I'm not sure why this works though, any ideas?
[...]
Updated by Bryant Smith almost 7 years ago
I went in and made some changes to the yaml file with the following. If I turn on the file-store and enable force-filestore then the signature will work. This is probably the same as the siganture
- file-store:
enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: yes # force logging magic on all stored files
# force logging of checksums, available hash functions are md5,
# sha1 and sha256
#force-hash: [md5]
force-filestore: yes # force storing of all files
# override global stream-depth for sessions in which we want to
# perform file extraction. Set to 0 for unlimited.
#stream-depth: 0
#waldo: file.waldo # waldo file to store the file_id across runs
Updated by Andreas Herz almost 7 years ago
- Assignee set to Anonymous
- Target version set to TBD
Updated by Gabriel Somlo over 6 years ago
- File test-mail-attach.pcap test-mail-attach.pcap added
I'm experiencing the same issue (failure to trigger on smtp base64 "file_data" pattern), both using the Fedora 28 build of version 4.0.5, and using a recent 4.1.0 rc -- the github master as of last week, commit 0b5a2ab4).
Using the default suricata.yaml, with missing rule files (mentioned but not installed by default) commented out, and with a single "local.rules" file added to the list, the following rule:
alert tcp $EXTERNAL_NET any -> $HOME_NET 25 (msg:"VIRUS INBOUND bad file attachment"; flow:to_server,established; content:"content-disposition|3a| attachment|3b|"; nocase; content:".zip|22|"; nocase; within:128; file_data; content:".pdf.exe"; within:64; classtype:suspicious-attachment-detect; sid:3000101; rev:1;)
fails to trigger on the attached test-mail-attach.pcap file. When I try it using snort, the same signature does find the match.
Updated by Andreas Herz over 5 years ago
@Gabriel what does your suricata.yaml look like?
Updated by Gabriel Somlo over 5 years ago
- File etc_suricata_suricata.yaml etc_suricata_suricata.yaml added
- File etc_suricata_classification.config etc_suricata_classification.config added
- File var_lib_suricata_rules_suricata.rules var_lib_suricata_rules_suricata.rules added
Andreas Herz wrote:
@Gabriel what does your suricata.yaml look like?
@Andreas: I am now using fedora 30 with suricata-4.1.4-1.fc30.x86_64. I am uploading suricata.yaml and classification.config, as well as my (one-liner) suricata.rules file (containing the rule I referred to in comment #9. The changes I made to the default suricata.yaml are highlighted below:
--- suricata.yaml.orig 2019-08-19 15:50:30.794671681 -0400
+++ suricata.yaml 2019-08-19 15:50:41.561663720 -0400
@@ -10,18 +10,18 @@
##
vars:
# more specific is better for alert accuracy and performance
address-groups:
- HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
+ #HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
#HOME_NET: "[192.168.0.0/16]"
#HOME_NET: "[10.0.0.0/8]"
#HOME_NET: "[172.16.0.0/12]"
- #HOME_NET: "any"
+ HOME_NET: "any"
- EXTERNAL_NET: "!$HOME_NET"
- #EXTERNAL_NET: "any"
+ #EXTERNAL_NET: "!$HOME_NET"
+ EXTERNAL_NET: "any"
HTTP_SERVERS: "$HOME_NET"
SMTP_SERVERS: "$HOME_NET"
SQL_SERVERS: "$HOME_NET"
DNS_SERVERS: "$HOME_NET"
The changes I made to classification.config:
--- classification.config.orig 2019-08-19 15:58:35.992307453 -0400 +++ classification.config 2019-08-19 16:00:03.886240250 -0400 @@ -73,5 +73,8 @@ config classification: domain-c2,Domain Observed Used for C2 Detected,1 config classification: pup-activity,Possibly Unwanted Program Detected,2 config classification: credential-theft,Successful Credential Theft Detected,1 config classification: social-engineering,Possible Social Engineering Attempted,2 config classification: coin-mining,Crypto Currency Mining Activity Detected,2 + +# NetFor +config classification: suspicious-attachment-detect,Suspicious Attachment Detected,1
With this setup, running
suricata -r test-mail-attach.pcap
should generate a hit in /var/log/suricata/fast.log, but doesn't.
Updated by Andreas Herz over 5 years ago
I can reproduce it, even increasing the limits within libhtp settings didn't change. It triggers at least for the zip if I remove the part beginning with file_data. We might need to dig deeper into that why it's not matching.
Updated by Victor Julien over 5 years ago
Can someone create a suricata-verify test for it?
Updated by Andreas Herz over 5 years ago
Updated by Andreas Herz over 5 years ago
- Assignee changed from Community Ticket to Andreas Herz
Updated by Victor Julien over 5 years ago
- Status changed from New to Assigned
- Assignee changed from Andreas Herz to Victor Julien
- Target version changed from TBD to 5.0rc1
Updated by Victor Julien over 5 years ago
- Status changed from Assigned to Closed
Updated by Victor Julien over 5 years ago
- Copied to Bug #3175: File_data inspection depth while inspecting base64 decoded data (4.1.x) added
Updated by Gabriel Somlo over 5 years ago
- File suricata.rules suricata.rules added
@Victor, thanks for the fix, it does indeed change the behavior of suricata to generate the appropriate alert on that rule!
However, there's still a problem when that rule is accompanied by another rule (see new suricata.rules upload):
alert tcp $EXTERNAL_NET any -> $HOME_NET 25 (msg:"VIRUS INBOUND bad file attachment"; flow:to_server,established; content:"content-disposition|3a| attachment|3b|"; nocase; content:".zip|22|"; nocase; within:128; file_data; content:".pdf.exe"; within:64; sid:3000721; rev:1;) # alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ATTACK-RESPONSES directory listing"; flow:established; content:"Volume Serial Number"; sid:3001292; rev:1;)
The attached test-mail-attach.pcap file should generate alerts for both rules, but the first (file_data) rule you wrote the fix for won't trigger unless I comment out the second one, which is the only one generating an alert as long as both are active!
This suggests there's still something wonky going on. Please advise if, instead of reopening this bug, I should open a new bug for this issue, and/or open an issue on github!
Updated by Gabriel Somlo over 5 years ago
Opened new bug #3190 to keep things simple.
Updated by Victor Julien over 5 years ago
- Related to Bug #3190: file_data inspection inhibited by additional (non-file_data) content match rule added