Support #2981
closed"state": “TRUNCATED” for large files (may be caused by CheckGap function)
Description
Me and my team work on integrating threat intelligence hash feeds with Suricata 4.1.3 using rules “filemd5” option.
Our goal is to receive alerts when files, which md5s we consider as dangerous, passes through Suricata. To achieve this goal we does the following:
1. Create a rule:
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"TCP: FILE MD5 Found"; filemd5:md5_list.txt; filestore; sid:11111112; rev:1;)
2. Create a file with set of hashes
1276481102f218c981e0324180bafd9f
f18208f33ca9f847dd2e348117e3bc54
cac7b533ba7abfb8591bb6a3f7a95eab
3. Include the rule to suricata.yaml and run Suricata
While testing the detection files using MD5-sum ('filemd5' keyword in rules) I faced to not clear behavior of Suricata: It looks like Suricata can’t calculate the MD5-sum of files with size about 50 MB and more. In that case the string in event-json.log looks like:
{ "timestamp": "05\/14\/2019-17:22:50.384309",
"ipver": 4,
"srcip": "10.16.159.190",
"dstip": "10.65.67.147",
"protocol": 6,
"sp": 80,
"dp": 41430,
"http_uri":
"\/dev\/INC000010408298\/test_file50mb.img",
"http_host": "ponybuntu.avp.ru",
"http_referer": "<unknown>",
"http_user_agent": "Wget\/1.14 (linux-gnu)",
"filename": "\/dev\/INC000010408298\/test_file50mb.img",
"state": "TRUNCATED",
"stored": false,
"size": 1013331 }
At the same time, in case of a small files everything works well:
{"id": 1,
"timestamp": "05\/14\/2019-17:22:42.507823",
"ipver": 4,
"srcip": "10.16.159.190",
"dstip": "10.65.67.147",
"protocol": 6,
"sp": 80,
"dp": 41428,
"http_uri": "\/dev\/INC000010408298\/test_file10kb.img",
"http_host": "ponybuntu.avp.ru",
"http_referer": "<unknown>",
"http_user_agent": "Wget\/1.14 (linux-gnu)",
"filename": "\/dev\/INC000010408298\/test_file10kb.img",
"state": "CLOSED",
"md5": "1276481102f218c981e0324180bafd9f",
"stored": true,
"size": 10240 }
I tried to understand why Suricata marks large files as Truncated. I followed Self-help diagram https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams, but it isn’t help me. So, I turned on debug logging and I found that according to https://github.com/OISF/suricata/blob/master/src/stream-tcp-reassemble.c the while-loop in ReassembleUpdateAppLayer may breaks if CheckGap function returns “false”, and it seems to be so in case of Suricata finds a GAP.
In that case Suricata stops to reassemble the tcp stream and doesn’t calculate the MD5-sum. It is proofed by debug logs – after CheckGap function was called, file assembling stopped and its size will not be increased anymore. Nevertheless on a client side file loads fully and correct.
Could you please give me any suggestions how to avoid such behavior and force Suricata calculate MD5-sums in large files with GAPs? Or this behavior is by design and should not be fixed?
Links:
Pcap-file: https://box.kaspersky.com/f/a54978a4b2924b4eb2c9/?dl=1
Debug-log (nohup.out): https://box.kaspersky.com/f/12ef7ee147744ce592b7/?dl=1
suricata.yaml: https://box.kaspersky.com/f/07cb539222fb4e66b870/?dl=1
Thank you in advance!