Suricata

Seems that this is related to the pcre match. Snort seems to be able to match on %20 as whitespace via \s we don't seem to do this.

Suspect it won't be ready before 1.0.1. Will include it if it turns out to be.

Pablo, can you pick this one up?

Hi, I've been researching a bit more on it, and I think we need to ping Ivan for this.

The uri ends with "%20=%20|" and the normalization is done correctly on other cases, but here (as you can see at the end of the log) printing the hexa data inspected at the uricontent matching function, show us that the buffer ends at migratemigrate instead of " = |".

I did a unittest to check for a normalized uricontent like /normalized%20uri, matching later with uricontent:"normalized uri", and it works. But it seems that on certain cases it's broken (like with the pcap).

request_uri_normalized: data len 35 (0x23)
00000000 2f 61 77 73 74 61 74 73 2e 70 6c 3f 2f 6d 69 67 |/awstats.pl?/mig|
00000010 72 61 74 65 6d 69 67 72 61 74 65 25 32 30 3d 25 |ratemigrate%20=%|
00000020 32 30 7c |20||

htp_connp_req_data: in state=REQ_PROTOCOL, progress=REQ_LINE
htp_connp_req_data: in state=REQ_HEADERS, progress=REQ_HEADERS

htp_connp_REQ_HEADERS: data len 22 (0x16)
00000000 48 6f 73 74 3a 20 77 77 77 2e 64 6f 6d 61 69 6e |Host: www.domain|
00000010 2e 74 6c 64 0d 0a |.tld..|

htp_connp_REQ_HEADERS: data len 123 (0x7b)
00000000 55 73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 |User-Agent: Mozi|
00000010 6c 6c 61 2f 35 2e 30 20 28 57 69 6e 64 6f 77 73 |lla/5.0 (Windows|
00000020 3b 20 55 3b 20 57 69 6e 64 6f 77 73 20 4e 54 20 |; U; Windows NT |
00000030 36 2e 31 3b 20 65 6e 2d 55 53 3b 20 72 76 3a 31 |6.1; en-US; rv:1|
00000040 2e 39 2e 31 2e 35 29 20 47 65 63 6b 6f 2f 32 30 |.9.1.5) Gecko/20|
00000050 30 39 31 31 30 32 20 46 69 72 65 66 6f 78 2f 33 |091102 Firefox/3|
00000060 2e 35 2e 35 20 28 2e 4e 45 54 20 43 4c 52 20 33 |.5.5 (.NET CLR 3|
00000070 2e 35 2e 33 30 37 32 39 29 0d 0a |.5.30729)..|

htp_connp_REQ_HEADERS: data len 17 (0x11)
00000000 4b 65 65 70 2d 41 6c 69 76 65 3a 20 33 30 30 0d |Keep-Alive: 300.|
00000010 0a |.|

htp_connp_REQ_HEADERS: data len 24 (0x18)
00000000 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 70 |Connection: keep|
00000010 2d 61 6c 69 76 65 0d 0a |-alive..|

htp_connp_REQ_HEADERS: data len 2 (0x2)
00000000 0d 0a |..|

htp_connp_req_data: in state=htp_connp_REQ_CONNECT_CHECK, progress=REQ_HEADERS
htp_connp_req_data: in state=REQ_BODY_DETERMINE, progress=REQ_HEADERS
htp_connp_req_data: in state=REQ_IDLE, progress=WAIT
htp_connp_req_data: returning STREAM_STATE_DATA

Uricontent:
0000 2F 61 77 73 74 61 74 73 2E 70 6C 3F /awstats .pl?
0000 2F 6D 69 67 72 61 74 65 /migrate

[2575] 2/9/2010 -- 20:49:17 - (suricata.c:1169) <Info> (main) -- all packets processed by threads, stopping engine
[2575] 2/9/2010 -- 20:49:17 - (suricata.c:1176) <Info> (main) -- time elapsed 0s
[527] 2/9/2010 -- 20:49:17 - (stream-tcp.c:2736) <Info> (StreamTcpExitPrintStats) -- (Decode & Stream) Packets 7
[5379] 2/9/2010 -- 20:49:17 - (alert-fastlog.c:303) <Info> (AlertFastLogExitPrintStats) -- (Outputs) Alerts 0

The cause is clear now: libhtp doesn't decode the query string part of the uri. The reason is that this would conflict with future parsing of individual query string parameters. Working on a solution with Ivan.

This issue is fixed in the current git master. Libhtp was updated to 0.2.5 for it.