Suricata

Yes, confirmed what they are seeing in their output. Will try to reproduce now.

I was able to isolate a reproduction with just rule 2008438. It turns out that the payload being logged was about 40k into the stream which aligns with the libhtp.default-config.response-body-minimal-inspect-size.

It turns out that Suricata is working as designed, but the output may not be ideal. The HTTP application-layer is saving up to 40kb of response data before running detection on it. Once 40kb has been buffered, detection is run, rule 2008438 triggers and the logger dumps payload data from the stream segments. The problem is that the TCP segments in memory are well past triggering traffic, and are somewhere around 40kb into the packet, so you get data much deeper into the stream logged than the triggering packet, and we do realize this is not ideal.

Note that this does not happen with the request buffers - while a similar approach is taken, the request buffers are much smaller, so when they trigger, the expected payload is still in memory.

Some tweaks can be done with configuration such as lowering response-body-minimal-inspect-size to 1500 which will run detection on the response after just 1500 bytes, but you may lose accuracy in the case where data being looked for passes 1500 byte boundaries, but looking into these values could be a possible work-around for now.

Note that this affects eve as well and is not specific to unified2.

Jason did you have a chance to see how to handle that with EVE? unified 2 isn't a target anymore so at least on half of the issue gone :)

Andreas Herz wrote:

Jason did you have a chance to see how to handle that with EVE? unified 2 isn't a target anymore so at least on half of the issue gone :)

With Eve we could have have buffers with different names, http_uri_buffer, http_method_buffer. Thats just off the top of my head and now sure if we'd really end up naming it those. But no further investigation has gone into this issue from the initial confirmation.