Suricata

TLDR; I think the docs are wrong. but distance:0; endswith; doesn't make a lot of sense and we'll get that cleaned up in the ruleset.

So, after a bit more review, there are some cases where distance/within/endswith makes perfect sense to combine.

Example 1¶

Consider these examples, taken straight from Suricata tests which includes both distance, within, and endswith.
https://github.com/OISF/suricata/blob/master/src/tests/detect-engine-content-inspection.c#L247-L271

TEST_RUN("xxxxxxxxxxxxxxxxxyYYYYYYYYYYYYYYYY", 34, "content:\"yYYYYYYYYYYYYYYYY\"; distance:9; within:29; endswith;", true, 1);

Attached (35_bytes.pcap) is a flowsynth generated pcap which contains the matching content, and fires on the rule provided.

converted into a normal rule it looks like this
alert tcp any any -> any any (msg:"Test"; content:"yYYYYYYYYYYYYYYYY"; distance:9; within:29; endswith; sid:1;)

In practice, the distance and within effectively limit how much of a payload can be present while ensuring the packet still "endswith" the desired content. In this case, if the content was greater than 38 (9+29), the content wouldn't match. As indicated by the lack of sid:1 alerting on the 39_bytes.pcap file.

Now, honestly , with the introduction of the bsize keyword, this same objective could be completed with keyword.

Example 2¶

The following rule demonstrates an example where the use of distance/within in combination with endswith is key to the detection logic, where bsize cannot be used.

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE HTTPCore CnC Task Response"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"/resp_"; content:".txt"; distance:32; within:4; endswith; reference:url,www.cyber.gov.au/sites/default/files/2020-06/ACSC-Advisory-2020-008-Copy-Paste-Compromises.pdf; classtype:command-and-control; sid:2030364; rev:1;)

Due to an unknown length of the uri, the rule is looking for /resp_ anywhere in the uri buffer. So the uri could be /foobar/resp_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt, and we'd want this rule to alert on this, ruling out the use of the bsize keyword. However, the detection logic indicates that there should be exactly 32 bytes between /resp_ and .txt and that .txt and be the end of a buffer.

The relationship established between /resp_ and .txt by using the distance keyword is critical to this rules inspection logic, and while I've not had coffee for awhile, I can't thinking of another way to do this without using PCRE.

distance:0; endswith;¶

There are many, many rules in the ET ruleset which contain distance:0; endswith; but when thinking about this logic, it's clear the distance:0 is repetitive, even when used in relation to another content match. The logic of "endswith" implies that all other content matches must be before it.

The following rule is a good example where endswith establishes the relationship with all other content matches making the use of distance:0 unnecessary.

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; fast_pattern; content:"|23|"; within:20; content:"|23|"; distance:0; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:2026724; rev:3;)

The only reason I could see if is there is some suricata logic which increases performance by using the distance:0;, to test this, I made two modified versions of 2026724, where I moved the fast_pattern to the |23| to ensure the rule would actually be evaluated, and one rule removed the distance:0; from the last content match.

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; content:"|23|"; fast_pattern; within:20; content:"|23|"; distance:0; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:1; rev:3;)

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC (modified)"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; content:"|23|"; fast_pattern; within:20; content:"|23|"; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:2; rev:3;)

However, in every pcap tested against, sid:2 was faster on avg ticks and total ticks, while the checks remained the same, including True Positive pcaps.

Summary¶

• The Suricata test combine the keywords
• There are use cases where the use of distance/within and endswith are critical to the rule's logic.
• distance:0; endswith; is slower and doesn't really do anything

After this research, I believe the docs are incorrect and the use of distance and/or within along with endswith is ok. I'm not sure about offset, I think bsize would take care of all those uses cases, but in my research, I didn't find a rule which used it in the ET ruleset.

Greetz¶

shoutout to isaac for helping out on this research