Documentation #5030
openDocumentation bugs for endswith, distance, within
Description
Documentation says: "endswith cannot be mixed with offset, within or distance for the same pattern."
Looks untrue, dozens of ET rules do this, for example:
alert dns $HOME_NET any -> any any (msg:"ET MALWARE DNS Query for DNSpionage CnC Domain"; dns.query; content:".microsoftonedrive.org"; distance:0; nocase; fast_pattern; endswith; reference:url,blog.talosintelligence.com/2018/11/dnspionage-campaign-targets-middle-east.html; classtype:command-and-control; sid:2026680; rev:4; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2018_11_29, deployment Perimeter, former_category MALWARE, performance_impact Low, signature_severity Major, tag DNSpionage, tag DNS_tunneling, updated_at 2020_09_16;)
Should it also be specified that distance/within are perfectly valid even for the first content match in a rule, in which case I guess they are relative to the start of the buffer (ie.are noops)? Or does that have some other expected behaviour? I mean, is it just an oversight that these are in some rules, or does it do something?
Files
Updated by Brandon Murphy over 2 years ago
This looks to be a poorly converted rule from before the dns.query sticky buffer was introduced. I can take a look through the ET ruleset and correct these. If you happen to have a list of them, that'd be great!
I think in theory, not considering the "endswith", distance:0 on the first "buffered" content match (in this case dns.query), would indicate the content should be found somewhere after the start of the buffer, so in practice, it does nothing.
It would be nice to have Suricata throw an error/warning on these. IMO, if documentation says something cannot be done, then the engine should enforce it.
Updated by Brandon Murphy over 2 years ago
- File 39_bytes.pcap 39_bytes.pcap added
- File 35_bytes.pcap 35_bytes.pcap added
TLDR; I think the docs are wrong. but distance:0; endswith; doesn't make a lot of sense and we'll get that cleaned up in the ruleset.
So, after a bit more review, there are some cases where distance/within/endswith makes perfect sense to combine.
Example 1¶
Consider these examples, taken straight from Suricata tests which includes both distance, within, and endswith.
https://github.com/OISF/suricata/blob/master/src/tests/detect-engine-content-inspection.c#L247-L271
TEST_RUN("xxxxxxxxxxxxxxxxxyYYYYYYYYYYYYYYYY", 34, "content:\"yYYYYYYYYYYYYYYYY\"; distance:9; within:29; endswith;", true, 1);
Attached (35_bytes.pcap) is a flowsynth generated pcap which contains the matching content, and fires on the rule provided.
converted into a normal rule it looks like thisalert tcp any any -> any any (msg:"Test"; content:"yYYYYYYYYYYYYYYYY"; distance:9; within:29; endswith; sid:1;)
In practice, the distance and within effectively limit how much of a payload can be present while ensuring the packet still "endswith" the desired content. In this case, if the content was greater than 38 (9+29), the content wouldn't match. As indicated by the lack of sid:1 alerting on the 39_bytes.pcap file.
Now, honestly , with the introduction of the bsize keyword, this same objective could be completed with keyword.
Example 2¶
The following rule demonstrates an example where the use of distance/within in combination with endswith is key to the detection logic, where bsize cannot be used.
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE HTTPCore CnC Task Response"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"/resp_"; content:".txt"; distance:32; within:4; endswith; reference:url,www.cyber.gov.au/sites/default/files/2020-06/ACSC-Advisory-2020-008-Copy-Paste-Compromises.pdf; classtype:command-and-control; sid:2030364; rev:1;)
Due to an unknown length of the uri, the rule is looking for /resp_ anywhere in the uri buffer. So the uri could be /foobar/resp_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt, and we'd want this rule to alert on this, ruling out the use of the bsize keyword. However, the detection logic indicates that there should be exactly 32 bytes between /resp_ and .txt and that .txt and be the end of a buffer.
The relationship established between /resp_ and .txt by using the distance keyword is critical to this rules inspection logic, and while I've not had coffee for awhile, I can't thinking of another way to do this without using PCRE.
distance:0; endswith;¶
There are many, many rules in the ET ruleset which contain distance:0; endswith; but when thinking about this logic, it's clear the distance:0 is repetitive, even when used in relation to another content match. The logic of "endswith" implies that all other content matches must be before it.
The following rule is a good example where endswith establishes the relationship with all other content matches making the use of distance:0 unnecessary.
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; fast_pattern; content:"|23|"; within:20; content:"|23|"; distance:0; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:2026724; rev:3;)
The only reason I could see if is there is some suricata logic which increases performance by using the distance:0;, to test this, I made two modified versions of 2026724, where I moved the fast_pattern to the |23| to ensure the rule would actually be evaluated, and one rule removed the distance:0; from the last content match.
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; content:"|23|"; fast_pattern; within:20; content:"|23|"; distance:0; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:1; rev:3;) alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE RedControle Communicating with CnC (modified)"; flow:established,to_server; content:"SE_ND_CO_NN_EC|23|"; depth:15; content:"|23|"; fast_pattern; within:20; content:"|23|"; endswith; reference:url,threatvector.cylance.com/en_us/home/poking-the-bear-three-year-campaign-targets-russian-critical-infrastructure.html; reference:md5,855b937f668ecd90b8be004fd3c24717; classtype:command-and-control; sid:2; rev:3;)
However, in every pcap tested against, sid:2 was faster on avg ticks and total ticks, while the checks remained the same, including True Positive pcaps.
Summary¶
- The Suricata test combine the keywords
- There are use cases where the use of distance/within and endswith are critical to the rule's logic.
distance:0; endswith;is slower and doesn't really do anything
After this research, I believe the docs are incorrect and the use of distance and/or within along with endswith is ok. I'm not sure about offset, I think bsize would take care of all those uses cases, but in my research, I didn't find a rule which used it in the ET ruleset.
Greetz¶
shoutout to isaac for helping out on this research
Updated by Gianni Tedesco over 2 years ago
Here are the locations and sids for all such rules, hope it helps!
rules/emerging-malware.rules:12697 2027222
rules/emerging-malware.rules:12699 2027223
rules/emerging-phishing.rules:2763 2027240
rules/emerging-phishing.rules:2779 2027241
rules/emerging-phishing.rules:2765 2027242
rules/emerging-phishing.rules:2767 2027243
rules/emerging-phishing.rules:2769 2027244
rules/emerging-phishing.rules:2781 2027245
rules/emerging-phishing.rules:2771 2027246
rules/emerging-phishing.rules:2773 2027247
rules/emerging-phishing.rules:2775 2027248
rules/emerging-phishing.rules:3657 2027249
rules/emerging-phishing.rules:2777 2027275
rules/emerging-malware.rules:17231 2031341
rules/emerging-malware.rules:12257 2026680
rules/emerging-malware.rules:12297 2026704
rules/emerging-malware.rules:12299 2026705
rules/emerging-malware.rules:12301 2026706
rules/emerging-malware.rules:12303 2026707
rules/emerging-malware.rules:12305 2026708
rules/emerging-malware.rules:12307 2026709
rules/emerging-malware.rules:12309 2026710
rules/emerging-malware.rules:12311 2026711
rules/emerging-malware.rules:12313 2026712
rules/emerging-malware.rules:12315 2026713
rules/emerging-malware.rules:12317 2026714
rules/emerging-malware.rules:12319 2026715
rules/emerging-malware.rules:12321 2026716
rules/emerging-malware.rules:11583 2028634
rules/emerging-malware.rules:11585 2028635
rules/emerging-malware.rules:11647 2030876
rules/emerging-malware.rules:11649 2030877
rules/emerging-malware.rules:14275 2023025
rules/emerging-malware.rules:14277 2023026
rules/emerging-malware.rules:14279 2023027
rules/emerging-malware.rules:17227 2031324
rules/emerging-malware.rules:17291 2031359
Updated by Gianni Tedesco over 2 years ago
"The relationship established between /resp_ and .txt by using the distance keyword is critical to this rules inspection logic, and while I've not had coffee for awhile, I can't thinking of another way to do this without using PCRE."
Yes, fully agree. Well, if you're using hyperscan for prefilter, technically hyperscan can actually do these, because it is possible to convert the sequence into a single regexp.
If suricata were to define hyperscan as being always available then a lot more could be done directly in the prefilter (eg. dotprefix, startswith, endswith, depth, offset, and various relative-matching idioms of the kind you mention like "^foo.{10-12}bar$").
Not sure how much you'd want to push in to the prefilter in that case in general but it could help in cases where the only fixed pattern available is very small and not so great. It could help.
I realise hyperscan isn't available in all installs, however :(
Updated by Philippe Antoine over 1 year ago
- Assignee set to Community Ticket
- Target version set to TBD
Updated by Philippe Antoine about 2 months ago
- Status changed from In Progress to New
Updated by Juliana Fajardini Reichow about 1 month ago
SV tests using the pcaps and rule created to showcase this: https://github.com/OISF/suricata-verify/pull/2060
If that gets approved, it's one step closer to fixing this :P
Updated by Juliana Fajardini Reichow 16 days ago
PR got merged, and has some good discussion on it, for when we/someone takes upon them to update the docs part.