Suricata

What if - if you try it with the sticky buffer "file.data" ?
Would the results be the same ?

file.data and file_data offer exactly the same functionality

Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -

cat test.rules 
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6034444; rev:1;)
alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file.data; strip_whitespace; content:"bricks-and-clicks"; sid:6032; rev:1;)

#alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)
#alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)

rm log/* ;  /opt/suritest/bin/suricata -S test.rules -k none -l log/ -r /home/pevma/Downloads/bodytexttest.txt-http-get.pcap ; jq 'select (.event_type == "alert" ) ' log/eve.json | jq .alert.signature_id
[779652] 7/4/2020 -- 09:47:08 - (suricata.c:1070) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (960c52d7f 2020-04-07) running in USER mode
[779652] 7/4/2020 -- 09:47:09 - (tm-threads.c:1887) <Notice> (TmThreadWaitOnThreadInit) -- all 9 packet processing threads, 4 management threads initialized, engine started.
[779652] 7/4/2020 -- 09:47:09 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[779653] 7/4/2020 -- 09:47:09 - (source-pcap-file.c:371) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes
3032
6032
3034444
6034444

Peter Manev wrote in #note-3:

Yes, was thinking if it would make a diff - however - either way it alerts on my local tests -

[...]

frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /var/lib/suricata/rules/local.rules 
alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace TCP test"; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3032; rev:1;)

alert http any any -> any any (msg:"NO strip_whitespace HTTP test"; flow:established; file_data; content:"bricks-and-clicks"; sid:3033333; rev:1;)

alert tcp any any -> any any (msg:"NO strip_whitespace TCP test"; file_data; content:"bricks-and-clicks"; sid:3031; rev:1;)

alert tcp any any -> any any (msg:"CANARY TEST"; flow:established; file_data; content:"bricks-and-clicks"; sid:3; rev:1;)
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ /opt/suricata/suricata-src/suricata-git/src/suricata -k none -l /tmp/log -r bodytexttest.txt-http-get.pcap -S /var/lib/suricata/rules/local.rules -c /etc/suricata/suricata.5.0.x.local.yaml 
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev (ed8f48b05 2020-04-06) running in USER mode
[7213] 7/4/2020 -- 09:12:53 - (log-pcap.c:901) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 0 files.
[7198] 7/4/2020 -- 09:12:53 - (tm-threads.c:1888) <Notice> (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started.
[7198] 7/4/2020 -- 09:12:53 - (suricata.c:2607) <Notice> (SuricataMainLoop) -- Signal Received.  Stopping engine.
[7212] 7/4/2020 -- 09:12:53 - (source-pcap-file.c:376) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 17 packets, 5022 bytes
frantrudeau@researchvm:~/testids/pcap/strip_whitespace_testing$ cat /tmp/log/local.eve.json | jq 'select (.event_type == "alert" )' | jq '.alert | "\(.signature),\(.signature_id)" '
"CANARY TEST,3" 
"NO strip_whitespace TCP test,3031" 
"NO strip_whitespace HTTP test,3033333" 

It doesn't hit here. Peter, hit me up on IRC/Jabber/etc. please.

Currently chasing it as it apparently alerts on a couple of different Buster installations but not others - not sure if ti is OS/pkg related or something else.

After Peter and I mucking with this we found that strip_whitespace rules work, if no other rule has file_data without strip_whitespace:

Works:

alert http any any -> any any (msg:"CANARY TEST"; file_data; strip_whitespace; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)

10/13/2008-07:55:36.182000  [**] [1:10001:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000  [**] [1:3034444:1] strip_whitespace HTTP test [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900
10/13/2008-07:55:36.182000  [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

Does not work:

alert http any any -> any any (msg:"CANARY TEST"; file_data; content:"commonly-accepted commonly-used"; sid:4294967294; rev:1;)

alert http any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:3034444; rev:1;)

alert tcp any any -> any any (msg:"strip_whitespace HTTP test"; flow:established; file_data; strip_whitespace; content:"bricks-and-clicks"; sid:10001; rev:1;)

10/13/2008-07:55:36.182000  [**] [1:4294967294:1] CANARY TEST [**] [Classification: (null)] [Priority: 3] {TCP} 173.37.145.84:80 -> 192.168.0.1:12900

https://github.com/OISF/suricata/pull/5075 (draft)

I would love to see a SMTP test case as well. The mechanics of how file data is inspected is very different between http and the other protocols.