Suricata

Per previous conversation, Peter suggested that I open a new one for this issue.

Summary of the issue: While using a blacklist, containing hashes to be blocked using the filemd5 keyword. At times the payload, if enabled in base64 format is not seen as valid (base64 format).
If I manually try to decode it using: base64 -d (-D macos) tempfile, the following error is shown: "invalid input". This issue have not been seen with any other typs of alerts.

If I enable Suricata to drop the payload as ascii (payload-printable) the full payload is shown without errors for the same files.

Due to the nature of the blacklist it's mostly checksum for PE files, but during tests I have gotten the same issue with plain html files as well

OS: Ubuntu 16.04
Suricata 4.0.4 (PPA) in IPS mode

Attached is a pcap of the alert with the corresponding malicious file in the pcap (you can carve the file if you need to re-run the test, it's a live malware so don't run it =)) also a plain html file (part of the default page for Apache) testfile that generates the same error

Steps to re-create the issue:

• One Ubuntu VM with Suricata 4.0.4 (PPA)
• Install Apache, move the attached html file (testfile) to /var/www/html
• Enable eve logging with base64 payload set to enabled (payload: yes)
• Include the rule set files.rules in suricata.yaml
• Create a rule like this: drop http any any -> any any (msg:"Blacklisted File Blocked"; flow:established; filemd5:bl.md5; classtype:misc-activity; sid:666; rev:1;) in /etc/suricata/rules/files.rule
• Hash a file (for example the testfile attached (10f7309902d8b2c1d801440302baf76e), that you can upload to your monitored asset
• Include the file hash in the bl.md5 file
• Start Suricata with: suricata --simulate-ips -c /etc/suricata/suricata.yaml -i ens33
• Curl/Wget down the file <IP>/testfile
• Review the eve log and copy the base64 payload to a file
• Run base64 -d (or -D) <filename>