Suricata

Note that I am opening this as a support ticket, but is probably a feature request. As best as I can tell, this behavior is not expected but is only present under improper network behavior so is likely a feature request to better handle abnormal traffic instead of a support issue. I wanted to bring it to the attention of the Suricata maintainers to get their input on the situation in case I am overlooking something and leave it up them for any action that may or may not result from this scenario.

Summary:
Under abnormal network conditions (e.g. where the SYN-ACK and SYN are seen by Suricata out of order) alerts are sometimes generated by traffic that is the opposite of that defined in rule.

Details:
We noticed a number of alerts in our environment where the traffic direction appeared to be reversed, and the source and destination IP addresses were reversed as well. For example SID 2828913 from the Emerging Threats Pro rule set detects Kovter activity. The rule header is "http $HOME_NET any -> $EXTERNAL_NET any" so a match should occur only on packets originating from the addresses in the HOME_NET variable with matching rule content. We observed an external host sending Kovter traffic to one of our hosts yet this rule was triggering. An example alert is as follows:

{
  "timestamp": "2019-09-25T14:16:44.037998-0500",
  "flow_id": 715291317194162,
  "event_type": "alert",
  "src_ip": "131.212.X.X",
  "src_port": 80,
  "dest_ip": "66.216.145.51",
  "dest_port": 33070,
  "proto": "TCP",
  "tx_id": 1,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2828913,
    "rev": 5,
    "signature": "ETPRO TROJAN WIN32/KOVTER.B Checkin 2 M3",
    "category": "A Network Trojan was Detected",
    "severity": 1,
    "metadata": {
      "updated_at": [
        "2018_02_16" 
      ],
      "performance_impact": [
        "Moderate" 
      ],
      "malware_family": [
        "Kovter" 
      ],
      "created_at": [
        "2017_12_15" 
      ],
      "signature_severity": [
        "Major" 
      ],
      "deployment": [
        "Perimeter" 
      ],
      "attack_target": [
        "Client_Endpoint" 
      ],
      "affected_product": [
        "Windows_XP_Vista_7_8_10_Server_32_64_Bit" 
      ],
      "former_category": [
        "TROJAN" 
      ]
    }
  },
  "http": {
    "hostname": "131.212.X.X",
    "url": "/",
    "http_user_agent": "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "length": 0
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 3,
    "pkts_toclient": 6,
    "bytes_toserver": 684,
    "bytes_toclient": 1007,
    "start": "2019-09-25T14:15:43.216498-0500" 
  },
  "payload_printable": "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nUser-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\r\nHost: 131.212.X.X\r\nContent-Length: 428\r\nCache-Control: no-cache\r\n\r\nSq1ckI+eZ1BNFIv60tuHkM3uIRR0DpByJRn7fEu9if/J9uQSyF89o1RXCp0DzJ/Fv81IpKLSqannQtCD2TB/7iML1UYpoKlwAr+oFnzmBxzfYS2N5oVBt7X+q6ddGs8MJqJjI0i/PdY8UCBeiEvv3XSmFZvosbDYXyXGVjiYrNURKVEYC7Kf2rqQda7tiSnEepvND78Ukslpgsz+9F7vgx79L/yZ1SDNhFmNuc8UsxHynEKWymnvZAglqk7exXrIFXNDmqc5if0ZB1JHJauUa4miiW7UirCNl4CARGExeEY9xjjwmLemSWzNXbdIJmruH1Cp5ZzOBWDU2lOLRNbJ/8KbC5ixIbPrAM3f7ouQ24AyYUyz66G4VoAdUAGlsAr5PIwPjIg3XkyLu8sNddmB2VLk/VOtFcD6ObLbxMK1Kp8=",
  "stream": 1,
  "packet": "RQAAKAAAAABABgYug9Qcw0LYkTMAUIEuCehEWy3hS5VQNwoA59IAAA==",
  "packet_info": {
    "linktype": 12
  }
}

Notice in the above alert that the source is 131.212.x.x, which we have in our HOME_NET variable, on port 80. Yet the rule is defined to match on "$HOME_NET any -> $EXTERNAL_NET any" for the packet that must contain this POST. I was able to confirm this traffic was in fact originating from the external source, which seemed likely anyway just based on the ports and the Host header of the PORT. I have seen other situations where I at first believed the source and destination were reversed but then after looking at the rule definition these were explainable with the direction defined.

I was able to get a packet capture on one of our Suricata sensors with the above traffic happening. Using this capture in pcap offline mode allowed me to reproduce the behavior, which required the midstream configuration to be enabled. In the capture there were a total of 149 flows that have content that match this rule if we ignore the direction. I confirmed this number by modifying the rule to remove the threshold and also cloned the rule then reversed the source and destination in the rule header. Though the traffic matching this Kovter rule always originated from the external host, there were 6 packets that triggered HOME_NET -> EXTERNAL_NET rules and 143 that triggered EXTERNAL_NET -> HOME_NET rules. I found that in all 6 cases triggering the HOME_NET -> EXTERNAL_NET rule that for these flows the SYN-ACK occurred before the SYN (due to an issue with our taps that we plan to look into fixing). For additional investigation I used Scapy to swap only the order of the SYN and SYN-ACK packets along with the packet times and this caused the HOME_NET -> EXTERNAL_NET rule to no longer fire. I repeated this with Scapy twice more with similar results. However, when using Scapy on flows with the correct ordering of SYN before the SYN-ACK to swap these so the SYN-ACK came first, it did not cause the HOME_NET -> EXTERNAL_NET rule to fire. It seems then there is something else involved that I could not find contributing to this behavior. I enabled debug to try to figure out what else there could be, but was unable to identify exactly what was happening.

I can provide packet captures and a configuration used to reproduce this situation but I would need to do so outside of attaching to this issue so that they are not publicly available. I also have debug logs if you are interested. I will attach to the ticket the suricata --build-info output on the VM I was using to reproduce this.

Steps to reproduce:
I was able to reproduce on 4.1.4 and 4.1.5.
1. Using a default config:
a. set HOME_NET as appropriate
b. disable vlan.use-for-tracking
c. enable stream.midstream
d. download the ETPRO rule sets to get rule with SID 2828913
e. (optional) remove the threshold from 2828913 and clone the rule to have the reversed direction of EXTERNAL_NET -> HOME_NET
2. Run packet capture I will provide (with SYN-ACK before SYN and some other strangeness) through Suricata using "suricata -r <capturefile.pcap>". Specifically I used the following with debug enabled:

SC_LOG_LEVEL=Debug suricata -c /etc/suricata/suricata_4_1_4.yaml -k none --pidfile suricata.pid  -l logs -r capturefile.pcap >> logs/debug.out

3. Check logs to confirm whether or not alerts were generated

Actual Results:
The rule is defined to match on traffic originating from HOME_NET to EXTERNAL_NET. Since the traffic is confirmed to have originated from EXTERNAL_NET and per https://suricata.readthedocs.io/en/suricata-4.1.5/rules/intro.html#direction, it is my understanding this should not trigger an alert.

Expected Results:
Although the traffic is abnormal, since there is a SYN-ACK and a SYN that follows from the same source and destination in the flow, Suricata could possibly use some heuristics to determine the correct traffic direction.