Project

General

Profile

Actions
Copy link

Bug #2623

closed

Missing http.status information in eve.log according to tcp packets.

Added by dahan han about 6 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Victor Julien
Target version:
TBD
Affected Versions:
4.0.5
Effort:
Difficulty:
Label:

Description

http.status is not checked intermittently in eve.log with event_type of alert.

In the case where the status code is not left, there is a common point that the FIN packet is terminated without terminating the ACK packet from the client to the server at the end.
Also, if the status code is not left in eve.log, the http.length value becomes 0, and the http_content_type and http.status values are not output.

pcap-1) Case without http.status and http.length (event_type: alert)
1 0.000000 172.16.11.111 → 216.239.34.21 TCP 66 3823 → 80 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 0.064806 216.239.34.21 → 172.16.11.111 TCP 66 80 → 3823 [SYN, ACK] Seq=0 Ack=1 Win=60720 Len=0 MSS=1380 SACK_PERM=1 WS=256
3 0.065170 172.16.11.111 → 216.239.34.21 TCP 60 3823 → 80 [ACK] Seq=1 Ack=1 Win=66048 Len=0
4 0.077885 172.16.11.111 → 216.239.34.21 HTTP 104 GET /ip HTTP/1.1
5 0.142290 216.239.34.21 → 172.16.11.111 TCP 60 80 → 3823 [ACK] Seq=1 Ack=51 Win=60928 Len=0
6 0.305146 216.239.34.21 → 172.16.11.111 HTTP 339 HTTP/1.1 200 OK (text/html)
7 0.305678 172.16.11.111 → 216.239.34.21 TCP 60 3823 → 80 [FIN, ACK] Seq=51 Ack=286 Win=65792 Len=0
8 0.369924 216.239.34.21 → 172.16.11.111 TCP 60 80 → 3823 [FIN, ACK] Seq=286 Ack=52 Win=60928 Len=0
9 0.370227 172.16.11.111 → 216.239.34.21 TCP 60 3823 → 80 [ACK] Seq=52 Ack=287 Win=65792 Len=0
pcap-1 - eve.log)
"event_type": "alert",
"http": {
"hostname": "ipinfo.io",
"url": "\/ip",
"http_method": "GET",
"protocol": "HTTP\/1.1",
"length": 0
},

pcap-2) Case where http.status and http.length exist. (event_type: alert)
1 0.000000 172.20.11.11 → 216.239.38.21 TCP 62 49382 → 80 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 SACK_PERM=1
2 0.064720 216.239.38.21 → 172.20.11.11 TCP 62 80 → 49382 [SYN, ACK] Seq=0 Ack=1 Win=60720 Len=0 MSS=1380 SACK_PERM=1
3 0.065120 172.20.11.11 → 216.239.38.21 TCP 60 49382 → 80 [ACK] Seq=1 Ack=1 Win=64860 Len=0
4 0.066075 172.20.11.11 → 216.239.38.21 HTTP 104 GET /ip HTTP/1.1
5 0.130564 216.239.38.21 → 172.20.11.11 TCP 60 80 → 49382 [ACK] Seq=1 Ack=51 Win=60720 Len=0
6 0.289601 216.239.38.21 → 172.20.11.11 HTTP 340 HTTP/1.1 200 OK (text/html)
7 0.289868 172.20.11.11 → 216.239.38.21 TCP 60 49382 → 80 [ACK] Seq=51 Ack=287 Win=64574 Len=0
8 0.289872 172.20.11.11 → 216.239.38.21 TCP 60 49382 → 80 [FIN, ACK] Seq=51 Ack=287 Win=64574 Len=0
9 0.354268 216.239.38.21 → 172.20.11.11 TCP 60 80 → 49382 [FIN, ACK] Seq=287 Ack=52 Win=60720 Len=0
10 0.354615 172.20.11.11 → 216.239.38.21 TCP 60 49382 → 80 [ACK] Seq=52 Ack=288 Win=64574 Len=0
pcap-2 - eve.log)
"event_type": "alert",
"http": {
"hostname": "ipinfo.io",
"url": "\/ip",
"http_content_type": "text\/html",
"http_method": "GET",
"protocol": "HTTP\/1.1",
"status": 200,
"length": 16
},

However, if event_type is http, both pcap can check http.status and http.length.

"event_type": "http",
"http": {
"hostname": "ipinfo.io",
"url": "\/ip",
"http_content_type": "text\/html",
"http_method": "GET",
"protocol": "HTTP\/1.1",
"status": 200,
"length": 16
}

Test Environment)
suricata version: 4.0.5, 4.0.4
eve.log Setting of suricata.yaml
types:
- alert:
payload: yes
payload-buffer-size: 4kb
payload-printable: yes
http: yes
http-body-printable: yes

Files

Download all files
normal.pcap (1.09 KB) normal.pcap jaeik Cho, 10/05/2018 01:52 AM
abnormal.pcap (1.02 KB) abnormal.pcap jaeik Cho, 10/05/2018 01:52 AM
Actions
Copy link
 #1

Updated by dahan han about 6 years ago

Is there a solution to this problem?

Actions
Copy link
 #2

Updated by jaeik Cho about 6 years ago

dahan han wrote:

Is there a solution to this problem?

When looking at the packet, it appears that the ACK packet dropped due to port mirroring.
In this case, Can stream-tcp.c's StreamTcpPacketStateEstablished function be able to handle ACK and FIN?

static int StreamTcpPacketStateEstablished(ThreadVars *tv, Packet *p,
                        StreamTcpThread *stt, TcpSession *ssn, PacketQueue *pq)
{
    if (ssn == NULL)
        return -1;

    if (p->tcph->th_flags & TH_RST) {
    /*  ......snippet......    */
    } else if (p->tcph->th_flags & TH_ACK) {    //  <==  TH_ACK -> TH_ACK|TH_FIN
        /* Urgent pointer size can be more than the payload size, as it tells
         * the future coming data from the sender will be handled urgently
         * until data of size equal to urgent offset has been processed
         * (RFC 2147) */

        /* If the timestamp option is enabled for both the streams, then
         * validate the received packet timestamp value against the
         * stream->last_ts. If the timestamp is valid then process the
         * packet normally otherwise the drop the packet (RFC 1323) */
        if (ssn->flags & STREAMTCP_FLAG_TIMESTAMP) {
            if (!StreamTcpValidateTimestamp(ssn, p))
                return -1;
        }

    } else if (p->tcph->th_flags & TH_ACK) {
        /* Urgent pointer size can be more than the payload size, as it tells
         * the future coming data from the sender will be handled urgently
         * until data of size equal to urgent offset has been processed
         * (RFC 2147) */

        /* If the timestamp option is enabled for both the streams, then
         * validate the received packet timestamp value against the
         * stream->last_ts. If the timestamp is valid then process the
         * packet normally otherwise the drop the packet (RFC 1323) */
        if (ssn->flags & STREAMTCP_FLAG_TIMESTAMP) {
            if (!StreamTcpValidateTimestamp(ssn, p))
                return -1;
        }
    /*  ......snippet......    */
    return 0;
}

Actions
Copy link
 #3

Updated by Victor Julien about 6 years ago

Can you share a pcap to show the issue and make it easier to reproduce?

Actions
Copy link Download all files
 #4

Updated by jaeik Cho about 6 years ago

Victor Julien wrote:

Can you share a pcap to show the issue and make it easier to reproduce?

Hi, I share a pcap to show the issue. (normal.pcap and abnormal.pcap)
Both pcaps are packets that attempt to access ipinfo.io.
But, abnormal.pcap has droped an ACK packet since the http response.

$ sudo suricata -r normal.pcap -c /etc/suricata/suricata.yaml
$ cat /var/log/suricata/eve.json|jq

{
"timestamp": "2018-09-13T17:08:59.138683-0700",
"flow_id": 2027069879546799,
"pcap_cnt": 7,
"event_type": "alert",
"src_ip": "172.20.48.94",
"src_port": 49382,
"dest_ip": "216.239.38.21",
"dest_port": 80,
"proto": "TCP",
"tx_id": 0,
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 2020716,
"rev": 4,
"signature": "ET POLICY Possible External IP Lookup ipinfo.io",
"category": "Potential Corporate Privacy Violation",
"severity": 1,
"metadata": {
"updated_at": [
"2015_03_19"
],
"created_at": [
"2015_03_19"
]
}
},
"http": {
"hostname": "ipinfo.io",
"url": "/ip",
"http_content_type": "text/html",
"http_method": "GET",
"protocol": "HTTP/1.1",
"status": 200,
"length": 16
},
"app_proto": "http",
"flow": {
"pkts_toserver": 4,
"pkts_toclient": 3,
"bytes_toserver": 286,
"bytes_toclient": 462,
"start": "2018-09-13T17:08:58.848815-0700"
}
}

$ sudo suricata -r abnormal.pcap -c /etc/suricata/suricata.yaml
$ cat /var/log/suricata/eve.json|jq

{
"timestamp": "2018-09-13T19:36:01.114910-0700",
"flow_id": 113718361872656,
"pcap_cnt": 7,
"event_type": "alert",
"src_ip": "172.20.56.126",
"src_port": 3823,
"dest_ip": "216.239.34.21",
"dest_port": 80,
"proto": "TCP",
"tx_id": 0,
"alert": {
"action": "allowed",
"gid": 1,
"signature_id": 2020716,
"rev": 4,
"signature": "ET POLICY Possible External IP Lookup ipinfo.io",
"category": "Potential Corporate Privacy Violation",
"severity": 1,
"metadata": {
"updated_at": [
"2015_03_19"
],
"created_at": [
"2015_03_19"
]
}
},
"http": {
"hostname": "ipinfo.io",
"url": "/ip",
"http_method": "GET",
"protocol": "HTTP/1.1",
"length": 0
},
"app_proto": "http",
"flow": {
"pkts_toserver": 4,
"pkts_toclient": 3,
"bytes_toserver": 290,
"bytes_toclient": 465,
"start": "2018-09-13T19:36:00.809232-0700"
}
}

※ stream checksum-validation should be set to "no" in "suricata.yaml" to check normally.

Actions
Copy link
 #5

Updated by Victor Julien over 5 years ago

  • Assignee set to Victor Julien
Actions
Copy link
 #6

Updated by Andreas Herz over 5 years ago

  • Target version set to Support
Actions
Copy link
 #7

Updated by Andreas Herz over 5 years ago

  • Tracker changed from Support to Bug
  • Target version changed from Support to TBD
Actions
Copy link
 #8

Updated by Philippe Antoine over 3 years ago

  • Status changed from New to Closed

I do not reproduce it with current version
./src/suricata -l log -r /Users/catena/Downloads/abnormal.pcap -c suricata.yaml -k none
gives with
jq 'select(.event_type=="http")' log/eve.json

{
  "timestamp": "2018-09-14T04:36:01.179156+0200",
  "flow_id": 2031831428913424,
  "pcap_cnt": 8,
  "event_type": "http",
  "src_ip": "172.20.56.126",
  "src_port": 3823,
  "dest_ip": "216.239.34.21",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 0,
  "http": {
    "hostname": "ipinfo.io",
    "url": "/ip",
    "http_content_type": "text/html",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 16
  }
}

The key was not to forget -k none

Actions
Copy link

Also available in: Atom PDF