Suricata

Hello,

We have recently upgraded our ids in production from suricata version 3.2.1 to version 4.0.4 via the package manager(ppa repository) on Ubunutu 14.04.5 LTS.

Since the upgrade, we have noticed in the http event logs weird unicode characters in the values of the http_method and http_user_agent fields and random string truncation.
We kept the suricata.yml from version 3.2.1 and did not make any change in http log setup.
This problem has also been noticed in our test ids that has a low load(system and bandwith) but was more rare due too low traffic(only a few events).

Problematic User Agent sample:

{"timestamp":"2018-06-04T05:40:49.313734+0000","flow_id":192722981970376,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x","src_port":56217,
"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":0,"http":{"hostname":"www.bing.com","url":"www.bing.com:443",
*"http_user_agent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 11) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/58.0.3029.110 Safari\/537.36 Edge\/16.16299"*,"pragma":"no-cache","http_method":"CONNECT","protocol":"HTTP\/1.0","length":8302}}
*=> the event is not problematci, it has a full ua string and the same flow id as the second event*

{"timestamp":"2018-06-04T05:40:49.313734+0000","flow_id":192722981970376,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x","src_port":56217,
"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":1,"http":{*"http_user_agent":"Mozi\u0017\u0003\u0003"*,
"protocol":"www.bing.com:443 HTTP\/1.0","length":0}}

*=> this seconde event has atruncated ua string with unicode characters and hte same flow id as the first event* 

{"timestamp":"2018-05-30T05:23:44.577489+0000","flow_id":1546874169271876,"in_iface":"bond0","event_type":"http","src_ip":"x.x.x.x",
"src_port":17561,"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":0,"http":{"url":"pool-adhese.com:443","http_user_agent":"Mozilla\/5.0\u0016\u0003\u0003" 
,"http_method":"CONNECT","protocol":"HTTP\/1.1","length":0}}

Problematic http_method sample

{"timestamp":"2018-06-03T08:02:10.003933+0000","flow_id":961959389296609,"event_type":"http","src_ip":"x.x.x.x",
"src_port":53722,"dest_ip":"x.x.x.x","dest_port":8080,"proto":"TCP","tx_id":1,"http":{"http_method":"\u0016\u0003\u0003\\0F\u0010\\0\\0BA\u0004H\u0002","length":0}}

Commandline: apt-get install --only-upgrade suricata
Install: libhtp2:amd64 (0.5.26-2ubuntu4, automatic)
Upgrade: suricata:amd64 (3.2.1-0ubuntu1, 4.0.4-2ubuntu4)

ii  libhtp1                             0.5.x.201707130636~ubuntu14.04.1    amd64        HTTP normalizer and parser library
ii  libhtp2                             1:0.5.26-2ubuntu4                   amd64        HTTP normalizer and parser library => installed with version 4.0.4
ii  suricata                            4.0.4-2ubuntu4                      amd64        Suricata open source multi-thread IDS/IPS/NSM system.

http eve log setup suricata.yml(Not changed)

- eve-log:
        enabled: yes
        filetype: regular #regular|syslog|unix_dgram|unix_stream
        filename: eve-http.json
        types:
          - http:
              extended: yes     # enable this for extended logging information
          # custom allows additional http fields to be included in eve-log
          # the example below adds three additional fields when uncommented
              #custom: [Accept-Encoding, Accept-Language, Authorization]
              custom: [accept, accept-charset, accept-encoding, accept-language,
              accept-datetime, authorization, cache-control, set-cookie, cookie, from,
              max-forwards, origin, pragma, proxy-authorization, proxy-connection, range, te, via,
              x-requested-with, dnt, x-forwarded-proto, x-requested-with, accept-range, age,
              allow, connection, content-encoding, content-language,
              content-length, content-location, content-md5, content-range,
              content-type, date, etags, last-modified, link, location,
              proxy-authenticate, referrer, refresh, retry-after, server,
              set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
              www-authenticate]

Could you help us to investigate that issue?

Thanks in advance,