Project

General

Profile

Actions

Feature #5774

open

Addressing Mixed Case in HTTP Headers Names and HTTP2

Added by Brandon Murphy almost 2 years ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

Today we have a large amount of rules which use content negations of specific header names similar to the following logic.

http.header_names; content:!"Accept-"; content:!"Content-Type";

However, we have been receiving some feedback regarding FPs due to a unique case where proxies are converting an HTTP/2 request from the client into HTTP/1 out to the destination server and maintaining all lowercase headers names, while adding some which are camelcased.

Consider the following HTTP request which was collected from an environment which exhibited this behavior and the ET Open rule which triggered on this traffic.

GET /catalog/123 HTTP/1.1
Host: foo.bar
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101" 
sec-ch-ua-mobile: ?0
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36
sec-ch-ua-platform: "macOS" 
accept: */*
sec-fetch-site: cross-site
sec-fetch-mode: no-cors
sec-fetch-dest: script
referer: https://example.com/
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
Cache-Control: max-stale=0
Connection: Keep-Alive
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Win32/Zemot URI Struct"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"/catalog/"; fast_pattern; pcre:"/\/catalog\/\d{3,}$/"; http.header; content:!"nap.edu|0d 0a|"; http.header_names; content:!"Accept-"; content:!"Referer|0d 0a|"; reference:md5,b8e0b97c8e9faa6e5daa8f0cac845516; classtype:trojan-activity; sid:2019458; rev:5; metadata:created_at 2014_10_17, updated_at 2020_10_07;)

Despite the content negations for the Accept- this rule triggers due to the all lowercase HTTP header name originally included in the HTTP/2 request between the client and the proxy.

One method of addressing this is to simply add nocase to all the negations of HTTP header names, though I'm not sure that is the best course of action.

I'm curious for thoughts/options that the engine might be able to offer to better address this problem.

Additionally, as HTTP/2 support grows and buffers are "overloaded" to allow backwards compatibility with HTTP/1 keywords, this issue will become more relevant.


Related issues 2 (1 open1 closed)

Related to Suricata - Feature #5775: http.headers - dynamic sticky buffersNewOISF DevActions
Related to Suricata - Feature #6290: support case insensitive testing of HTTP header name existenceClosedPhilippe AntoineActions
Actions #1

Updated by Brandon Murphy almost 2 years ago

While this specific example highlights content negations, the same issues exists where we are explicitly looking for header names with camelcase, which will then FN when "native" HTTP2 or if the conversion of HTTP2 to HTTP/1 passes along a lowercased header name.

I wonder if a transform to convert the buffer to lowecase might be able to help. Perhaps a buffer which applies the transform http.header_names.lowercase. I'm not sure which would be most efficient, probably the transform.

Though, either solution isn't much better than applying nocase to all the rules.

Actions #2

Updated by Victor Julien over 1 year ago

I think this is a hard one to address. We can't change the meaning of http.header_names for HTTP/1. I guess it's not too late to change the meaning for HTTP/2 though, as HTTP/2 support is experimental and thus subject to change in Suricata 6.

Can we reliably rename the names to CamelCase? Or will this lead to 80% of the names working ok, but the remaining 20% causing issues? If we normalize the HTTP/2 names we need an additional buffer that contains the original "raw" names as well I suppose.

Actions #3

Updated by Brandon Murphy over 1 year ago

I agree, this is hard to address.

The more I think about it, the more I think https://redmine.openinfosecfoundation.org/issues/5775 might be able to solve at least some of these issues.

If https://redmine.openinfosecfoundation.org/issues/5775 were implemented, it would allow for case of the dynamic header to be case-insensitive.

I think this then shifts this convo specific to http.header and http.header_names.

I wonder how snort3 is handling this.

The snort3 docs mention (https://github.com/snort3/snort3/blob/b74e347d374b1043ffb9f1fc5fdb9a895a72d36d/doc/user/http2_inspect.txt) The HTTP/2 inspector parses and strips the HTTP/2 protocol framing and outputs HTTP/1.1 messages, but I'm not sure if that includes the normalization of header names....and I don't know enough to read the code and find out. I'll do some testing.

Actions #4

Updated by Brandon Murphy over 1 year ago

Off the top of my head, I can only think of two solutions.

1) normalize the buffers header names... (i'd probably suggest lowercasing them all to "move towards HTTP/2" instead of sticking with HTTP/1 formats)
2) maintain an internal flag of which protocol generated the traffic and modify the behavior of rule elements (or which normalization is done) based on the protocol

Either of these solutions are very likely to have challenges and require rule updates. I think the important part is to decide on how it will be done (would be nice if snort3/suri handle it the same), get it documented, and maintain backwards compatibility (with the protocols, not necessarily the that old rules work without changes to HTTP/2 traffic)

Once HTTP/2 is "stable" within suricata, rule writers will have to go through and figure out how to update the rules to work with whatever features/limitations/normalization/new buffers, etc. But that can't happen until HTTP/2 is stable.

Actions #5

Updated by Brandon Murphy over 1 year ago

Brandon Murphy wrote in #note-3:

I wonder how snort3 is handling this.

... I'll do some testing.

Ok, testing completed. When looking for HTTP header names via content matches against the http_header keyword, when presented with HTTP/2 traffic, the headers names were in lowercase, as they appeared in the pcap. when presented with HTTP/1 traffic, the header names were CamelCased, as they appeared in the pcap.

In order to get a single rule to fire on both HTTP/1 and HTTP/2 traffic inspecting for an HTTP header name using the http_header keyword, nocase was required.

Actions #6

Updated by Brandon Murphy over 1 year ago

Another example I just found today - this is an emotet payload request/response from a onenote file.

I'm guessing the client <--> server/proxy was HTTP/1 while the server, acting as a proxy, made an HTTP/2 connection to the upstream HTTP server.

This actually caused an FN as the rule which was written using http.header for the content-transfer-encoding and content-disposition headers had Camel Casing.

GET /wp-admin/0ipWMQYggLOD8Waf/?053635&c=1 HTTP/1.1
Connection: Keep-Alive
Accept: */*
User-Agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
Host: gdcgroup.vn

HTTP/1.1 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=5, max=100
x-powered-by: PHP/7.4.14
set-cookie: 641d28d5e590c=1679632597; expires=Fri, 24-Mar-2023 04:37:37 GMT; Max-Age=60; path=/
cache-control: no-cache, must-revalidate
pragma: no-cache
last-modified: Fri, 24 Mar 2023 04:36:37 GMT
expires: Fri, 24 Mar 2023 04:36:37 GMT
content-type: application/octet-stream
content-disposition: attachment; filename="rmY1YU5e4iNMZt22aot5qXA.zip" 
content-transfer-encoding: binary
transfer-encoding: chunked
date: Fri, 24 Mar 2023 04:36:37 GMT
server: LiteSpeed

10000
PK........}.wV.}..;.....>!....cENck0bAOmz8LMoQDQN9C8Xls.dll..y|.E...?3.!..g...%A.....GB...3..L.X@QAP.q.XxFP9..D2<..............B@...r.B.. ..<z......y....L........|~~$.Ow.U]]]]]]]r.:.".B...uA.........$...m.......*........$k....Z|.}Y.n......;.g-...u..Y.....{...#....."...<;|..oF...u..#.wB.....~?{s9..7..w................=.......
...........Lq    ...%
Actions #7

Updated by Brandon Murphy over 1 year ago

I had to write a rule yesterday which allowed for both HTTP/1 and HTTP/2 style headers. I though a real life example of what had to be done might help. In this case, I needed only the header names to be case insensitive while allowing for exact case matching on the values.

While I can't provide the entire rule here, this snippet should be enough.

content:"content-description|3a 20|"; nocase; content:"File|20|Transfer|0d 0a|"; within:15; content:"content-type|3a 20|"; within:14; nocase; content:"application/octet-stream|0d 0a|"; within:26;

This "pattern" was repeated for each header and exact value, it was very annoying to write.

Actions #8

Updated by Philippe Antoine over 1 year ago

Jumping in, and trying to sum up :

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?
Wether you test these header names with their values in a big buffer, to their absence in the multi-buffer header_name keyword

Actions #9

Updated by Brandon Murphy over 1 year ago

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?

Yes, in general. My intention with this ticket was really to bring the issue to everyone's awareness and discuss solutions.

It'd be great if I can get some direction in how to handle the issue. Be that adjusting the rules or if the engine is capable of addressing it somehow. Right now I feel we're in a state of ambiguity in how to ensure coverage, regardless of the HTTP version in use.

So far, I'm under the impression that for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward.

Though this leaves http.header, http.header_names, and http.start needing a solution.

For those buffers, it would be nice to have a solution that will allow for case-insenstive matching of HTTP header names while still allowing for case sensitive header values (excluding http.header_names)

Actions #10

Updated by Victor Julien over 1 year ago

  • Related to Feature #5775: http.headers - dynamic sticky buffers added
Actions #11

Updated by Brandon Murphy 12 months ago

quick update to this comment post Suricon 2023.

Brandon Murphy wrote in #note-9:

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?

Yes, in general. My intention with this ticket was really to bring the issue to everyone's awareness and discuss solutions.

It'd be great if I can get some direction in how to handle the issue. Be that adjusting the rules or if the engine is capable of addressing it somehow. Right now I feel we're in a state of ambiguity in how to ensure coverage, regardless of the HTTP version in use.

So far, I'm under the impression that for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward.

I still believe for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward. This feature allows all the benefits of sticky buffers containing the value of the header that is not currently possible with http.request_header or http.response_header

Though this leaves http.header, http.header_names, and http.start needing a solution.

For those buffers, it would be nice to have a solution that will allow for case-insenstive matching of HTTP header names while still allowing for case sensitive header values (excluding http.header_names)

http.header_names can be addressed (with changes to content mathces) via to_lowercase transformation -- https://redmine.openinfosecfoundation.org/issues/6439
http.header, http.request_header, and http.response_header can be addressed via header_lowercase transformation -- https://redmine.openinfosecfoundation.org/issues/6290#change-30876

http.start doesn't exist for http/2, so I guess that's not really a concern anymore... (the lack of http.start for HTTP/2 probably needs documented in the http.start keyword)

Actions #12

Updated by Philippe Antoine 12 months ago

  • Related to Feature #6290: support case insensitive testing of HTTP header name existence added
Actions #13

Updated by Philippe Antoine 5 months ago

  • Tracker changed from Support to Feature
  • Target version set to TBD
Actions

Also available in: Atom PDF