Suricata

While this specific example highlights content negations, the same issues exists where we are explicitly looking for header names with camelcase, which will then FN when "native" HTTP2 or if the conversion of HTTP2 to HTTP/1 passes along a lowercased header name.

I wonder if a transform to convert the buffer to lowecase might be able to help. Perhaps a buffer which applies the transform http.header_names.lowercase. I'm not sure which would be most efficient, probably the transform.

Though, either solution isn't much better than applying nocase to all the rules.

I think this is a hard one to address. We can't change the meaning of http.header_names for HTTP/1. I guess it's not too late to change the meaning for HTTP/2 though, as HTTP/2 support is experimental and thus subject to change in Suricata 6.

Can we reliably rename the names to CamelCase? Or will this lead to 80% of the names working ok, but the remaining 20% causing issues? If we normalize the HTTP/2 names we need an additional buffer that contains the original "raw" names as well I suppose.

I agree, this is hard to address.

The more I think about it, the more I think https://redmine.openinfosecfoundation.org/issues/5775 might be able to solve at least some of these issues.

If https://redmine.openinfosecfoundation.org/issues/5775 were implemented, it would allow for case of the dynamic header to be case-insensitive.

I think this then shifts this convo specific to http.header and http.header_names.

I wonder how snort3 is handling this.

The snort3 docs mention (https://github.com/snort3/snort3/blob/b74e347d374b1043ffb9f1fc5fdb9a895a72d36d/doc/user/http2_inspect.txt) The HTTP/2 inspector parses and strips the HTTP/2 protocol framing and outputs HTTP/1.1 messages, but I'm not sure if that includes the normalization of header names....and I don't know enough to read the code and find out. I'll do some testing.

Off the top of my head, I can only think of two solutions.

1) normalize the buffers header names... (i'd probably suggest lowercasing them all to "move towards HTTP/2" instead of sticking with HTTP/1 formats)
2) maintain an internal flag of which protocol generated the traffic and modify the behavior of rule elements (or which normalization is done) based on the protocol

Either of these solutions are very likely to have challenges and require rule updates. I think the important part is to decide on how it will be done (would be nice if snort3/suri handle it the same), get it documented, and maintain backwards compatibility (with the protocols, not necessarily the that old rules work without changes to HTTP/2 traffic)

Once HTTP/2 is "stable" within suricata, rule writers will have to go through and figure out how to update the rules to work with whatever features/limitations/normalization/new buffers, etc. But that can't happen until HTTP/2 is stable.

Brandon Murphy wrote in #note-3:

I wonder how snort3 is handling this.

... I'll do some testing.

Ok, testing completed. When looking for HTTP header names via content matches against the http_header keyword, when presented with HTTP/2 traffic, the headers names were in lowercase, as they appeared in the pcap. when presented with HTTP/1 traffic, the header names were CamelCased, as they appeared in the pcap.

In order to get a single rule to fire on both HTTP/1 and HTTP/2 traffic inspecting for an HTTP header name using the http_header keyword, nocase was required.

Another example I just found today - this is an emotet payload request/response from a onenote file.

I'm guessing the client <--> server/proxy was HTTP/1 while the server, acting as a proxy, made an HTTP/2 connection to the upstream HTTP server.

This actually caused an FN as the rule which was written using http.header for the content-transfer-encoding and content-disposition headers had Camel Casing.

GET /wp-admin/0ipWMQYggLOD8Waf/?053635&c=1 HTTP/1.1
Connection: Keep-Alive
Accept: */*
User-Agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
Host: gdcgroup.vn

HTTP/1.1 200 OK
Connection: Keep-Alive
Keep-Alive: timeout=5, max=100
x-powered-by: PHP/7.4.14
set-cookie: 641d28d5e590c=1679632597; expires=Fri, 24-Mar-2023 04:37:37 GMT; Max-Age=60; path=/
cache-control: no-cache, must-revalidate
pragma: no-cache
last-modified: Fri, 24 Mar 2023 04:36:37 GMT
expires: Fri, 24 Mar 2023 04:36:37 GMT
content-type: application/octet-stream
content-disposition: attachment; filename="rmY1YU5e4iNMZt22aot5qXA.zip" 
content-transfer-encoding: binary
transfer-encoding: chunked
date: Fri, 24 Mar 2023 04:36:37 GMT
server: LiteSpeed

10000
PK........}.wV.}..;.....>!....cENck0bAOmz8LMoQDQN9C8Xls.dll..y|.E...?3.!..g...%A.....GB...3..L.X@QAP.q.XxFP9..D2<..............B@...r.B.. ..<z......y....L........|~~$.Ow.U]]]]]]]r.:.".B...uA.........$...m.......*........$k....Z|.}Y.n......;.g-...u..Y.....{...#....."...<;|..oF...u..#.wB.....~?{s9..7..w................=.......
...........Lq    ...%

I had to write a rule yesterday which allowed for both HTTP/1 and HTTP/2 style headers. I though a real life example of what had to be done might help. In this case, I needed only the header names to be case insensitive while allowing for exact case matching on the values.

While I can't provide the entire rule here, this snippet should be enough.

content:"content-description|3a 20|"; nocase; content:"File|20|Transfer|0d 0a|"; within:15; content:"content-type|3a 20|"; within:14; nocase; content:"application/octet-stream|0d 0a|"; within:26;

This "pattern" was repeated for each header and exact value, it was very annoying to write.

Jumping in, and trying to sum up :

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?
Wether you test these header names with their values in a big buffer, to their absence in the multi-buffer header_name keyword

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?

Yes, in general. My intention with this ticket was really to bring the issue to everyone's awareness and discuss solutions.

It'd be great if I can get some direction in how to handle the issue. Be that adjusting the rules or if the engine is capable of addressing it somehow. Right now I feel we're in a state of ambiguity in how to ensure coverage, regardless of the HTTP version in use.

So far, I'm under the impression that for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward.

Though this leaves http.header, http.header_names, and http.start needing a solution.

For those buffers, it would be nice to have a solution that will allow for case-insenstive matching of HTTP header names while still allowing for case sensitive header values (excluding http.header_names)

quick update to this comment post Suricon 2023.

Brandon Murphy wrote in #note-9:

You want your rules to match both HTTP/1 and HTTP/2 while these have different capitalization for headers names, do I get it right ?

Yes, in general. My intention with this ticket was really to bring the issue to everyone's awareness and discuss solutions.

It'd be great if I can get some direction in how to handle the issue. Be that adjusting the rules or if the engine is capable of addressing it somehow. Right now I feel we're in a state of ambiguity in how to ensure coverage, regardless of the HTTP version in use.

So far, I'm under the impression that for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward.

I still believe for individual HTTP headers https://redmine.openinfosecfoundation.org/issues/5775 seems to be the path forward. This feature allows all the benefits of sticky buffers containing the value of the header that is not currently possible with http.request_header or http.response_header

Though this leaves http.header, http.header_names, and http.start needing a solution.

For those buffers, it would be nice to have a solution that will allow for case-insenstive matching of HTTP header names while still allowing for case sensitive header values (excluding http.header_names)

http.header_names can be addressed (with changes to content mathces) via to_lowercase transformation -- https://redmine.openinfosecfoundation.org/issues/6439
http.header, http.request_header, and http.response_header can be addressed via header_lowercase transformation -- https://redmine.openinfosecfoundation.org/issues/6290#change-30876

http.start doesn't exist for http/2, so I guess that's not really a concern anymore... (the lack of http.start for HTTP/2 probably needs documented in the http.start keyword)