Suricata

Could keyword app-layer-event support prefilter ?

Found by QuadFuzz for reference

Looks like https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=62416 is another variant found by oss-fuzz,
No MQTT, but HTTP, still using app-layer-event. keyword in signatures and spending much time qsort in DetectRunTx

So, diagnostic is
- create and make live many transaction for one flow (there is no limit for HTTP1 per #5921)
- use multiple rules with app-layer-event keyword (may work with others) but there is no pre filtering beyond the app-layer protocol

So, for each packet, we run DetectRunTx which runs for each transaction
As there is no prefilter, we merge 'state' rules from the regular prefilter
Then, we merge stored state into results, which is a duplicate of the signatures with the flags set (pointer set integer 0)

Problems :
- no limit or too big for the number of transaction per live flow
- we use qsort instead of merging two sorted lists when we merge 'state' rules from the regular prefilter (when quicksort may be at its quadratic worst-case implementation as the list is about already sorted)
- we store a state with flags set to 0 for these app-layer events, leading to duplicate signatures`

@Victor Julien I wonder what you think about this

Summing up the subtasks :
- merge sorted lists instead of using qsort
- do not store state with flags set to 0
- do not run detection if nothing has been updated (need to investigate the QA diff for SMTP halved transactions)
- MQTT should only raise event UnintroducedMessage once per flow, and maybe not if it was picked up midstream
- MQTT should reuse an exiting transaction for this event instead of creating a new empty one just for this purpose
- limit the number of live HTTP1 transactions tracked in #5921

After the merge of https://github.com/OISF/suricata/pull/10160, I wonder if there is one more subticket to do :

Is there a way that app-layer-event keyword is evaluated only once per transaction ? And not 50 times if there are 50 rules for 50 different app-layer events...
I wonder especially in the simple case when there is no event. Could we have a prefilter-ish mechanism saying that none of these signatures will match on the transaction in its current state ?