Suricata

This bug is largely a copy of #6120. While 6120 fixed the non file_data case, the issue still occurs with file_data.

When downloading a large file over http and using a configuration tuned for larger file extraction, the StreamingBuffer will exceed its maximum size of 1GB when using content with file_data.

Warning: streaming-buffer: StreamingBuffer::GrowRegionToSize() tried to alloc 1073968568 bytes, exceeds limit of 1073741824 [GrowRegionToSize:util-streaming-buffer.c:710]

My test case was a 9GB Linux ISO, but I have created a truncated version of the pcap that stops just after 1GB of data: https://codemonkey.net/tmp/iso-truncated.pcap.xz

To reproduce, use a rule like the following:

alert http any any -> any any (msg:"ISO"; flow:established,to_client; file_data; content:"Operating system load error"; filestore; classtype:policy-violation; sid:9000000; rev:1;)

Then in suricata.yaml:
- Enable file-store.
- Comment out stream.reassembly.depth or set to 0 or some number over 1gb (like 1.1gb)

Then run Suricata (unfortunately -k none is required):

./src/suricata -k none -l ./log -c ./suricata.yaml -r ~/iso-truncated.pcap -S ./test.rules

Output:

Notice: suricata: This is Suricata version 7.0.0-rc3-dev (643e674cb2 2023-06-16) running in USER mode [LogVersion:suricata.c:1157]
Notice: threads: Threads created -> W: 1 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1888]
Warning: streaming-buffer: StreamingBuffer::GrowRegionToSize() tried to alloc 1073968568 bytes, exceeds limit of 1073741824 [GrowRegionToSize:util-streaming-buffer.c:710]
Error: pcap: error code -1 truncated dump file; tried to read 26130 captured bytes, only got 5024 for /home/jason/iso-truncated.pcap [PcapFileDispatch:source-pcap-file-helper.c:156]
Notice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2831]
Notice: pcap: read 0 files, 389566 packets, 1140641880 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:388]

Additionally, now the FilePruneFile is being called, #5868 is now occurring in git master with file store saving 0 byte files.