Suricata

When downloading a large file over http and using a configuration tuned for larger file extraction, the StreamingBuffer will exceed its maximum size of 1GB.

Warning: streaming-buffer: StreamingBuffer::GrowRegionToSize() tried to alloc 1074011072 bytes, exceeds limit of 1073741824 [GrowRegionToSize:util-streaming-buffer.c:592]

My test case was a 9GB Linux ISO, but I have created a truncated version of the pcap that stops just after 1GB of data: https://codemonkey.net/tmp/iso-truncated.pcap.xz

To reproduce, use a rule like the following:

alert http any any -> any any (msg:"ISO"; flow:established,to_client; content:"Operating system load error"; filestore; classtype:policy-violation; sid:9000000; rev:1;)

Then in suricata.yaml:
- Enable file-store.
- Comment out stream.reassembly.depth or set to 0 or some number over 1gb (like 1.1gb)

Then run Suricata (unfortunately -k none is required):

./src/suricata -k none -l ./log -c ./suricata.yaml -r ~/iso-truncated.pcap -S ./test.rules

Output:

Notice: suricata: This is Suricata version 7.0.0-rc2-dev (6154bab49f 2023-06-02) running in USER mode [LogVersion:suricata.c:1153]
Notice: threads: Threads created -> RX: 1 W: 20 FM: 1 FR: 1   Engine started. [TmThreadWaitOnThreadRunning:tm-threads.c:1886]
Warning: streaming-buffer: StreamingBuffer::GrowRegionToSize() tried to alloc 1074011072 bytes, exceeds limit of 1073741824 [GrowRegionToSize:util-streaming-buffer.c:592]
Error: pcap: error code -1 truncated dump file; tried to read 26130 captured bytes, only got 5024 for /home/jason/iso-truncated.pcap [PcapFileDispatch:source-pcap-file-helper.c:156]
Notice: suricata: Signal Received.  Stopping engine. [SuricataMainLoop:suricata.c:2827]
Notice: pcap: read 0 files, 389566 packets, 1140641880 bytes [ReceivePcapFileThreadExitStats:source-pcap-file.c:388]