Project

General

Profile

Actions

Bug #6678

closed

datasets: discard datasets that hit the memcap while loading correctly

Added by Andreas Herz 10 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Loading a dataset via a signature with memcap should be discarded properly instead of a partial load.

[51158] Config: datasets: dataset: httphosts-seen loading from '/usr/local/var/lib/suricata/data/foobar.set'
[51158] Config: datasets: dataset: httphosts-seen loaded 2168171 records
[51158] Error: detect-dataset: dataset too large for set memcap
[51158] Error: detect: error parsing signature "alert http $HOME_NET any -> any any (msg:"TEST"; flow:established,to_server; http.host; dataset:set,foobar,type string,state foobar.set,memcap 150mb,hashsize 1000000; sid:1337; rev:1;)" from file/usr/local/var/lib/suricata/rules/test.rules at line 3
[51158] Config: detect: No rules loaded from /usr/local/var/lib/suricata/rules/test.rules
[51158] Warning: detect: 1 rule files specified, but no rules were loaded!

The error would assume the rule is not loaded and the dataset is also not present, but it actually is in current versions (at least part of the dataset data that fit into the memcap limit):

>>> dataset-lookup foobar string Zm9vYmFyMTkwMDAwMC5kZQo=
Success:
"item found in set" 

We will provide a fix, the root cause is in codepath the data is loaded but not discarded in a correct way once the memcap check is hit.


Files

foobar.set.zst (3.03 MB) foobar.set.zst Andreas Herz, 01/15/2024 03:55 PM

Subtasks 1 (0 open1 closed)

Bug #6679: datasets: discard datasets that hit the memcap while loading correctly (7.0.x backport)ClosedAndreas HerzActions
Actions

Also available in: Atom PDF