Project

General

Profile

Actions

Bug #6678

closed

datasets: discard datasets that hit the memcap while loading correctly

Added by Andreas Herz 10 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Loading a dataset via a signature with memcap should be discarded properly instead of a partial load.

[51158] Config: datasets: dataset: httphosts-seen loading from '/usr/local/var/lib/suricata/data/foobar.set'
[51158] Config: datasets: dataset: httphosts-seen loaded 2168171 records
[51158] Error: detect-dataset: dataset too large for set memcap
[51158] Error: detect: error parsing signature "alert http $HOME_NET any -> any any (msg:"TEST"; flow:established,to_server; http.host; dataset:set,foobar,type string,state foobar.set,memcap 150mb,hashsize 1000000; sid:1337; rev:1;)" from file/usr/local/var/lib/suricata/rules/test.rules at line 3
[51158] Config: detect: No rules loaded from /usr/local/var/lib/suricata/rules/test.rules
[51158] Warning: detect: 1 rule files specified, but no rules were loaded!

The error would assume the rule is not loaded and the dataset is also not present, but it actually is in current versions (at least part of the dataset data that fit into the memcap limit):

>>> dataset-lookup foobar string Zm9vYmFyMTkwMDAwMC5kZQo=
Success:
"item found in set" 

We will provide a fix, the root cause is in codepath the data is loaded but not discarded in a correct way once the memcap check is hit.


Files

foobar.set.zst (3.03 MB) foobar.set.zst Andreas Herz, 01/15/2024 03:55 PM

Subtasks 1 (0 open1 closed)

Bug #6679: datasets: discard datasets that hit the memcap while loading correctly (7.0.x backport)ClosedAndreas HerzActions
Actions #1

Updated by Victor Julien 10 months ago

  • Target version changed from 7.0.3 to 8.0.0-beta1
  • Label Needs backport to 7.0 added
Actions #2

Updated by OISF Ticketbot 10 months ago

  • Subtask #6679 added
Actions #3

Updated by OISF Ticketbot 10 months ago

  • Label deleted (Needs backport to 7.0)
Actions #5

Updated by Andreas Herz 10 months ago

Actions #6

Updated by Philippe Antoine 10 months ago

  • Status changed from Assigned to In Review
Actions #7

Updated by Philippe Antoine 7 months ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF