Project

General

Profile

Actions
Copy link

Task #570

open

tracking: memory fragmentation

Added by Victor Julien about 12 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assignee:
OISF Dev
Target version:
TBD
Effort:
Difficulty:
Label:

Description

During start up (mostly detection engine init) we alloc and free enormous amounts of small memory blocks. This leads to memory fragmentation.

This task is a place holder for tracking that. Addressing all/most of this is going to be a long term effort.

Files

fragmentation-leak.png (150 KB) fragmentation-leak.png Gianni Tedesco, 05/24/2021 11:26 PM

Subtasks 3 (1 open — 2 closed)

Bug #492: memory not released after live rule swapClosedAnoop Saldanha06/30/2012Actions
Optimization #564: pattern id: rewrite pattern id assignmentClosedAnoop Saldanha09/28/2012Actions
Optimization #573: reduce allocs in signature parsingNewOISF DevActions
Actions
Copy link
 #1

Updated by Victor Julien about 11 years ago

  • Target version set to TBD
Actions
Copy link
 #2

Updated by Victor Julien about 5 years ago

  • Tracker changed from Optimization to Task
  • Subject changed from memory fragmentation to tracking: memory fragmentation
Actions
Copy link
 #3

Updated by Gianni Tedesco over 3 years ago

Actually the main culprit of heap fragmentation is reassembly. Keep doing increasing realloc() as we do for tcp stream reassembly is worst-case behaviour, and it can cause several gigabytes of excess heap to be used up. The only way to get it back is to attach gdb and call malloc_trim(0).

Reassembly buffers need to be a chain of fixed size blocks to avoid this situation. You can copy the blocks out to the final (per-thread) buffer at the end, just before injecting the packet for inspection.

Actions
Copy link
 #4

Updated by Gianni Tedesco over 3 years ago

Or at least, appears to be around reassembly and state for protocols at least. Will investigate further.

Actions
Copy link

Also available in: Atom PDF