Project

General

Profile

Actions
Copy link

Bug #4421

closed

flow manager: using too much CPU during idle (6.0.x backport)

Added by Jeff Lucovsky over 3 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Jeff Lucovsky
Target version:
6.0.7
Affected Versions:
Effort:
Difficulty:
Label:

Description

Less extreme variant of #4096: the new usleep based loops in the flow manager and recycler consume too much CPU time. In busy systems this has been show to be more efficient than the old pthread_condtimed logic, but in lower end systems the usleep approach has too much overhead.

The reason for getting rid of the pthread_condtimed logic was that it would actually wake up millions of times per second on busy systems, leading to lots of overhead.

Perhaps we can make the usleep value configurable. On lower end systems there is no need to wake up frequently. Or optionally bring back the old behavior. Overall I'm not a great fan these kinds of options.

Files

Download all files
5.0.7 pidstat in vm measuring suricata process (5 KB) 5.0.7 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:56 PM
5.0.7 pidstat on host measuring pid vm running suricata (5.37 KB) 5.0.7 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:56 PM
5.0.10 pidstat in vm measuring suricata process (5.18 KB) 5.0.10 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:56 PM
5.0.10 pidstat on host measuring pid vm running suricata (5.71 KB) 5.0.10 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:56 PM
6.0.0 pidstat in vm measuring suricata process (5 KB) 6.0.0 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:56 PM
6.0.0 pidstat on host measuring pid vm running suricata (5.03 KB) 6.0.0 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:56 PM
6.0.1 pidstat in vm measuring suricata process (4.99 KB) 6.0.1 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:56 PM
6.0.1 pidstat on host measuring pid vm running suricata (5.03 KB) 6.0.1 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:56 PM
6.0.2 pidstat in vm measuring suricata process (5 KB) 6.0.2 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:56 PM
6.0.2 pidstat on host measuring pid vm running suricata (6.08 KB) 6.0.2 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:56 PM
6.0.3 pidstat on host measuring pid vm running suricata (5.38 KB) 6.0.3 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:57 PM
6.0.3 pidstat in vm measuring suricata process (5 KB) 6.0.3 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:57 PM
6.0.4 pidstat in vm measuring suricata process (5 KB) 6.0.4 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:57 PM
6.0.4 pidstat on host measuring pid vm running suricata (5.38 KB) 6.0.4 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:57 PM
6.0.5 pidstat in vm measuring suricata process (5 KB) 6.0.5 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:57 PM
6.0.5 pidstat on host measuring pid vm running suricata (5.38 KB) 6.0.5 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:57 PM
6.0.6 pidstat in vm measuring suricata process (5 KB) 6.0.6 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:57 PM
6.0.6 pidstat on host measuring pid vm running suricata (5.03 KB) 6.0.6 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:57 PM
6.0.6 usleep 100 pidstat in vm measuring suricata process (5.01 KB) 6.0.6 usleep 100 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:57 PM
6.0.6 usleep 100 pidstat on host measuring pid vm running suricata (5.04 KB) 6.0.6 usleep 100 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:57 PM
6.0.6 usleep 50000 pidstat in vm measuring suricata process (5.02 KB) 6.0.6 usleep 50000 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:58 PM
6.0.6 usleep 50000 pidstat on host measuring pid vm running suricata (5.04 KB) 6.0.6 usleep 50000 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:58 PM
6.0.6 usleep 1000000 pidstat in vm measuring suricata process (5.01 KB) 6.0.6 usleep 1000000 pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:58 PM
6.0.6 usleep 1000000 pidstat on host measuring pid vm running suricata (5.04 KB) 6.0.6 usleep 1000000 pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:58 PM
6.0.7-dev (f40ad90ad 2022-09-19) pidstat in vm measuring suricata process (5.08 KB) 6.0.7-dev (f40ad90ad 2022-09-19) pidstat in vm measuring suricata process Michiel Janssens, 09/23/2022 04:58 PM
6.0.7-dev (f40ad90ad 2022-09-19) pidstat on host measuring pid vm running suricata (5.11 KB) 6.0.7-dev (f40ad90ad 2022-09-19) pidstat on host measuring pid vm running suricata Michiel Janssens, 09/23/2022 04:58 PM
Baseline pidstat in vm measuring all (7.58 KB) Baseline pidstat in vm measuring all Michiel Janssens, 09/23/2022 04:58 PM
Baseline pidstat on host measuring pid vm (5.02 KB) Baseline pidstat on host measuring pid vm Michiel Janssens, 09/23/2022 04:58 PM
Stress-ng pidstat in vm measuring stress-ng cyclic usleep process (1.08 KB) Stress-ng pidstat in vm measuring stress-ng cyclic usleep process Michiel Janssens, 09/23/2022 04:58 PM
Stress-ng pidstat in vm running stressng cyclic usleep measuring all (6.67 KB) Stress-ng pidstat in vm running stressng cyclic usleep measuring all Michiel Janssens, 09/23/2022 04:58 PM
Stress-ng pidstat on host measuring pid vm running stress-ng cyclic usleep (5.02 KB) Stress-ng pidstat on host measuring pid vm running stress-ng cyclic usleep Michiel Janssens, 09/23/2022 04:59 PM
Summary usleep tests.ods (17.9 KB) Summary usleep tests.ods Michiel Janssens, 09/23/2022 04:59 PM
Baseline top in freebsd vm measuring all (1.41 KB) Baseline top in freebsd vm measuring all Michiel Janssens, 10/02/2022 01:24 PM
6.0.6 pidstat on host measuring pid freebsd vm running suricata (5.03 KB) 6.0.6 pidstat on host measuring pid freebsd vm running suricata Michiel Janssens, 10/02/2022 01:24 PM
6.0.6 vmstat in freebsd vm measuring all (974 Bytes) 6.0.6 vmstat in freebsd vm measuring all Michiel Janssens, 10/02/2022 01:24 PM
6.0.6 top in freebsd vm measuring suricata pid (1.71 KB) 6.0.6 top in freebsd vm measuring suricata pid Michiel Janssens, 10/02/2022 01:24 PM
6.0.8 pidstat on host measuring pid freebsd vm running suricata (5.03 KB) 6.0.8 pidstat on host measuring pid freebsd vm running suricata Michiel Janssens, 10/02/2022 01:24 PM
Baseline pidstat on host measuring pid freebsd vm (5.03 KB) Baseline pidstat on host measuring pid freebsd vm Michiel Janssens, 10/02/2022 01:24 PM
Baseline vmstat in freebsd vm measuring all (978 Bytes) Baseline vmstat in freebsd vm measuring all Michiel Janssens, 10/02/2022 01:24 PM
6.0.8 vmstat in freebsd vm measuring all (978 Bytes) 6.0.8 vmstat in freebsd vm measuring all Michiel Janssens, 10/02/2022 01:24 PM
6.0.8 top in freebsd vm measuring suricata pid (1.75 KB) 6.0.8 top in freebsd vm measuring suricata pid Michiel Janssens, 10/02/2022 01:24 PM
6.0.8 pidstat on host measuring pid vm running suricata (5.03 KB) 6.0.8 pidstat on host measuring pid vm running suricata Michiel Janssens, 10/02/2022 01:25 PM
Summary usleep tests with freebsd and 6.0.8.ods (19.3 KB) Summary usleep tests with freebsd and 6.0.8.ods Michiel Janssens, 10/02/2022 01:25 PM
6.0.8 pidstat in vm measuring suricata process (5 KB) 6.0.8 pidstat in vm measuring suricata process Michiel Janssens, 10/02/2022 01:25 PM

Related issues 1 (0 open — 1 closed)

Copied from Suricata - Bug #4379: flow manager: using too much CPU during idleClosedVictor JulienActions
Actions
Copy link
 #1

Updated by Jeff Lucovsky over 3 years ago

  • Copied from Bug #4379: flow manager: using too much CPU during idle added
Actions
Copy link
 #2

Updated by Victor Julien over 3 years ago

  • Target version changed from 6.0.3 to 6.0.4
Actions
Copy link
 #3

Updated by Victor Julien about 3 years ago

  • Target version changed from 6.0.4 to 6.0.5
Actions
Copy link
 #4

Updated by Victor Julien over 2 years ago

  • Target version changed from 6.0.5 to 6.0.6
Actions
Copy link
 #5

Updated by Victor Julien over 2 years ago

  • Target version changed from 6.0.6 to 6.0.7
Actions
Copy link
 #6

Updated by Victor Julien over 2 years ago

  • Subject changed from flow manager: using too much CPU during idle to flow manager: using too much CPU during idle (6.0.x backport)
Actions
Copy link
 #7

Updated by Jeff Lucovsky over 2 years ago

  • Status changed from Assigned to In Progress
  • Assignee changed from Shivani Bhardwaj to Jeff Lucovsky
Actions
Copy link
 #8

Updated by Jeff Lucovsky over 2 years ago

  • Status changed from In Progress to In Review

Cherry-pick commit(s):
- e6ac2e4
- f271fb4

Actions
Copy link
 #10

Updated by Jeff Lucovsky over 2 years ago

  • Status changed from In Review to Closed
Actions
Copy link Download all files
 #11

Updated by Michiel Janssens over 2 years ago

Hi, I noticed your heads-up here and that it was merged to the 6.0.x branch, thanks for that.
I was following this for a while already in the forum and here on the issue list, so I thought now it's time to do effort and try to measure the changes for several suricata releases.
And it seems that my hunch that mainly the cost of context switching is one of the import ingredients for the high cpu load is on the right track.
I'll let you decide and interpret the results I found so far.
In a summary I put together most important results from the files I uploaded in this report, just to get a feeling for the differences. Not all values are totally correct as some are distributed on multiple threads.

My interest in suricata is mainly through opnsense, which I run virtualized in Proxmox VE.
I haven't tested suricata on bare metal.
Hardware specs for the system I tested on: older amd opteron 6380, 1 socket 1cpu, 16 core.
VM with Ubuntu 22.04.1 LTS and 8 cores configured in proxmox. This VM runs Suricata for all the tests. No other vm's running during testing.
I hope that the naming of the uploaded files speak for themselves.
I felt that pidstat gave me the most important results, so all the files have output from pidstat.
As an extra I also did a test with stress-ng to simulate context switching. It's not exactly comparable with the other tests, but you see the same sort of cpu increase.
Please ask if some of my comment or uplodaded files is not clear or correct.

Actions
Copy link
 #15

Updated by Michiel Janssens over 2 years ago

And to add:
All suricata installs were done from source, with make install-full.
Starting suricata with default yaml.
Measurement was done during idling and waiting a bit for the process to settle down, no traffic on suricata.
On thee Proxmox host cpufreq.default_governor performance was set. normally this system runs schedutil governor, but that would interfere witch the measurements.

If these results are correct, I think many people will be pleased when 6.0.7 will be released with this backport included, especially considering current energy costs.

Actions
Copy link
 #16

Updated by Jeff Lucovsky over 2 years ago

@Michiel Janssens Thanks for the additional data files and analysis.

Actions
Copy link

Also available in: Atom PDF