Feature #7036
openFeature #6805: cpu-affinity: enhance CPU affinity logic with per-interface NUMA preferences
DPDK NUMA setup: choose correct CPUs from worker-cpu-set
Description
My machine has even numbered cores on NUMA0 and odd numbered cores on NUMA1. I have defined two vdev interfaces, one on each socket, and defined worker-cpu-set as:cpu: [ 2,4,6,8,10,12,14,3,5,7,9,11,13,15 ]
The desired behaviour is that first 7 threads (2,4,6,8,10,12,14) use NIC on NUMA0 and the remaining 7 (3,5,7,9,11,13,15) use NIC on NUMA1.
Actual behaviour is that the cpu list is probably sorted internally and read as [ 2,3,4,5,6,7,8,9,10,11,12,13,14,15 ]. As a result, I receive a warning stating - dpdk: net_bonding0: NIC is on NUMA 0, 3 threads on different NUMA node(s) , which indicates that threads 2-8 are using NIC on NUMA0 and 9-15 are using NIC on NUMA1.
Updated by Adhirath Kabra 5 months ago
- Subject changed from NUMA: choose correct CPUs from worker-cpu-set to DPDK NUMA setup: choose correct CPUs from worker-cpu-set
- Assignee changed from OISF Dev to Lukas Sismis
Updated by Lukas Sismis 5 months ago
- Tracker changed from Bug to Feature
- Status changed from New to Assigned
- Target version changed from TBD to 8.0.0-beta1
- Parent task set to #6805
- Affected Versions deleted (
7.0.5)
Updated by Lukas Sismis 5 months ago
- Status changed from Assigned to In Progress
Updated by Victor Julien 5 months ago
- Related to Task #3318: Research: NUMA awareness added
Updated by Victor Julien 4 months ago
- Related to Bug #7137: "invalid cpu range" when trying to use CPU affinity added
Updated by Lukas Sismis 4 months ago
Here is the suggestion for the feature.
// threading.cpu-assignment:
// - legacy - assign as usual
// - auto - use hwloc to determine NUMA locality of the NIC and try to assign a core from this NUMA node.
// If it fails then use the other NUMA node.
// Using this approach e.g. on bonded devices/aliased and any other will not work
// Warn/Notify a user when device's NUMA node cannot be determined.
// Mention in the docs that NUMA locatity supports PCIe addresses and Kernel interfaces
// - manual - in workers CPU set either:
// - Specify in one line ([ "eth0@1,2,3,4,7-9", "eth1@10,11" ])
// - Specify threading in a list:
// - worker-cpu-set:
// - interface: eth0
// cpu: [ 1,2,3,4 ]
// mode: "exclusive"
// prio:
// high: [ 3 ]
// default: "medium"
Updated by Lukas Sismis 4 months ago ยท Edited
Current draft containing something like auto mode (not fully implemented yet):
https://github.com/OISF/suricata/compare/master...lukashino:suricata:feat/6805-numa-cpu-locality-v1
(Manual support not included)
Updated by Adhirath Kabra 4 months ago
Lukas Sismis wrote in #note-6:
Here is the suggestion for the feature.
[...]
Nice suggestion, especially the flexibility in resource allocation that manual mode would offer. For example as an IDS, monitored links can be assigned CPUs proportionate to the amount of average traffic they have.
Updated by Victor Julien 4 months ago
Lukas Sismis wrote in #note-7:
Current draft containing something like auto mode (not fully implemented yet):
https://github.com/OISF/suricata/compare/master...lukashino:suricata:feat/6805-numa-cpu-locality-v1(Manual support not included)
Please do this as a draft PR so we can discuss implementation things there.
Updated by Lukas Sismis 4 months ago
Just created:
https://github.com/OISF/suricata/pull/11521
Updated by Lukas Sismis about 2 months ago
New update is here - https://github.com/OISF/suricata/pull/11706
It has been polished and now I have added a manual setting as well.