Optimization #2803
closedtest phase should use less memory
Description
While suricata-update runs it's test phase, its memory use remains high. This seems unnecessary, as all the rule parsing & sorting is already done. The only result of the test is pass or fail, no complicated rule actions should be done anymore.
Updated by Jason Ish over 5 years ago
2 things come to mind to immediately solve this. Set some of the dict's that are storing all the state to None and see if they get garbage collected. Or break the big main() up into smaller functions, which is more likely to garbage collect stuff as they get reference counted to 0 on function return, plus main() could probably benefit some breaking it up a bit.
Updated by Victor Julien over 5 years ago
I should add that I only tested with python2. The way I observed this is simple: htop showed the mem use of suricata-update while the test command (a suricata -T variant) was running. It can probably be simulated by simply having a test script that sleeps for some time.
Updated by Jason Ish over 5 years ago
- Status changed from Assigned to Closed
- Target version changed from 1.1.0rc1 to 1.0.4
Fixed by setting some collection types to None before running the test. https://github.com/OISF/suricata-update/commit/5cea9cf4f29fca00ede5c0882f2f9356415f3aba
This is probably about the best we can do without some redesign of suricata-update internals.