Bug #1482
closedsuricata 2.1 beta4: StoreStateTxFileOnly crashes
Description
While testing with HTTP traffic with IXIA (10K concurrent HTTP connections) we've noticed that suricata crashes in StoreStateTxFileOnly. Suricata runs in af-packet inline mode on two interfaces. It works under load for some time and than gives:
suricata: detect-engine-state.c:387: StoreStateTxFileOnly: Assertion `!(1)' failed.
gdb shows the following:
Core was generated by `suricata -c /etc/suricata/suricata.yaml --af-packet'.
Program terminated with signal 6, Aborted.
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install suricata-2.1.b4-1.el6.x86_64
(gdb) bt
#0 0x00007f97ce7e0625 in raise () from /lib64/libc.so.6
#1 0x00007f97ce7e1e05 in abort () from /lib64/libc.so.6
#2 0x00007f97ce7d974e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f97ce7d9810 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f97d20aa90a in DeStateDetectStartDetection ()
#5 0x00007f97d2075edc in SigMatchSignatures ()
#6 0x00007f97d2076498 in Detect ()
#7 0x00007f97d2177868 in TmThreadsSlotVarRun ()
#8 0x00007f97d214e058 in AFPReadFromRing ()
#9 0x00007f97d214e5f1 in ReceiveAFPLoop ()
#10 0x00007f97d21780ce in TmThreadsSlotPktAcqLoop ()
#11 0x00007f97ced4d9d1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f97ce8968fd in clone () from /lib64/libc.so.6
Is it a bug or it's caused with inappropriate settings for the given load? I've attached the YAML-file (a bit stripped and shuffled, 'cause it was processed with yaml-cpp library)
Files
Updated by Alexander Gozman over 9 years ago
Also, there was a truckload of signatures - about 28k, using 'ac' algorithm.
Updated by Victor Julien over 9 years ago
This means that AppLayerParserSetTxDetectState failed. Can you check what error code it failed with? Most likely would be -ENOMEM.
Updated by Alexander Gozman over 9 years ago
If I understood correctly, the issue may be caused with htp application layer - no-one else seems to return -ENOMEM... Also, htp layer uses HTPMalloc, which may return NULL if memcap is insufficient. I'll check this out, but if it's the case, probably, there should be an error message that describes a possible reason and how it may be fixed. This assertion failure is not self-explanatory ;)
Updated by Alexander Gozman over 9 years ago
Also, I'm not sure if coredumping with SIGABRT is a good way to handle this error :)
Updated by Victor Julien over 9 years ago
Yeah the BUG_ON should certainly be removed before 2.1 goes stable. I added it to make bugs/issues in this new code stand out, but it's probably time to remove it.
Updated by Alexander Gozman over 9 years ago
- Status changed from New to Closed
Seems that it can be closed 'cause increasing http memcap solved the problem. Probably suricata should give a hint when memcap is exhausted.
Updated by Peter Manev over 9 years ago
Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -
that could be very helpful in verbose mode when doing troubleshooting.
Updated by Victor Julien over 9 years ago
This already exists, the http.memcap counter.
Updated by Peter Manev over 9 years ago
I meant it differently - the counter exists - but the warning that it is reached is what can be helpful in suricata.log for example. It could potentially be achieved through lua scripting as well - similar to your github surilua perf script - https://github.com/inliniac/surilua.
Updated by Alexander Gozman over 9 years ago
Peter Manev wrote:
Just to add a comment - could you please open a feature request for that -
a capability to use warning/hint in log format or similar when certain memcaps are reached -that could be very helpful in verbose mode when doing troubleshooting.
Added low priority feature #1489.
Updated by Victor Julien over 9 years ago
- Assignee set to Victor Julien
- Priority changed from High to Normal
- Target version set to 3.0RC1
I removed the BUG_ON.