Suricata

Thank you Andy for the bug report!
Could you please share some details how to reproduce this with a pcap it is possible ?

Unfortunately running against live traffic so not sure if I'll be able to find a pcap or not. I'll see what I can find.

Not sure if this helps but

index is -1 for detect-engine-hcbd.c:203

    StreamingBufferGetDataAtOffset(htud->request_body.sb,
            &det_ctx->hcbd[index].buffer, &det_ctx->hcbd[index].buffer_len,
            offset);

here is other gdb stuff that might help

(gdb) p index
$12 = -1
(gdb) p *det_ctx
$13 = {tenant_id = 0, ticker = 1139548, tv = 0xb9f09a0, non_pf_id_array = 0x621a8630, non_pf_id_cnt = 3, mt_det_ctxs_cnt = 0, mt_det_ctxs = 0x0, mt_det_ctxs_hash = 0x0, tenant_array = 0x0,
  tenant_array_size = 0, TenantGetId = 0x0, raw_stream_progress = 17889, buffer_offset = 0, pcre_match_start_offset = 0, filestore_cnt = 0, hcbd = 0x9de88f90, hcbd_start_tx_id = 6,
  hcbd_buffers_size = 50, hcbd_buffers_list_len = 0, counter_alerts = 172, inspect_list = 0, inspect = {buffers = 0xeb7fb4c0, buffers_size = 67, to_clear_idx = 1,
    to_clear_queue = 0x94cf29b0}, multi_inspect = {buffers = 0xa0175970, buffers_size = 67, to_clear_idx = 0, to_clear_queue = 0x9647c8c0}, discontinue_matching = 1, flags = 0,
  tx_id_set = 0, tx_id = 0, p = 0x0, so_far_used_by_detect_sc_atomic__ = 1, inspection_recursion_counter = 1, match_array = 0xeb6cbed0, match_array_len = 31062, match_array_cnt = 8,
  tx_candidates = 0xeb7089c0, tx_candidates_size = 31062, non_pf_store_ptr = 0xa46b8990, non_pf_store_cnt = 5, mtc = {ctx = 0xb968de0, memory_cnt = 1, memory_size = 16}, mtcu = {
    ctx = 0xbc9f6e0, memory_cnt = 1, memory_size = 16}, mtcs = {ctx = 0x8a30dd0, memory_cnt = 1, memory_size = 16}, pmq = {rule_id_array = 0x14eabc170, rule_id_array_cnt = 2,
    rule_id_array_size = 3590}, spm_thread_ctx = 0x62111b20, io_ctx = {sig_match_array = 0xeb6caf70 "", sig_match_size = 3860}, bj_values = 0x621a86c0, replist = 0x0, varlist = 0x0,
  filestore = {{file_id = 0, tx_id = 0} <repeats 15 times>}, de_ctx = 0xb35c3f0, keyword_ctxs_array = 0x0, keyword_ctxs_size = 0, global_keyword_ctxs_size = 3,
  global_keyword_ctxs_array = 0x5770b2e0, base64_decoded = 0x0, base64_decoded_len = 0, base64_decoded_len_max = 0, decoder_events = 0x0, events = 0}
(gdb) p *htud
$14 = {detect_flags_ts = 0, detect_flags_tc = 0, request_body_init = 1 '\001', response_body_init = 1 '\001', request_has_trailers = 0 '\000', response_has_trailers = 0 '\000', logged = 0,
  request_body = {first = 0x14eadd110, last = 0x14eadd110, sb = 0x5b142d40, content_len_so_far = 2659, body_parsed = 0, body_inspected = 0}, response_body = {first = 0x1bde7d70,
    last = 0x1bde7d70, sb = 0x1bde7d00, content_len_so_far = 146, body_parsed = 146, body_inspected = 0}, request_uri_normalized = 0x9919aac0,
  request_headers_raw = 0x442162e0 "REDACTED"...,
  response_headers_raw = 0xb24e000 "Content-Type: application/json\r\nCache-Control: private, no-cache, no-store, must-revalidate, max-age=0\r\nP3P: policyref=\"REDACTED", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi"..., request_headers_raw_len = 1168, response_headers_raw_len = 401, decoder_events = 0x0, boundary = 0x0, boundary_len = 0 '\000',
  tsflags = 0 '\000', tcflags = 0 '\000', request_body_type = 2 '\002', de_state = 0x0}

Thanks Andy. The gdb output in note number #3, is that from the same crash as the bt above? If not, could you produce both outputs for a single crash?

I'm getting lots of different crashes. #3 was from valgrind break on error of the valgrind stuff in #1. I worry that memory is already corrupt for the bt of #1. (Otherwise why would a realloc fail?) Anyway here is another crash with some vars printed:

(gdb) bt
#0  0x00007f3ba486b207 in raise () from /lib64/libc.so.6
#1  0x00007f3ba486c8f8 in abort () from /lib64/libc.so.6
#2  0x00007f3ba48add27 in __libc_message () from /lib64/libc.so.6
#3  0x00007f3ba48b45d4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f3ba48ba9e9 in realloc () from /lib64/libc.so.6
#5  0x00000000004b3b80 in HCBDCreateSpace (det_ctx=0x7f3b69be8500, size=<optimized out>) at detect-engine-hcbd.c:80
#6  0x00000000004b3ea7 in DetectEngineHCBDGetBufferForTX (tx=0x7f3a88f26940, tx_id=158, det_ctx=det_ctx@entry=0x7f3b69be8500, flags=flags@entry=132 '\204',
    buffer_len=buffer_len@entry=0x7f3b7282c738, stream_start_offset=stream_start_offset@entry=0x7f3b7282c73c, htp_state=<optimized out>, f=<optimized out>, de_ctx=0x0)
    at detect-engine-hcbd.c:140
#7  0x00000000004b40a2 in PrefilterTxHttpRequestBody (det_ctx=0x7f3b69be8500, pectx=0x7f3b88c4c0e0, p=<optimized out>, f=<optimized out>, txv=<optimized out>, idx=<optimized out>,
    flags=132 '\204') at detect-engine-hcbd.c:241
#8  0x00000000004c32f1 in DetectRunPrefilterTx (det_ctx=det_ctx@entry=0x7f3b69be8500, sgh=sgh@entry=0x7f3b88c4f530, p=p@entry=0x7f3b69afed50, ipproto=ipproto@entry=6 '\006',
    flow_flags=flow_flags@entry=132 '\204', alproto=alproto@entry=1, alstate=alstate@entry=0x7f3a88f6c1b0, tx=tx@entry=0x7f3b7282c960) at detect-engine-prefilter.c:117
#9  0x000000000048a67a in DetectRunTx (scratch=0x7f3b7282c920, f=0x7f3aa1aa42b0, p=0x8, det_ctx=0x7f3b69be8500, de_ctx=0x33cf680, tv=0x7f3b94f1d870) at detect.c:1398
#10 DetectRun (th_v=th_v@entry=0x7f3b94f1d870, de_ctx=0x33cf680, det_ctx=0x7f3b69be8500, p=p@entry=0x7f3b69afed50) at detect.c:141
#11 0x000000000048b843 in DetectRun (p=0x7f3b69afed50, det_ctx=<optimized out>, de_ctx=<optimized out>, th_v=0x7f3b94f1d870) at detect.c:1641
#12 DetectNoFlow (p=<optimized out>, det_ctx=<optimized out>, de_ctx=<optimized out>, tv=<optimized out>) at detect.c:1679
#13 Detect (tv=tv@entry=0x7f3b94f1d870, p=p@entry=0x7f3b69afed50, data=data@entry=0x7f3b69be8500, pq=pq@entry=0x0, postpq=postpq@entry=0x0) at detect.c:1739
#14 0x000000000051cccb in FlowWorker (tv=0x7f3b94f1d870, p=0x7f3b69afed50, data=0x7f3b69b165d0, preq=0x7f3b94f53680, unused=<optimized out>) at flow-worker.c:260
#15 0x000000000059accd in TmThreadsSlotVarRun (tv=tv@entry=0x7f3b94f1d870, p=p@entry=0x7f3b69afed50, slot=slot@entry=0x1afd8d80) at tm-threads.c:145
#16 0x0000000000575c3e in TmThreadsSlotProcessPkt (p=0x7f3b69afed50, s=0x1afd8d80, tv=0x7f3b94f1d870) at tm-threads.h:147
#17 AFPReadFromRing (ptv=0x7f3b69aff700) at source-af-packet.c:1016
#18 0x0000000000578fbe in ReceiveAFPLoop (tv=0x7f3b94f1d870, data=0x7f3b69aff700, slot=<optimized out>) at source-af-packet.c:1579
#19 0x000000000059d432 in TmThreadsSlotPktAcqLoop (td=0x7f3b94f1d870) at tm-threads.c:348
#20 0x00007f3ba54b7dd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f3ba4932ead in clone () from /lib64/libc.so.6

(gdb) p *det_ctx
$2 = {tenant_id = 0, ticker = 831308, tv = 0x7f3b94f1d870, non_pf_id_array = 0x7f3b69be8a40, non_pf_id_cnt = 3, mt_det_ctxs_cnt = 0, mt_det_ctxs = 0x0, mt_det_ctxs_hash = 0x0,
  tenant_array = 0x0, tenant_array_size = 0, TenantGetId = 0x0, raw_stream_progress = 68649, buffer_offset = 5, pcre_match_start_offset = 580, filestore_cnt = 0,
  hcbd = 0x7f3b6a7e6bd0, hcbd_start_tx_id = 6, hcbd_buffers_size = 100, hcbd_buffers_list_len = 1, counter_alerts = 172, inspect_list = 0, inspect = {buffers = 0x7f3b69d18f70,
    buffers_size = 67, to_clear_idx = 1, to_clear_queue = 0x7f3b69d19e20}, multi_inspect = {buffers = 0x7f3b69d19f40, buffers_size = 67, to_clear_idx = 0,
    to_clear_queue = 0x7f3b69d1a380}, discontinue_matching = 1, flags = 0, tx_id_set = 0, tx_id = 0, p = 0x0, so_far_used_by_detect_sc_atomic__ = 1, inspection_recursion_counter = 2,
  match_array = 0x7f3b69be99b0, match_array_len = 31062, match_array_cnt = 8, tx_candidates = 0x7f3b69c26470, tx_candidates_size = 31062, non_pf_store_ptr = 0x7f3b8797b1d0,
  non_pf_store_cnt = 5, mtc = {ctx = 0x7f3b69be87b0, memory_cnt = 1, memory_size = 16}, mtcu = {ctx = 0x7f3b69be87f0, memory_cnt = 1, memory_size = 16}, mtcs = {ctx = 0x7f3b69be87d0,
    memory_cnt = 1, memory_size = 16}, pmq = {rule_id_array = 0x7f3b6abf0ea0, rule_id_array_cnt = 2, rule_id_array_size = 3590}, spm_thread_ctx = 0x7f3b69be8a20, io_ctx = {
    sig_match_array = 0x7f3b69be8a90 "", sig_match_size = 3860}, bj_values = 0x7f3b69d18f40, replist = 0x0, varlist = 0x0, filestore = {{file_id = 0, tx_id = 0} <repeats 15 times>},
  de_ctx = 0x33cf680, keyword_ctxs_array = 0x0, keyword_ctxs_size = 0, global_keyword_ctxs_size = 3, global_keyword_ctxs_array = 0x7f3b69d1a4a0, base64_decoded = 0x0,
  base64_decoded_len = 0, base64_decoded_len_max = 0, decoder_events = 0x0, events = 0}
(gdb) p grow_by
$3 = 53
(gdb) p size
$4 = <optimized out>
(gdb) up
#6  0x00000000004b3ea7 in DetectEngineHCBDGetBufferForTX (tx=0x7f3a88f26940, tx_id=158, det_ctx=det_ctx@entry=0x7f3b69be8500, flags=flags@entry=132 '\204',
    buffer_len=buffer_len@entry=0x7f3b7282c738, stream_start_offset=stream_start_offset@entry=0x7f3b7282c73c, htp_state=<optimized out>, f=<optimized out>, de_ctx=0x0)
    at detect-engine-hcbd.c:140
140    in detect-engine-hcbd.c
(gdb) p txs
$5 = 153

and here is an example where I think its showing memory is already corrupt

(gdb) bt
#0  0x00007f3a1b98028a in _int_free () from /lib64/libc.so.6
#1  0x0000000000590839 in ReassembleFree (ptr=<optimized out>, size=2048) at stream-tcp-reassemble.c:237
#2  0x00000000005fe97f in StreamingBufferClear (sb=sb@entry=0x7f39e30f10f8) at util-streaming-buffer.c:139
#3  0x0000000000584f00 in StreamTcpStreamCleanup (stream=stream@entry=0x7f39e30f10c0) at stream-tcp.c:207
#4  0x0000000000584f2a in StreamTcpSessionCleanup (ssn=ssn@entry=0x7f39e30f1030) at stream-tcp.c:223
#5  0x0000000000584f8e in StreamTcpSessionClear (ssnptr=0x7f39e30f1030) at stream-tcp.c:256
#6  0x000000000051687e in FlowClearMemory (f=f@entry=0x7f394b6c0180, proto_map=<optimized out>) at flow.c:965
#7  0x000000000051882e in FlowRecycler (th_v=0x7f3a0b4b8ac0, thread_data=0x7f393c0008c0) at flow-manager.c:923
#8  0x000000000059b9b7 in TmThreadsManagement (td=0x7f3a0b4b8ac0) at tm-threads.c:719
#9  0x00007f3a1c581dd5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f3a1b9fcead in clone () from /lib64/libc.so.6

Does suricata make it easy to compile with asan, maybe that would help?

Did you run 4.1.2 before? Does that show the issue as well?

I agree this looks like mem corruption.

We were running 4.0.1 so made a big jump. Should we try 4.1.2?

Are you able to compile Suricata 4.1.3 with ASAN enabled? Then it should blow up at the first error, not at some later time.

Sure, thats what I was asking before, is it easy to do? Pointers how?

If you compiler is recent enough, you should just be able to add '-fsanitize=address -fno-omit-frame-pointer' to your CFLAGS.

4.1.2 has been up over 45 minutes with no cores, compared to a few minutes that 4.1.3 would last before coring. I'll look at getting 4.1.3 recompiled with ASAN

Hi Andy, have you seen no crashes with 4.1.2 still?

Yes, still no cores on 4.1.2. I still owe your 4.1.3 compiled with ASAN

=================================================================
==31185==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0078003d8 at pc 0x0000008eb257 bp 0x7f478992c300 sp 0x7f478992c2f0
WRITE of size 8 at 0x62d0078003d8 thread T1 (W#01-ens5f1)
    #0 0x8eb256 in StreamingBufferGetDataAtOffset /root/suricata-4.1.3/src/util-streaming-buffer.c:875
    #1 0x5cff41 in DetectEngineHCBDGetBufferForTX /root/suricata-4.1.3/src/detect-engine-hcbd.c:203
    #2 0x5d0564 in PrefilterTxHttpRequestBody /root/suricata-4.1.3/src/detect-engine-hcbd.c:241
    #3 0x5f4f9d in DetectRunPrefilterTx /root/suricata-4.1.3/src/detect-engine-prefilter.c:117
    #4 0x567599 in DetectRunTx /root/suricata-4.1.3/src/detect.c:1398
    #5 0x567599 in DetectRun /root/suricata-4.1.3/src/detect.c:141
    #6 0x569fda in DetectNoFlow /root/suricata-4.1.3/src/detect.c:1679
    #7 0x569fda in Detect /root/suricata-4.1.3/src/detect.c:1739
    #8 0x6c39b4 in FlowWorker /root/suricata-4.1.3/src/flow-worker.c:260
    #9 0x801f1e in TmThreadsSlotVarRun /root/suricata-4.1.3/src/tm-threads.c:145
    #10 0x78ed09 in TmThreadsSlotProcessPkt /root/suricata-4.1.3/src/tm-threads.h:147
    #11 0x78ed09 in AFPReadFromRing /root/suricata-4.1.3/src/source-af-packet.c:1016
    #12 0x7982db in ReceiveAFPLoop /root/suricata-4.1.3/src/source-af-packet.c:1579
    #13 0x804b93 in TmThreadsSlotPktAcqLoop /root/suricata-4.1.3/src/tm-threads.c:348
    #14 0x7f47988dddd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
    #15 0x7f4797d58eac in __clone (/lib64/libc.so.6+0xfdeac)

0x62d0078003d8 is located 40 bytes to the left of 34000-byte region [0x62d007800400,0x62d0078088d0)
allocated by thread T1 (W#01-ens5f1) here:
    #0 0x7f4799dd8c90 in realloc (/lib64/libasan.so.4+0xdec90)
    #1 0x5cf572 in HCBDCreateSpace /root/suricata-4.1.3/src/detect-engine-hcbd.c:80
    #2 0x5d0119 in DetectEngineHCBDGetBufferForTX /root/suricata-4.1.3/src/detect-engine-hcbd.c:125
    #3 0x5d0564 in PrefilterTxHttpRequestBody /root/suricata-4.1.3/src/detect-engine-hcbd.c:241
    #4 0x5f4f9d in DetectRunPrefilterTx /root/suricata-4.1.3/src/detect-engine-prefilter.c:117
    #5 0x567599 in DetectRunTx /root/suricata-4.1.3/src/detect.c:1398
    #6 0x567599 in DetectRun /root/suricata-4.1.3/src/detect.c:141
    #7 0x569fda in DetectNoFlow /root/suricata-4.1.3/src/detect.c:1679
    #8 0x569fda in Detect /root/suricata-4.1.3/src/detect.c:1739
    #9 0x6c39b4 in FlowWorker /root/suricata-4.1.3/src/flow-worker.c:260
    #10 0x801f1e in TmThreadsSlotVarRun /root/suricata-4.1.3/src/tm-threads.c:145
    #11 0x78ed09 in TmThreadsSlotProcessPkt /root/suricata-4.1.3/src/tm-threads.h:147
    #12 0x78ed09 in AFPReadFromRing /root/suricata-4.1.3/src/source-af-packet.c:1016
    #13 0x7982db in ReceiveAFPLoop /root/suricata-4.1.3/src/source-af-packet.c:1579
    #14 0x804b93 in TmThreadsSlotPktAcqLoop /root/suricata-4.1.3/src/tm-threads.c:348
    #15 0x7f47988dddd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)

Thread T1 (W#01-ens5f1) created by T0 (Suricata-Main) here:
    #0 0x7f4799d31a7f in pthread_create (/lib64/libasan.so.4+0x37a7f)
    #1 0x8085d5 in TmThreadSpawn /root/suricata-4.1.3/src/tm-threads.c:1895
    #2 0x8d1416 in RunModeSetLiveCaptureWorkersForDevice /root/suricata-4.1.3/src/util-runmodes.c:362
    #3 0x8d5569 in RunModeSetLiveCaptureWorkers /root/suricata-4.1.3/src/util-runmodes.c:394
    #4 0x771451 in RunModeIdsAFPWorkers /root/suricata-4.1.3/src/runmode-af-packet.c:845
    #5 0x7835c0 in RunModeDispatch /root/suricata-4.1.3/src/runmodes.c:384
    #6 0x435aa9 in main /root/suricata-4.1.3/src/suricata.c:3003
    #7 0x7f4797c7d3d4 in __libc_start_main (/lib64/libc.so.6+0x223d4)
SUMMARY: AddressSanitizer: heap-buffer-overflow /root/suricata-4.1.3/src/util-streaming-buffer.c:875 in StreamingBufferGetDataAtOffset
Shadow bytes around the buggy address:
  0x0c5a80ef8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80ef8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c5a80ef8070: fa fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa fa
  0x0c5a80ef8080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef8090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80ef80c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==31185==ABORTING

This is probably index being -1 I would guess.

I also encountered this bug, but 4.1.3 will not crash without IDS rules.

Hi Andy, are you able to test the following? https://github.com/OISF/suricata/pull/3757 It is a collection of fixes backported from the master branch, and it replaces the whole buffer logic by something much simpler.

Jacky, are you able to test this PR as well? The crash is triggered in http_client_body inspection, which depends on rules being present.

Ran on 1 box all weekend with no cores

That is excellent news. Are you going to expand your tests further? Would love to get a bit more confidence this fixes the issue. Thanks!

Hi Victor, I found that there are 3 rules that can cause crash.
After I blocked the rules that triggered the crash, I ran for several days without crash.
I will test this PR

Ran on several more hosts overnight with no cores. All of these hosts were constantly coring before with 4.1.3

Thanks Andy, that is good news.

Jacky, are you able to share the rule (id's) for the offending 3 rules?