Project

General

Profile

Actions

Bug #6766

closed

multi-tenancy: dead lock during tenant loading

Added by Victor Julien 9 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Happens in CI sporadically.


Subtasks 2 (0 open2 closed)

Bug #6767: multi-tenancy: dead lock during tenant loading (6.0.x backport)ClosedVictor JulienActions
Bug #6768: multi-tenancy: dead lock during tenant loading (7.0.x backport)ClosedVictor JulienActions
Actions #1

Updated by Victor Julien 9 months ago

Running multi-tenant-02-test in a loop hits the condition pretty quickly:

Attaching to process 2468975
[New LWP 2468976]
[New LWP 2468977]
[New LWP 2468978]
[New LWP 2468979]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48
48    ./nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) i thr
  Id   Target Id                                           Frame 
* 1    Thread 0x7f0bff784240 (LWP 2468975) "Suricata-Main" 0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48
  2    Thread 0x7f0bff079640 (LWP 2468976) "DL#01"         __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57
  3    Thread 0x7f0bfe878640 (LWP 2468977) "DL#02"         __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57
  4    Thread 0x7f0bfe077640 (LWP 2468978) "DL#03"         __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57
  5    Thread 0x7f0bfd876640 (LWP 2468979) "DL#04"         __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57
(gdb) bt
#0  0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48
#1  ___pthread_mutex_lock (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:93
#2  0x000055680c31f457 in DetectLoadersSync () at detect-engine-loader.c:467
#3  0x000055680c316377 in DetectEngineMultiTenantSetup (unix_socket=false) at detect-engine.c:4345
#4  0x000055680c281629 in PostConfLoadedDetectSetup (suri=0x55680cb17c60 <suricata>) at suricata.c:2541
#5  0x000055680c282634 in SuricataMain (argc=24, argv=0x7ffdded05e48) at suricata.c:2966
#6  0x000055680c27b99c in main (argc=24, argv=0x7ffdded05e48) at main.c:22
(gdb)  thr 2
[Switching to thread 2 (Thread 0x7f0bff079640 (LWP 2468976))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57
57    ./nptl/futex-internal.c: No such file or directory.
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e371548, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at ./nptl/futex-internal.c:139
#3  0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2a4b70, cond=0x55680e371520) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55680e371520, mutex=0x55680e2a4b70) at ./nptl/pthread_cond_wait.c:627
#5  0x000055680c31fc1c in DetectLoader (th_v=0x55680e8879f0, thread_data=0x7f0bf8000b70) at detect-engine-loader.c:618
#6  0x000055680c285a27 in TmThreadsManagement (td=0x55680e8879f0) at tm-threads.c:557
#7  0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)  thr 3
[Switching to thread 3 (Thread 0x7f0bfe878640 (LWP 2468977))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57
57    in ./nptl/futex-internal.c
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2d4558, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at ./nptl/futex-internal.c:139
#3  0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e276b70, cond=0x55680e2d4530) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55680e2d4530, mutex=0x55680e276b70) at ./nptl/pthread_cond_wait.c:627
#5  0x000055680c31fc1c in DetectLoader (th_v=0x55680e8881e0, thread_data=0x7f0bf0000b70) at detect-engine-loader.c:618
#6  0x000055680c285a27 in TmThreadsManagement (td=0x55680e8881e0) at tm-threads.c:557
#7  0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)  thr 4
[Switching to thread 4 (Thread 0x7f0bfe077640 (LWP 2468978))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57
57    in ./nptl/futex-internal.c
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2eec78, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at ./nptl/futex-internal.c:139
#3  0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2b1b40, cond=0x55680e2eec50) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55680e2eec50, mutex=0x55680e2b1b40) at ./nptl/pthread_cond_wait.c:627
#5  0x000055680c31fc1c in DetectLoader (th_v=0x55680e8884c0, thread_data=0x7f0bf4000b70) at detect-engine-loader.c:618
#6  0x000055680c285a27 in TmThreadsManagement (td=0x55680e8884c0) at tm-threads.c:557
#7  0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)  thr 5
[Switching to thread 5 (Thread 0x7f0bfd876640 (LWP 2468979))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57
57    in ./nptl/futex-internal.c
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2bebf8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
    at ./nptl/futex-internal.c:139
#3  0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2d4d70, cond=0x55680e2bebd0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55680e2bebd0, mutex=0x55680e2d4d70) at ./nptl/pthread_cond_wait.c:627
#5  0x000055680c31fc1c in DetectLoader (th_v=0x55680e8887a0, thread_data=0x7f0be8000b70) at detect-engine-loader.c:618
#6  0x000055680c285a27 in TmThreadsManagement (td=0x55680e8887a0) at tm-threads.c:557
#7  0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Actions #2

Updated by Victor Julien 9 months ago

  • Status changed from Assigned to In Review
  • Label Needs backport to 6.0, Needs backport to 7.0 added
Actions #3

Updated by OISF Ticketbot 9 months ago

  • Subtask #6767 added
Actions #4

Updated by OISF Ticketbot 9 months ago

  • Label deleted (Needs backport to 6.0)
Actions #5

Updated by OISF Ticketbot 9 months ago

  • Subtask #6768 added
Actions #6

Updated by OISF Ticketbot 9 months ago

  • Label deleted (Needs backport to 7.0)
Actions #8

Updated by Jeff Lucovsky 8 months ago

  • Status changed from In Review to Closed
Actions

Also available in: Atom PDF