Actions
Bug #6766
closedmulti-tenancy: dead lock during tenant loading
Added by Victor Julien 9 months ago. Updated 8 months ago.
Affected Versions:
Effort:
Difficulty:
Label:
Description
Happens in CI sporadically.
Updated by Victor Julien 9 months ago
Running multi-tenant-02-test
in a loop hits the condition pretty quickly:
Attaching to process 2468975 [New LWP 2468976] [New LWP 2468977] [New LWP 2468978] [New LWP 2468979] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48 48 ./nptl/pthread_mutex_lock.c: No such file or directory. (gdb) i thr Id Target Id Frame * 1 Thread 0x7f0bff784240 (LWP 2468975) "Suricata-Main" 0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48 2 Thread 0x7f0bff079640 (LWP 2468976) "DL#01" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57 3 Thread 0x7f0bfe878640 (LWP 2468977) "DL#02" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57 4 Thread 0x7f0bfe077640 (LWP 2468978) "DL#03" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57 5 Thread 0x7f0bfd876640 (LWP 2468979) "DL#04" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57 (gdb) bt #0 0x00007f0bffe44f44 in lll_mutex_lock_optimized (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:48 #1 ___pthread_mutex_lock (mutex=0x55680e8879a8) at ./nptl/pthread_mutex_lock.c:93 #2 0x000055680c31f457 in DetectLoadersSync () at detect-engine-loader.c:467 #3 0x000055680c316377 in DetectEngineMultiTenantSetup (unix_socket=false) at detect-engine.c:4345 #4 0x000055680c281629 in PostConfLoadedDetectSetup (suri=0x55680cb17c60 <suricata>) at suricata.c:2541 #5 0x000055680c282634 in SuricataMain (argc=24, argv=0x7ffdded05e48) at suricata.c:2966 #6 0x000055680c27b99c in main (argc=24, argv=0x7ffdded05e48) at main.c:22 (gdb) thr 2 [Switching to thread 2 (Thread 0x7f0bff079640 (LWP 2468976))] #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57 57 ./nptl/futex-internal.c: No such file or directory. (gdb) bt #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e371548) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e371548, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2a4b70, cond=0x55680e371520) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x55680e371520, mutex=0x55680e2a4b70) at ./nptl/pthread_cond_wait.c:627 #5 0x000055680c31fc1c in DetectLoader (th_v=0x55680e8879f0, thread_data=0x7f0bf8000b70) at detect-engine-loader.c:618 #6 0x000055680c285a27 in TmThreadsManagement (td=0x55680e8879f0) at tm-threads.c:557 #7 0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) thr 3 [Switching to thread 3 (Thread 0x7f0bfe878640 (LWP 2468977))] #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57 57 in ./nptl/futex-internal.c (gdb) bt #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2d4558) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2d4558, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e276b70, cond=0x55680e2d4530) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x55680e2d4530, mutex=0x55680e276b70) at ./nptl/pthread_cond_wait.c:627 #5 0x000055680c31fc1c in DetectLoader (th_v=0x55680e8881e0, thread_data=0x7f0bf0000b70) at detect-engine-loader.c:618 #6 0x000055680c285a27 in TmThreadsManagement (td=0x55680e8881e0) at tm-threads.c:557 #7 0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) thr 4 [Switching to thread 4 (Thread 0x7f0bfe077640 (LWP 2468978))] #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57 57 in ./nptl/futex-internal.c (gdb) bt #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2eec78) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2eec78, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2b1b40, cond=0x55680e2eec50) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x55680e2eec50, mutex=0x55680e2b1b40) at ./nptl/pthread_cond_wait.c:627 #5 0x000055680c31fc1c in DetectLoader (th_v=0x55680e8884c0, thread_data=0x7f0bf4000b70) at detect-engine-loader.c:618 #6 0x000055680c285a27 in TmThreadsManagement (td=0x55680e8884c0) at tm-threads.c:557 #7 0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) thr 5 [Switching to thread 5 (Thread 0x7f0bfd876640 (LWP 2468979))] #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57 57 in ./nptl/futex-internal.c (gdb) bt #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55680e2bebf8) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55680e2bebf8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f0bffe40a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55680e2d4d70, cond=0x55680e2bebd0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x55680e2bebd0, mutex=0x55680e2d4d70) at ./nptl/pthread_cond_wait.c:627 #5 0x000055680c31fc1c in DetectLoader (th_v=0x55680e8887a0, thread_data=0x7f0be8000b70) at detect-engine-loader.c:618 #6 0x000055680c285a27 in TmThreadsManagement (td=0x55680e8887a0) at tm-threads.c:557 #7 0x00007f0bffe41ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f0bffed3850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Updated by Victor Julien 9 months ago
- Status changed from Assigned to In Review
- Label Needs backport to 6.0, Needs backport to 7.0 added
Updated by Juliana Fajardini Reichow 9 months ago
Updated by Jeff Lucovsky 8 months ago
- Status changed from In Review to Closed
Actions