Here's the backtrace I was able to get:

#0  cos_cache_attr_compare (e1=<optimized out>, e2=0x7f2cfa43df18) at ldap/servers/plugins/cos/cos_cache.c:2924
        com_Result = <optimized out>
        pAttr = 0x7f2cfa409200
        pTemplate = 0x7f2cfa414280
        pAttr1 = 0x312e
        pTemplate1 = <optimized out>
#1  0x00007f2d1585fca6 in insertion_sort_qsort_partitions (arg=<optimized out>, cmp=0x7f2d13b50db0 <cos_cache_attr_compare>, swap_type=SWAP_WORDS_64, size=8, total_elems=<optimized out>, pbase=<optimized out>) at qsort.c:220
        base_ptr = <optimized out>
        thresh = <optimized out>
        end_ptr = <optimized out>
        tmp_ptr = 0x7f2cfa43df18 ".1"
        max_thresh = 32
        run_ptr = <optimized out>
        base_ptr = <optimized out>
        max_thresh = 32
        swap_type = SWAP_WORDS_64
        depth = <optimized out>
#2  __GI___qsort_r (pbase=<optimized out>, total_elems=<optimized out>, size=size@entry=8, cmp=cmp@entry=0x7f2d13b50db0 <cos_cache_attr_compare>, arg=arg@entry=0x0) at qsort.c:399
        base_ptr = <optimized out>
        max_thresh = 32
        swap_type = SWAP_WORDS_64
        depth = <optimized out>
#3  0x00007f2d158601ac in __GI_qsort (b=<optimized out>, n=<optimized out>, s=s@entry=8, cmp=cmp@entry=0x7f2d13b50db0 <cos_cache_attr_compare>) at qsort.c:408
#4  0x00007f2d13b55b4c in cos_cache_index_all (pCache=<optimized out>) at ldap/servers/plugins/cos/cos_cache.c:2797
        attrcount = <optimized out>
        pDef = <optimized out>
        pAttrVal = <optimized out>
        tmpindex = <optimized out>
        cmpindex = 0
        actualCount = 0
        ret = -1
        ret = 0
        pNewCache = 0x7f2cfa401000
        cache_built = 0
        ret = -1
        max_tries = <optimized out>
#5  cos_cache_create_unlock () at ldap/servers/plugins/cos/cos_cache.c:463
        ret = 0
        pNewCache = 0x7f2cfa401000
        cache_built = 0
        ret = -1
        max_tries = <optimized out>
#6  cos_cache_creation_lock () at ldap/servers/plugins/cos/cos_cache.c:586
        ret = -1
        max_tries = <optimized out>
#7  0x00007f2d13b56185 in cos_cache_wait_on_change (arg=<optimized out>) at ldap/servers/plugins/cos/cos_cache.c:419
#8  0x00007f2d16139739 in _pt_root (arg=0x7f2d102e2680) at ../../../../nspr/pr/src/pthreads/ptthread.c:201
        rv = <optimized out>
        thred = 0x7f2d102e2680
        detached = 1
        tid = 8246
#9  0x00007f2d158abbd1 in start_thread (arg=<optimized out>) at pthread_create.c:447
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139831168464576, 2270171665239543636, 139831168464576, -3040, 0, 140732134582048, 2270171665218572116, 2270544497101803348}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#10 0x00007f2d1592be9c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
quit

This happens during FreeIPA server deployment. It seems like ns-slapd gets started up, shut down and started up again several times during server deployment. This happens at the "[7/10]: upgrading server" stage of the process.

Thanks. The comparison function can never return zero: https://github.com/389ds/389-ds-base/blob/main/ldap/servers/plugins/cos/cos_cache.c#L2933

This is clearly a 389-ds-base bug. The old qsort implementation in glibc did not tickle it because it rarely called the comparison function with equal pointer arguments. We already worked around similar application problems in other places in the new implementation, we can probably do it in the insertion sort phase as well.

The crash occurs in cos_cache:2924 on that line

cosTemplates *pTemplate1 = (cosTemplates *)pAttr1->pParent;

The only reason of the crash would be the dereferenced pAttr1. It exists a known heap corruption bug (https://github.com/389ds/389-ds-base/issues/5984) that was not fixed in the Rawhide version of the DS that was tested. This ticket is now fixed. It is highly probable that the crash was due to #5984.

I agree that cos_cache_attr_compare can not return a 0 (equal). It is not a problem for DS to force an arbitrary sorting rather than letting qsort decide. I do not see why not returning 0 would crash in that routine

Before the workaround, glibc had this loop:

 220       while (cmp (run_ptr, tmp_ptr, arg) < 0)
 221         tmp_ptr -= size;

https://sourceware.org/git/?p=glibc.git;a=blob;f=stdlib/qsort.c;h=ad110e8a892a66e1fc90f850b828e1a2d09e2ac5;hb=HEAD#l220

The loop is known to terminate if the comparison function is correct because eventually, run_ptr == tmp_ptr, and cmp must return zero. If that never happens, we eventually run into non-allocated memory regions. The only access to that memory is from the cmp function here, not from the qsort implementation, so that crash will happen in the comparison callback.

Okay I agree with your explanations. The loop overrun the buffer and make the cmp function sigsev.

I created https://github.com/389ds/389-ds-base/issues/5992 to track this issue.