Comments
3359 Comments
adamwill
karma
It seems like this crashes on startup in openQA testing. The F42 and F43 builds were fine; this may be caused by gcc or glibc or something, I guess.
This is the traceback:
Thread 1 (Thread 0x7f7be6f69780 (LWP 3656)):
#0 nsCOMPtr<nsIFile>::assign_assuming_AddRef () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/objdir/dist/include/nsCOMPtr.h:317
No locals.
#1 nsCOMPtr<nsIFile>::operator=(decltype(nullptr)) () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/objdir/dist/include/nsCOMPtr.h:596
No locals.
#2 mozilla::net::nsStandardURL::InvalidateCache () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/netwerk/base/nsStandardURL.cpp:399
No locals.
#3 mozilla::net::nsStandardURL::Init () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/netwerk/base/nsStandardURL.cpp:3194
No locals.
#4 0x00007f7bd1e54ddc in mozilla::net::nsStandardURL::TemplatedMutator<mozilla::net::nsStandardURL>::Init () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/netwerk/base/nsStandardURL.h:445
No locals.
#5 NS_MutateURI::Apply<nsresult (nsIStandardURLMutator::*)(unsigned int, int, nsTSubstring<char> const&, char const*, nsIURI*, nsIURIMutator**), nsIStandardURL::{unnamed type#1}, int, nsTSubstring<char> const&, char const*, nsCOMPtr<nsIURL>&, decltype(nullptr)>(nsresult (nsIStandardURLMutator::*)(unsigned int, int, nsTSubstring<char> const&, char const*, nsIURI*, nsIURIMutator**), nsIStandardURL::{unnamed type#1}&&, int&&, nsTSubstring<char> const&, char const*&&, nsCOMPtr<nsIURL>&, decltype(nullptr)&&) () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/objdir/dist/include/nsIURIMutator.h:589
No locals.
#6 nsJARURI::SetSpecWithBase () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/modules/libjar/nsJARURI.cpp:260
No locals.
#7 0x00007f7bd4de428e in nsJARURI::Mutator::SetSpecBaseCharset () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/modules/libjar/nsJARURI.h:134
No locals.
#8 NS_MutateURI::Apply<nsresult (nsIJARURIMutator::*)(nsTSubstring<char> const&, nsIURI*, char const*), nsTSubstring<char> const&, nsIURI*&, char const*&> () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/objdir/dist/include/nsIURIMutator.h:589
No locals.
#9 NS_NewURI () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/netwerk/base/nsNetUtil.cpp:2003
No locals.
#10 0x00007f7bd4dfcede in NS_NewURI () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/netwerk/base/nsNetUtil.cpp:1780
No locals.
#11 0x00007f7bd55e1f94 in nsChromeRegistry::ManifestProcessingContext::ResolveURI () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/chrome/nsChromeRegistryChrome.cpp:434
No locals.
#12 0x00007f7bd55e1dc2 in nsChromeRegistryChrome::ManifestContent () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/chrome/nsChromeRegistryChrome.cpp:465
No locals.
#13 0x00007f7bd4df1365 in ParseManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/ManifestParser.cpp:659
No locals.
#14 0x00007f7bd55e1017 in DoRegisterManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:509
No locals.
#15 0x00007f7bd55e1809 in nsComponentManagerImpl::RegisterManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:520
No locals.
#16 nsComponentManagerImpl::ManifestManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:527
No locals.
#17 0x00007f7bd4df12ff in ParseManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/ManifestParser.cpp:662
No locals.
#18 0x00007f7bd55e1017 in DoRegisterManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:509
No locals.
#19 0x00007f7bd55e0ef1 in nsComponentManagerImpl::RegisterManifest () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:520
No locals.
#20 nsComponentManagerImpl::RereadChromeManifests () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:544
No locals.
#21 0x00007f7bd556c14d in nsComponentManagerImpl::Init () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/components/nsComponentManager.cpp:424
No locals.
#22 0x00007f7bd555f18d in NS_InitXPCOM () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/xpcom/build/XPCOMInit.cpp:444
No locals.
#23 0x00007f7bd555e96b in ScopedXPCOMStartup::Initialize () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/toolkit/xre/nsAppRunner.cpp:2021
No locals.
#24 0x00007f7bd55498e9 in XREMain::XRE_main () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/toolkit/xre/nsAppRunner.cpp:6156
No locals.
#25 XRE_main () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/toolkit/xre/nsAppRunner.cpp:6246
No locals.
#26 0x000055c95e36cd0b in do_main () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/browser/app/nsBrowserApp.cpp:268
No locals.
#27 0x000055c95e36a0dc in main () at /usr/src/debug/firefox-147.0.1-1.fc44.x86_64/browser/app/nsBrowserApp.cpp:532
No locals.
I don't know why there are 'No locals', since I have the correct firefox-debuginfo installed :/
karma
This breaks FreeIPA replica deployment. The replica install command we run is:
ipa-replica-install --ip-address=172.16.2.107 --setup-dns --auto-forwarders --setup-ca --allow-zone-overlap -U --principal admin --admin-password monkeys123
this fails. ipareplica-install.log shows:
DEBUG: https://ipa002.test.openqa.fedoraproject.org:443 "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 500 None
then a big traceback which is just complaining about that 500 error, really.
The system journal shows:
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.906391382 -0500] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.925763852 -0500] - NOTICE - attrcrypt_cipher_init - No symmetric key found for cipher AES in backend ipaca, attempting to create one...
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.929425937 -0500] - INFO - attrcrypt_cipher_init - Key for cipher AES successfully generated and stored
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.930715149 -0500] - NOTICE - attrcrypt_cipher_init - No symmetric key found for cipher 3DES in backend ipaca, attempting to create one...
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.934158835 -0500] - INFO - attrcrypt_cipher_init - Key for cipher 3DES successfully generated and stored
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.935641837 -0500] - NOTICE - mdb_start_autotune - found 3.8 GB physical memory
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.936243177 -0500] - NOTICE - mdb_start_autotune - found 2.5 GB available
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.936837399 -0500] - NOTICE - mdb_start_autotune - cache autosizing: ipaca entry cache (2 total): 64.0 MB
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.937584360 -0500] - NOTICE - mdb_start_autotune - cache autosizing: ipaca dn cache (2 total): 64.0 MB
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.938290571 -0500] - NOTICE - mdb_start_autotune - total cache size: 1.1 GB
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.943660819 -0500] - ERR - ipa-topology-plugin - ipa_topo_be_state_change - backend ipaca is coming online; checking domain level and init shared topology
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:02.953036833 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=test,dc=openqa,dc=fedoraproject,dc=org--no CoS Templates found, which should be added before the CoS Definition.
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org python3[2832]: GSSAPI client step 1
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org python3[2832]: GSSAPI client step 1
Jan 19 11:59:02 ipa003.test.openqa.fedoraproject.org python3[2832]: GSSAPI client step 1
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org python3[2832]: GSSAPI client step 2
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:03.757862332 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding the replication changelog RUV, this may take several minutes...
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:03.760116115 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding replication changelog RUV complete. Result 0 (Success)
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:03 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 2
Jan 19 11:59:04 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:04.040131946 -0500] - ERR - ipa-topology-plugin - ipa_topo_be_state_changebackend ipaca is going offline; inactivate plugin
Jan 19 11:59:04 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:04.042487719 -0500] - NOTICE - NSMMReplicationPlugin - multisupplier_be_state_change - Replica o=ipaca is going offline; disabling replication
Jan 19 11:59:04 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:04.091180164 -0500] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=caToipa002.test.openqa.fedoraproject.org" (ipa002:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica.
Jan 19 11:59:04 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:04.248385866 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=test,dc=openqa,dc=fedoraproject,dc=org--no CoS Templates found, which should be added before the CoS Definition.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.508121555 -0500] - INFO - dbmdb_import_monitor_threads - import ipaca: Import writer thread usage: run: 3.21% read: 90.51% write: 5.66% pause: 0.18% txnbegin: 0.00% txncommit: 0.46%
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.518795902 -0500] - ERR - agmt="cn=meToipa002.test.openqa.fedoraproject.org" (ipa002:389) - clcache_load_buffer - Can't locate CSN 696e8d13000100030000 in the changelog (DB rc=-12797). If replication stops, the consumer may need to be reinitialized.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.538975563 -0500] - INFO - dbmdb_import_monitor_threads - import ipaca: Workers finished; cleaning up...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.541354337 -0500] - INFO - dbmdb_import_monitor_threads - import ipaca: Workers cleaned up.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.542230618 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Indexing complete. Post-processing...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.543016549 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Generating numsubordinates (this may take several minutes to complete)...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.545466343 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Generating numSubordinates complete.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.546236064 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Flushing caches...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.547045375 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Closing files...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.547930766 -0500] - INFO - dbmdb_public_dbmdb_import_main - import ipaca: Import complete. Processed 179 entries in 11 seconds. (16.27 entries/sec)
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.559681885 -0500] - ERR - ipa-topology-plugin - ipa_topo_be_state_change - backend ipaca is coming online; checking domain level and init shared topology
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.560961796 -0500] - NOTICE - NSMMReplicationPlugin - multisupplier_be_state_change - Replica o=ipaca is coming online; enabling replication
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.561797918 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding the replication changelog RUV, this may take several minutes...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.562565609 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding replication changelog RUV complete. Result 0 (Success)
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.564784743 -0500] - WARN - NSMMReplicationPlugin - replica_reload_ruv - New data for replica o=ipaca does not match the data in the changelog.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: Recreating the changelog file. This could affect replication with replica's consumers in which case the consumers should be reinitialized.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.667844381 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding the replication changelog RUV, this may take several minutes...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.668666402 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding replication changelog RUV complete. Result 0 (Success)
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.671010066 -0500] - INFO - dbmdb_import_all_done - Backend ipaca is now online.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.672946189 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=test,dc=openqa,dc=fedoraproject,dc=org--no CoS Templates found, which should be added before the CoS Definition.
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.673843400 -0500] - ERR - ipa-topology-plugin - segment to be deleted does not exist
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.675056192 -0500] - NOTICE - mdb_start_autotune - found 3.8 GB physical memory
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.676273944 -0500] - NOTICE - mdb_start_autotune - found 2.5 GB available
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.677032955 -0500] - NOTICE - mdb_start_autotune - total cache size: 1.1 GB
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.677710356 -0500] - ERR - ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.678532807 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding the replication changelog RUV, this may take several minutes...
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.679442669 -0500] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUVs - Rebuilding replication changelog RUV complete. Result 0 (Success)
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.680292850 -0500] - NOTICE - NSMMReplicationPlugin - replica_subentry_check - Need to create replication keep alive entry <cn=repl keep alive 5,o=ipaca>
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:15.680953161 -0500] - INFO - NSMMReplicationPlugin - replica_subentry_create - add dn: cn=repl keep alive 5,o=ipaca
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: objectclass: top
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: objectclass: ldapsubentry
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: objectclass: extensibleObject
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: keepalivetimestamp: 0
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: cn: repl keep alive 5
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 1
Jan 19 11:59:15 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: GSSAPI client step 2
Jan 19 11:59:16 ipa003.test.openqa.fedoraproject.org /usr/sbin/ipa-replica-install[2832]: [IPA.API] host/ipa003.test.openqa.fedoraproject.org@TEST.OPENQA.FEDORAPROJECT.ORG: idrange_show: SUCCESS [ldap2_140711578134704] {"cn": "TEST.OPENQA.FEDORAPROJECT.ORG_id_range", "rights": false, "all": false, "raw": false, "version": "2.257"}
Jan 19 11:59:16 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:16.437302675 -0500] - ERR - ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
Jan 19 11:59:16 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:16.440955550 -0500] - ERR - ipa-topology-plugin - segment to be deleted does not exist
Jan 19 11:59:16 ipa003.test.openqa.fedoraproject.org ns-slapd[5817]: [19/Jan/2026:14:59:16.578517572 -0500] - ERR - ipa-topology-plugin - ipa_topo_post_mod - segment to be modified does not exist
Jan 19 11:59:18 ipa003.test.openqa.fedoraproject.org python3.14[6892]: detected unhandled Python exception in '/usr/lib/python3.14/site-packages/pki/server/pkispawn.py'
Not sure what's going on there, but it's definitely triggered by this update. The same test is passing on previous and later updates.
https://github.com/shadow-maint/shadow/pull/1510 fixes that problem.
I found another major problem caused by 4.19.0 (I found it by installing it in my toolbox containers to debug the above problem, actually): https://github.com/shadow-maint/shadow/issues/1509
We should definitely not land 4.19.0 anywhere until that's fixed too, because it makes toolbox containers almost inaccessible.
Thanks!
The 'test code' is more or less https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/role_deploy_domain_controller.pm (where we deploy a FreeIPA server) and https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/role_deploy_domain_controller_check.pm (where we uninstall it), but it probably boils down to more or less:
<start with older authselect>
ipa-server-install -U --auto-forwarders --realm=TEST.OPENQA.FEDORAPROJECT.ORG --domain=test.openqa.fedoraproject.org --ds-password=monkeys123 --admin-password=monkeys123 --setup-dns --reverse-zone=2.16.172.in-addr.arpa --allow-zone-overlap
<upgrade authselect>
ipa-server-install -U --uninstall
I'm not sure how many of the setup steps we do before running ipa-server-install are strictly required to get it to work far enough to hit this failure.
Oh, and I've thought about running openQA tests on pull requests, but haven't had the cycles to really look into it yet. Until recently we definitely didn't have enough resources (test runners) to do it anyway. Now we possibly do but I'd need to check, and figure out all the work to trigger jobs from the scratch builds done for pull requests, and report results back to dist-git...and we're moving dist-git to forgejo anyway, so it sort of makes sense to wait for that...
FreeIPA tests are very complex, they run across multiple interconnected machines. Porting them to Fedora CI isn't trivial (and partly impossible, as we test GUI enrolment via Cockpit, which Fedora CI just can't do). There also isn't any kind of dependency-triggered testing in Fedora CI ATM, the tests that run on a PR for a package are just the tests specified for that package, we don't automatically run tests for its dependencies (in either direction).
For this specific situation we could probably come up with a much simpler reproducer that doesn't involve FreeIPA, though, I guess? Just a test that runs a backup before updating the package and tries to restore it afterwards. That shouldn't be too difficult to add as an authselect-specific package test, per https://docs.fedoraproject.org/en-US/ci/tmt/ ...
well, maybe printf rather than echo, not sure if the newline matters.
I was waiting for them to decide which of the two PRs to merge, but I'm confident what the issue is, and that any fix which allows ! as the hash will be fine.
It's effectively doing echo "root:!" | chpasswd -R /mnt/sysroot -e (just using Python subprocess), you can reproduce the problem with that.
ah, it looks like it did refire the test, but the test system doesn't send a QUEUED or RUNNING result, so we don't know it's been re-run till it finishes and the real result arrives.
@fhrdina who created the test plan...
This fails gating because the configured test plans require authentication to check out:
Command 'git clone https://gitlab.com/redhat/centos-stream/tests/xapian.git /var/tmp/tmt/run-001/plans/tier1-public/discover/xapian-core/tests' returned 128.
# stderr (3/3 lines)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cloning into '/var/tmp/tmt/run-001/plans/tier1-public/discover/xapian-core/tests'...
remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://gitlab.com/help/topics/git/troubleshooting_git.md#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.com/redhat/centos-stream/tests/xapian.git/'
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
please fix the test configuration. @canderson9
This is blocked because coreos.cosa.build-and-test failed. Unfortunately it's long enough ago that the logs have been garbage collected, and the "Re-Trigger Tests" button doesn't re-fire the test (which implies there's a bug or missing feature in whatever schedules that test).
Test discovery fails:
/var/tmp/tmt/run-001
/plans/podman/system/local-root
discover
url: https://github.com/lsm5/podman
cloned-commit-hash: ef584d4d6d
ref: tmt-fedora-centos
plan failed.
The exception was caused by 1 earlier exceptions
Cause number 1:
Command 'git checkout -f tmt-fedora-centos' returned 1.
# stderr (1/1 lines)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: pathspec 'tmt-fedora-centos' did not match any file(s) known to git
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---^---^---^---^---^---
[2026-01-16 00:51:31] [E] [test-scheduler-testing-farm] Exception raised: Failed to discover tests, TMT metadata are absent or corrupted.
https://github.com/lsm5/podman has no tmt-fedora-centos branch. It looks like this worked 10 months ago but not now. @lsm5 did you wipe the branch, or something?
This seems to be failing because netcat is broken for https://fedoraproject.org/wiki/Changes/droppingOfCertPemFile :
:: [ 22:10:36 ] :: [ BEGIN ] :: Starting netcat :: actually running 'nc -l 1125 -c mockserver.sh &'
nc: failed to open CA file '/etc/pki/tls/cert.pem': No such file or directory
This update has been unpushed.
let's just unpush this.
Also, it only crashes on x86_64, not aarch64. But it's reproducible - it's failing across three different tests on both prod and staging.