This update implements the bin-sbin merge (https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin).
On systems where all packages have been rebuilt with %_sbindir==/usr/bin, /usr/sbin will be symlinked to /usr/bin. Packages which have files in /usr/sbin but weren't rebuilt (e.g. nbdkit, wesnoth) will prevent the symlink from being created. Packages which have files with the same name in /usr/bin and /usr/sbin will FTI (backintime-qt, beesu, bind9-next, chkrootkit, etherape, hddtemp, lshw-gui, mate-system-log, msktutil, rdist, subscription-manager, system-switch-java, tmpwatch, xawtv, setuptool). New system installations will have /usr/sbin symlinked to /usr/bin. If local files or symlinks are present in /usr/sbin, they will prevent the directory from being replaced by a symlink.
/usr/local/sbin is treated similarly to /usr/sbin. Since no packages provide files there, it'll be made into a symlink to /usr/local/bin if the user didn't install any local files there.
Automatic update for httpd-2.4.61-1.fc41.
Automatic update for httpd-2.4.59-4.fc41.
Automatic update for postfix-3.9.0-4.fc41.
Automatic update for postfix-3.9.0-3.fc41.
Automatic update for postfix-3.9.0-2.fc41.
Automatic update for postfix-3.9.0-1.fc41.
Automatic update for glibc-2.39.9000-32.fc41.
Please login to add feedback.
| 0 | 0 | Test Case HTTPd |
| 0 | 0 | Test Case Postfix |
| 0 | 0 | Test Case Sendmail |
| 0 | 0 | Test Case bind |
| 0 | 0 | Test Case coreutils |
| 0 | 0 | Test Case dmidecode |
| 0 | 0 | Test Case dosfstools |
| 0 | 0 | Test Case e2fsprogs |
| 0 | 0 | Test Case filesystem |
| 0 | 0 | Test Case multipath kpartx crash |
| 0 | 0 | Test Case multipathd error status |
| 0 | 0 | Test Case policycoreutils semanage |
| 0 | 0 | Test Case policycoreutils semodule |
This update's test gating status has been changed to 'waiting'.
This update has obsoleted httpd-2.4.61-1.fc41, and has inherited its bugs and notes.
This update has obsoleted postfix-3.9.0-4.fc41, and has inherited its bugs and notes.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
That's the well-know failure that apparently is insurmountable.
So apparently all
fedora-ci.koji-build.tier0.functionalfail like above.coreos.cosa.build-and-testfails like this:I have no idea what's this about, but it doesn't seem to be caused by the changes in this update.
zbyszek edited this update.
New build(s):
Karma has been reset.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
The openQA tests all failed because this seems to prevent console login from working. Trying to log in to a console as root just cycles you back to the login prompt. We didn't get any logs from the openQA tests because it can't upload any if it can't login. I'll try and reproduce this manually and find the cause tomorrow.
Oh, I see it now: Jul 10 01:08:16 localhost.localdomain login[1425]: LOGIN ON tty2 BY fedora Jul 10 01:08:16 localhost.localdomain audit[1434]: AVC avc: denied { transition } for pid=1434 comm="login" path="/usr/bin/bash" dev="vda4" ino=2386 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0
Not sure why I didn't see it before. With permissive it works.
Arrgh, the selinux policy update never went active. FEDORA-2024-64134f8805 obsoleted the earlier update, and then was blocked.
FEDORA-2024-2711625691 fixed the selinux policy. When can we rely on the updated selinux package to be visible to tests?
zbyszek edited this update.
New build(s):
Karma has been reset.
[Pipeline] { (Scratch-Build in Koji) (hide)
I guess infra is having troubles again.
zbyszek edited this update.
New build(s):
Removed build(s):
Karma has been reset.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
The updated selinux-policy was not ready for primetime and has been untagged, but while it was tagged tests re-ran on this update and several further issues are visible...
podman runcommand fails with errorError: netavark: iptables: No such file or directory (os error 2)/usr/libexec/generate-rndc-key.shto fail. That script seems to be carried downstream and calls/usr/sbin/rndc-confgenand/sbin/restoreconp11-kit-trustfails; according to the journal, it fails because/proc/self/fd/5: line 1: /usr/sbin/alternatives: No such file or directoryThanks, this is very useful.
podman:
/usr/sbin/iptablesis provided byiptables.rpm. There is nothing special going on, the file was moved, but the compat symlink should be created by filesystem.rpm.fsck.hfs: I sent a mail to the maintainers and proposed a pull request to drop one of the symlinks.
rndc-confgen: the name was missing from the scriptlet in filesystem.rpm to the compat symlink wasn't created. I'll update filesystem.rpm.
alternatives: same as with iptables, the file was moved but the compat symlink should be created. I assume that this symlink is not being created in ostree. I'll write to the ostree folks to figure something out.
nbdkit and libselinux: I rebuilt nbdkit yesterday, I'll rebuild libselinux now.
I downloaded the
02723234-Fedora.x86_64-Rawhide.oci.tar.xzfrom https://openqa.fedoraproject.org/tests/2723234, and it has a symlink:usr/sbin -> bin. But iptables is not installed. So it seems to be some bug in how the image is put together.Ah, we have a log of all packages installed in the image in https://openqa.fedoraproject.org/tests/2723234/file/_container_build_kiwi-image-root.log. iptables is never mentioned there. So I guess this worked by accident because iptables was pulled in via dependency and broke on some unrelated changes.
Nvm, it's trying to run iptables on the host.
I downloaded the ISO and on the host we have
/usr/sbin is a symlink. I see that something mangles $PATH in the user session to insert the duplicate directories. That's annoying, but not a big issue. We'll solve it later. (systemd itself deduplicates:
)
This might require more work on the Image Mode side of things. I've made: - https://gitlab.com/fedora/bootc/tracker/-/issues/29 - https://github.com/coreos/fedora-coreos-tracker/issues/1714#issuecomment-2223100778
In the meantime, as a short term workaround, we can probably rebuild p11-kit in this side tag to update the sbin macros mention in the spec file.
zbyszek edited this update.
New build(s):
Removed build(s):
Karma has been reset.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
On the latest set of results:
losetupviasubprocess.runby the looks of itnamed.servicefails to start: "named.service: Failed at step EXEC spawning /usr/sbin/named: No such file or directory". This may be because, in bind, the executable was moved from /usr/sbin/named to /usr/bin/named , but /usr/lib/systemd/system/named.service still specifiesExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS. (It also hasExecReloadandExecStoplines that specify paths under/usr/sbin)I wanted to say "that matches what I wrote", but I now see that I didn't actually press Enter.
\2. The anaconda installer image is worrying: the image is missing various binaries. E.g. losetup, rpm, mkswap, etc. But the rpmdb is there and dnf… So it looks like some files got randomly ignored. I would suspect files that were moved, but /usr/bin/rpm hasn't moved. I'm surprised that the image boots as far as it does.
Where can I look at the logs of how that image is put together?
\3. server_role_deploy_domain_controller logs show many many AVCs. But I see libsystemd-core-256.1-8.fc41.so in one of the error messages, while systemd-256.2-1.fc41 is stable. So this particular error was already solved. I don't know why the old version of systemd is used.
But the fatal error seems to be this: Jul 12 19:59:16 ipa001.test.openqa.fedoraproject.org (named)[11074]: named.service: Failed at step EXEC spawning /usr/sbin/named: No such file or directory This is actually caused by a bug in my patches for filesystem.rpm: 'named' (and a bunch of other filenames in /usr/sbin) were missing from the list. That it fixed in the latest build.
\5. _advisory_post module fails in many places. This is because kexec-tools was rebuilt outside of the side tag. I'll rebuild all the packages that do that (kexec-tools, glibc so far) again later.
OK, I think I figured out how my plan can't work. Unfortunately we end up mixing old and new packages in the CI. Many more packages should have dependencies on the updates filesystem package. I covered the cases which are required to satisfy explicit dependencies in packages, but not the cases where a script uses the old path. All packages that have been rebuilt for merged-sbin which have files in /usr/sbin should have
Requires: filesystem(unmerged-sbin-symlinks). I think the only reasonable way to do this is to install an rpm provides generator.zbyszek edited this update.
New build(s):
Removed build(s):
Karma has been reset.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'failed'.
2 - oh, yeah, thinking about it, this is going to be an issue. the installer environment is built by lorax, which has a whole system for installing packages then removing bits of them to save space. basically I think you are going to need to look at every instance of the string
sbinin https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl .5 - as of the latest refresh, it's now device-mapper which is behind rawhide and needs rebuilding again.
zbyszek edited this update.
New build(s):
Removed build(s):
Karma has been reset.
This update has obsoleted glibc-2.39.9000-32.fc41, and has inherited its bugs and notes.
zbyszek edited this update.
New build(s):
Removed build(s):
Karma has been reset.
I screwed up with glibc. I saw that it has been rebuilt, but I forgot that @codonell was kind enough to block glibc-2.39.9000-32.fc41 from going to stable. So I didn't need to build again.
https://src.fedoraproject.org/rpms/filesystem/pull-request/15 adds a generator that fixes the problem described 3 messages up. As packages are rebuilt, they'll get an autogenerated Requires on the new filesystem. I tested this locally in mock so I'm pretty sure it'll work, but of course reviews would be great. It'll also need to be hooked up in redhat-rpm-macros or such.
Big thanks for the pointer about lorax! → https://github.com/weldr/lorax/pull/1409
https://src.fedoraproject.org/rpms/selinux-policy/pull-request/448 should help with the ostree failures. If that's merged, then packages using %selinux_modules_install will need to be rebuilt (container-selinux, passt-selinux).
_advisory_post failed in some cases with: Warning: Failed to open the file /usr/local/bin/setup_repos.py: No such file or directory. Maybe /usr/local/bin doesn't exist? Though I have no idea why it wouldn't.
In the attached 02729556-Fedora-KDE-Live-x86_64-FEDORA-2024-3aafcac6a8.iso, there is indeed no /usr/local/bin or /usr/local/sbin. Is this a bug in the test to assume that this directory exists or a bug in the image?
the test process downloads that script, it wouldn't be in the ISO (it's part of internal openQA stuff, it's a script for downloading packages from the update under test and workaround packages, to make that faster). Anything going wrong around that script is definitely an openQA issue, I'll look at it and fix it.
well...okay, I kinda take it back.
The issue is that the openQA test logic assumes
/usr/local/binwill exist, it doesn't create it before trying to download the script to it. So, sure, on the one hand, I can just change the test to do that. But...it seems like a significant change that this update causes/usr/local/binto no longer exist as part of the stock Fedora filesystem, and it doesn't seem like it's within the scope of the change. I'd like to be sure that the removal of/usr/local/binis intentional and within the scope of the Change before "fixing" this.filesystem-3.18-9.fc41 contained
/usr/local/bin, filesystem-3.18-18.fc41 does not.I think the issue is that the file list in the spec previously had an entry that was just
/usr/local, so it would include all directories under/usr/local, but it no longer does. It has%dir /usr/local, which includes only the directory/usr/localitself, and then a bunch of individual entries under it:but there's no
/usr/local/binentry, so/usr/local/binis not in the package. I would guess some other paths under/usr/localmay have dropped out due to this change too, but I haven't checked for sure.FTR:
missing
/usr/local/binwas my error, fixed in https://src.fedoraproject.org/rpms/filesystem/c/9a7998077e1f257bb2caffe7054b32bc4a2c3954.https://github.com/weldr/lorax/pull/1409 was merged.
As discussed on fedora-devel, I'll dump the side tag. The changes in filesystem and rpm have been reverted for now.
This update has been unpushed.