Comments

1260 Comments

OK, so I played around with this locally, using a couple of test packages, foo and bar. I built a foo-1-1 which is just a completely empty package. bar has bar and bar-udev binary packages (also completely empty), with bar-udev requiring bar = %{version}-%{release}.

I built a bar-1-1 where the bar subpackage obsoletes foo < 2-1 and provides foo = %{version}-%{release}, and a bar-1-2 where the obsoletes and provides are in the bar-udev subpackage. I put those packages in a side repo and ran dnf update, and it reproduces the problem: dnf refuses to do anything because it can't install both bar-1-1 and bar-udev-1-2.

I played around with some possible solutions. An explicit Conflicts: bar < 1-2 in bar-udev-1-2 doesn't help. But an explicit Obsoletes: bar < 1-2 in bar-udev-1-2 does help: it makes the upgrade work in the intended way, by only installing bar-udev-1-2.

I was worried it would cause a problem if you have bar-1-1 but not bar-udev-1-1 installed - I was worried it would pull in bar-udev-1-2 on update. But it doesn't seem to:

[root@adam noarch]# dnf --disablerepo=* install /home/adamw/local/repo/x86_64/bar-1-1.noarch.rpm ... Installed: bar-1-1.noarch

Complete!
[root@adam noarch]# dnf --disablerepo=* --enablerepo=side --refresh update
Private side repo                                                                     2.9 MB/s | 3.0 kB     00:00    
Dependencies resolved.
======================================================================================================================
 Package                   Architecture                 Version                      Repository                  Size
======================================================================================================================
Upgrading:
 bar                       noarch                       1-2                          side                       6.1 k

Transaction Summary
======================================================================================================================
Upgrade  1 Package

Total size: 6.1 k

so I think this is a plausible solution: adding Obsoletes: systemd < 245.6-1 to the systemd-udev subpackage. Can you see any other possible issues with that? If not, do you mind if I try it? Thanks!

The change was made a while back, but there's been no update in the interim. The last systemd to hit Bodhi was systemd-245.4-1.fc32, which is the build that's in the frozen stable F32 repo, the one this update is conflicting with. No other systemd update has been issued for F32 since.

I think the problem is that dnf can "see" both systemd-245.4-1.fc32 and systemd-udev-245.6-1.fc32 - the former in the fedora repo, the latter in updates-testing (or in the openQA test, in the repo called advisory, which is where we put the update under test). If the update was pushed stable this would still be the case - systemd-245.4-1.fc32 would not disappear, it would still be in the fedora repo and still visible to dnf. I believe dnf's behaviour for obsoletes is "greedy": it will try to pull in every different package it can see that obsoletes an installed package. So it tries to pull in the older systemd as well as the newer systemd-udev, and that's obviously problematic.

We don't see this problem in Rawhide because we don't have this "two different builds available in different repos" wrinkle there. We always only have the one main repo for Rawhide, dnf only sees one build of systemd at a time there.

I'm going to test if having the newer systemd-udev explicitly conflict with or obsolete the older systemd will help here...

openQA failure here needs a bit of investigation. It's probably some kind of bug somewhere, not sure if it's an issue with the package deps or with dnf. I'm going to disable autopush on this update so it doesn't get autopushed tonight, and will look into it tomorrow if no-one else beats me to it.

This update has been unpushed.

karma

Yup, broken. openQA hit the same crash dwmw2 hit, in FreeIPA server deployment.

BZ#1831086 softhsm use-after-free on process exit
karma

yup, openQA caught the same crash.

BZ#1831086 softhsm use-after-free on process exit

Looks like my fix worked, tests are passing now.

@dvdhrm - additionally, you write in the commit message of this change as if it would not be backported to existing stable releases...but then you went ahead and submitted this update to F32. You also seem to have gotten everything one release cycle wrong in that commit message: we just released 32, not 33, and Rawhide at present is tracking 33, not 34.

I caught an error in the spec file that may have caused the problem. I've sent a fix and new builds are running now, I will edit this update with the new F32 build when it's done and see how the tests go.

karma

openQA is showing installer images built with this dbus failing to boot, and live image build failing when this dbus is included in the transaction. I'm running a meeting right now and then my power is going to be cut for a few hours, so I'm getting an initial negative karma in, will investigate in more detail later.

I think this update is a mistake. This build, webkit2gtk3-2.28.1-3.fc32, is from 2020-04-17. A newer build, webkit2gtk3-2.28.2-1.fc32, has already been submitted as an update and gone stable, 11 days ago.

I'm revoking this one.

Tracking the issue openQA hit at https://bugzilla.mozilla.org/show_bug.cgi?id=1635833 . Disabling the password manager seems to work around it, I have re-run the tests with that workaround and they all pass now.

karma

Looks to fix the bug from -1 in openQA testing.

There are multiple test failures in openQA for all three Firefox 76 updates (F30, F31, F32). It seems like, somehow, the update makes openQA much worse at typing passwords: all the failures are for password creation, where the two passwords that are typed don't match.

This only very very rarely goes wrong in other tests, somehow this build of Firefox is making it much worse, but I'm not sure how yet.

Figured out some more details of the problem in this bug report. Seems like pipewire is bodging an attempt to replace Pulseaudio. I think installed F32 systems may be OK on update (as they'll already have pulseaudio-libs-glib2 installed and the broken pipewire-libpulse package won't replace it on update), but the bug would likely affect F32 network installs if this update went stable, and would also affect the semi-official live respins.

It also seems like a fundamentally inappropriate thing to backport to F32, to me. pipewire shouldn't be attempting to replace pulseaudio after the fact in a stable release. This should be F33 only stuff, I think.

Booting an affected Rawhide image and examining it locally - https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200430.n.1/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20200430.n.1.iso - the logs show multiple occurrences of this error:

"Failed to load shared library 'libgvc.so' referenced by the typelib: libpulse-mainloop-glib.so.0: cannot open shared object file: No such file or directory"

this seems to break both gnome-shell and gsd-media-keys.

The bug is also affecting Rawhide today: https://openqa.fedoraproject.org/tests/590517

I re-ran the openQA test on an earlier update, just to check if some other package which went stable or some kickstarts change or something is the cause, but it worked, so that's a strong indication that this update really is the problem.

Since multiple people have reported serious problems with this, I've unpushed it to ensure no-one else hits those problems till we know what's going on.

This update has been unpushed.

karma

A couple of openQA tests seem to be failing consistently on this update. The tests that fail are tests of Workstation live images built with this update included: we boot the live image and try to install from it and boot the installed system, the two tests are the same but one is UEFI and one BIOS.

What seems to be happening is that there's no window manager running in the booted live session. We get the 'Welcome' window with no chrome, then when the test clicks on the button to launch anaconda, we get it in a window about a quarter the size of the screen, at top left, also with no chrome, like this:

https://openqa.stg.fedoraproject.org/tests/837735#step/_boot_to_anaconda/4

in the background you can see the 'GNOME didn't manage to start properly' screen. So this is clearly throwing off GNOME startup somehow.

I'll grab the live image that was built and throw it up somewhere for manual testing. @mcatanzaro for info