openQA testing looks good, including manually triggered universal tests. Note the iSCSI failure is normal when running the tests this way - it's because the update installer image build test produces a netinst image, but the iSCSI test only works with a DVD image (one that contains packages) as the client end of the iSCSI test doesn't have network access to anything but the server so the server end has to provide a package repo for it, but it can only do that by using the packages from the DVD image...it's just an unfortunate limitation in the tests I can't really fix easily. So it doesn't mean this update broke stuff. Unfortunately we'll only be able to get a 'legit' iSCSI test result by pushing this stable and seeing what happens in the next compose.
openQA tests look good, and I also hand-triggered the universal tests for the ISO built by the netinst test: all pass except iscsi, which is known to fail on netinst images, it's a test issue not a bug with this update. So this looks good, thanks.
I think this broke on new installs, in fact. GNOME Software update tests in openQA are failing consistently with an fwupd-related error since this landed in stable:
the system logs show fwupd.service failing to start:
Oct 01 04:39:34 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd: fwupd.service: Failed to set up mount namespacing: /run/systemd/unit-root/var/cache/fwupd: No such file or directory Oct 01 04:39:34 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd: fwupd.service: Failed at step NAMESPACE spawning /usr/libexec/fwupd/fwupd: No such file or directory Oct 01 04:39:34 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd: fwupd.service: Main process exited, code=exited, status=226/NAMESPACE Oct 01 04:39:34 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd: fwupd.service: Failed with result 'exit-code'.
We had an identical issue in iwd recently. The problem here is probably that the fwupd service file refers to
/var/cache/fwupd but the package does not create it. This will cause problems until it is created manually.
CI is failing on the crun dep: "nothing provides crun >= 0.10-1 needed by buildah-1.11.2-3.git0bafbfe.fc31.x86_64". That's because https://bodhi.fedoraproject.org/updates/FEDORA-2019-d2f808ac65 is not yet stable, though it's queued for it. We'll want to re-trigger the CI tests after that goes stable, somehow.