openQA showed issues on the previous update but not this one, interestingly. All the openQA tests passed for this update on both stg and prod. So it seems like, unless we got very 'lucky', the problem was fixed in openQA's environment in 72.0.1, but broken in some other environments instead...
This seems to be failing openQA tests (whereas the F31 update worked fine). Several of the tests that use a browser - both the desktop_browser tests for KDE and GNOME, and the test which enrols to FreeIPA via Cockpit then tests the FreeIPA webUI - had issues. It seems that in each case, navigating to a page just sometimes apparently failed, showing an entirely grey pane, like this. The browser is not still trying to contact the server at that point, it thinks the operation has completed and does not show any error, but it also doesn't show any page.
On the desktop_browser tests, openQA types the URL into a running Firefox and hits enter; on the FreeIPA/Cockpit test, it seems to have failed in the same way when launching Firefox with the URL as an argument (
startx /usr/bin/firefox -width 1024 -height 768 https://ipa001.domain.local).
I see these failures in both staging and production, and F31 passed in both staging and production, so it doesn't seem like a blip, it seems like some kind of genuine issue.
There have been some ongoing issues with Koji and Bodhi lately which are causing issues with updates actually being tagged correctly and making it to testing, so you'll probably want to keep an eye on it manually to make sure it actually gets out. If it doesn't, contact whoever's on call for releng...
As part of the 5.4 rebase, this update backported a bug from master without backporting the fix: it's missing https://src.fedoraproject.org/rpms/kernel/c/0b30cc5df58af3a1385f218163d33634b51ed67f , so its
kernel-modules-extra subpackage is 7x as big as it should be because it includes the kernel header files. This is the root cause of the openqa test failure - took me a bit of digging to figure it out, but that's why :)
zdzichu points out this commit as the likely suspect, and indeed I think he's right. That commit missed changing at least one place to catch a
dnsutil.DNSZoneAlreadyExists exception instead of a
ValueError: the bit of
ipaserver/install/dns.py where it checks the reverse zone. Here's the relevant bit of the traceback:
File "/usr/lib/python3.7/site-packages/ipaserver/install/server/__init__.py", line 562, in main master_install_check(self) File "/usr/lib/python3.7/site-packages/ipaserver/install/server/install.py", line 276, in decorated func(installer) File "/usr/lib/python3.7/site-packages/ipaserver/install/server/install.py", line 673, in install_check dns.install_check(False, api, False, options, host_name) File "/usr/lib/python3.7/site-packages/ipaserver/install/dns.py", line 145, in install_check dnsutil.check_zone_overlap(reverse_zone) File "/usr/lib/python3.7/site-packages/ipapython/dnsutil.py", line 383, in check_zone_overlap raise DNSZoneAlreadyExists(zone=zone.to_text(), ns=ns)
the block with the
dnsutil.check_zone_overlap(reverse_zone) call isn't touched by that commit, but obviously should have been.
I think this update broke the
--allow-zone-overlap option to
We have a little problem with the openQA FreeIPA test environment, which is that the IPs we use for the VMs that run the tests actually overlap with RH's internal DNS zones. So if we just use
ipa-server-install to deploy our server we get an error "DNS zone 2.0.10.in-addr.arpa. already exists in DNS and is handled by server(s): blahblahblah". Fortunately,
ipa-server-install has the
--allow-zone-overlap option which is supposed to accept this situation and go ahead anyway, so the tests use that. But the tests for this update failed with that error even though the option was passed, which suggests the option is broken...
openQA test failure is telling us this can't be installed (there's a check that the packages in the update are actually used when they should be, this is a failure of that check). This is because the update currently contains
389-ds-base-220.127.116.11-1.fc31 which has a typo in its dependencies. @mreynolds already noticed this and sent a -2 build that should fix it, presumably he will edit that into this update once it's done. I'll -1 for now so it doesn't go stable inadvertently.