So, the previous update broke Windows boot for a large portion of our users because it spent mere 13 hours in testing, and this new update spent just 7 hours in testing and is again being pushed stable for everyone. This is just great. What if it breaks it even more?
@rharwood, would you please consider disabling "stable by karma" option for such critical packages as grub? Or at least increase the number to something like 10-15 instead of the default 3? We can't afford to push packages like grub so quickly, this is a recipe for a disaster. Thank you.
As for "what if it breaks even more?"... well, that's tricky, isn't it? It's a risk on all changes, sure, but the reverse is also true. Especially in this case, where there's a high-profile bug we want fixed: which means that the longer we leave it unfixed, the longer ostree is broken and we can't chainload windows right. To be specific, adding to the profile of this bug is that it is believed to block a release criteria and was proposed as both Prioritized and Common. Obviously some people care a lot about getting this fixed quickly.
I hear your concerns about not wanting to play fast and loose with critpath packages. So far, I'm not aware of problems that have stemmed from the standard +3/2 weeks policy, though if that does start causing issues we can of course revisit. Given most of my updates are going to time already, I don't see the value in raising to 10-15 as you suggest: this seems like it would guarantee they all go to time.
Let me clarify, I'm not saying grub updates should sit here for 2 weeks. I'm just saying they should sit here at least for some reasonable minimum amount of time (my personal choice would be 2-3 days at least). Or they can be pushed faster, if the maintainer is certain and the feedback is great, but it should be a conscious decision, i.e. a manual push. Windows boot was broken because there was automatic push enabled, and the first negative comment came just 1 hour too late. By disabling automatic push or raising the limits high enough, you can be sure it's not triggered in just a few hours and you can use your own judgement whether it's the right time to push it now or wait a day longer.
Also, regarding karma feedback, it's a little misleading. It's true that overall we don't have enough testers providing karma (your examples, however, were from when Fedora 36 was pre-release, we have even fewer of them at that times). But that doesn't mean they don't get tested. We have hundreds, maybe thousands, maybe more people running with updates-testing enabled. They just don't submit karma. But if something important breaks for them, you can be sure some of them will file a bug. So while it seems like the Bodhi feedback is quiet, there's a lot of value in waiting some time, because there will certainly be some feedback, if the update is broken. But not immediately, people are distributed all over the globe, and they don't update every day. In a few days, there will be some bug reports, if things break horribly.
One option I do want to draw your attention to in bodhi is "Require bugs", which "If checked, this will require that positive feedback be given on all associated bugs before the update can pass to stable. If your update has no associated bugs, this option has no effect.". We use this option (because it's default) - this means that the only reason the update advanced as quickly as it did was that someone asserted testing of the attached Windows bug. So the idea that Windows would somehow be more broken seems vanishingly unlikely to me.
Or they can be pushed faster, if the maintainer is certain and the feedback is great, but it should be a conscious decision, i.e. a manual push.
No, please do not inject more manual steps into our workflows. There are too many context switches already. If bodhi adds the ability to default to a minimum 24h in testing (for example), I would definitely consider enabling that. (I think it probably should be global, but now we're getting out of scope.)
In a few days, there will be some bug reports, if things break horribly.
And we get them, and fix them, as in this update :) Bugs will happen, no matter how much testing we throw at updates. That's just software.
Also, regarding karma feedback, it's a little misleading. It's true that overall we don't have enough testers providing karma (your examples, however, were from when Fedora 36 was pre-release, we have even fewer of them at that times).
I don't think so, because the same is true of released Fedoras. But the point is not to argue about specific updates: the point is that, as I think we both know, the amount of karma received is wildly variable and cannot be depended on. (You might recall the grub2 CVE update to post-release Fedora 35, where only you (thanks!) and one other person provided karma and it sat for the full 2 weeks.)
More generally, it seems to me that your issue is with the quality of feedback being provided by other testers on bodhi and the speed with which they provide it. The guidance for +1 is "Is the update generally functional?" - which I take to mean that the tester asserts that, according to their testing, they believe the package to function in an expected manner consistent with the release criteria. I am honestly not seeing a way it could mean anything else, nor can I vet the quality of testing based on the information they currently provide (and I don't think it's reasonable for me to do so).
After installing this grub2 version, running "grub2-install /dev/sda", and "grub2-mkconfig -o /boot/grub2/grub.conf" (on a BIOS machine), then installing a new kernel, the boot options "rhgb quiet" were used upon booting it even though I had removed them from /etc/default/grub. This only happens with the latest kernel installed after grub2, the older kernels omit "rhgb quiet" as expected. Anyone else seeing this?
Changing to -1. AWS cloud images are broken with this package installed. Reinstalling the kernel with this grub2 version renders EC2 instances unbootable (they drop to emergency mode).
Hi bojan, pbrobinson - I don't have enough information for that feedback to be actionable. Maybe you could open bugzillas, or mention which existing ones apply?
@rharwood "More generally, it seems to me that your issue is with the quality of feedback being provided by other testers on bodhi and the speed with which they provide it. The guidance for +1 is "Is the update generally functional?" - which I take to mean that the tester asserts that, according to their testing, they believe the package to function in an expected manner consistent with the release criteria. I am honestly not seeing a way it could mean anything else, nor can I vet the quality of testing based on the information they currently provide (and I don't think it's reasonable for me to do so)."
That's not really it, no. It's more about context. I don't dual boot with Windows or run cloud instances. So if I had tested one of the broken updates I could, with a clear conscience, have given a thumbs-up to "Is the update generally functional?", because for me it was. But the bugs were still there.
Unless a package is extremely trivial, no single tester is exercising every codepath in it. So you can get three quick +1s and yet still have a huge bug. This happened here. It has also happened to other packages before; it can happen to the kernel (if an update broke wireless networking entirely, but got three quick +1s from people who all use ethernet, and autopush was enabled, we'd be in trouble).
This is why @kparal is saying it's a good idea to disable autopush and wait a little for potential negative reports to appear.
One thing I can think of is we could put a delay on autopush. It could be set not to happen until 24 or 48 hours after the update hit updates-testing, and be cancelled if any negative karma appeared during that period. This wouldn't require maintainers to do anything manually in the 'success case'.
I hear what you're saying about quick +1s, but I don't get more than 3 karma basically ever - the fix for the update @kparal is lamenting as a huge problem to be avoided only got 4, so...
This update has been submitted for testing by rharwood.
This update's test gating status has been changed to 'waiting'.
rharwood edited this update.
rharwood edited this update.
New build(s):
Karma has been reset.
This update's test gating status has been changed to 'failed'.
This update's test gating status has been changed to 'waiting'.
This update's test gating status has been changed to 'passed'.
This update has been pushed to testing.
Works.
Works.
This update can be pushed to stable now if the maintainer wishes
This update has been submitted for stable by bodhi.
So, the previous update broke Windows boot for a large portion of our users because it spent mere 13 hours in testing, and this new update spent just 7 hours in testing and is again being pushed stable for everyone. This is just great. What if it breaks it even more?
@rharwood, would you please consider disabling "stable by karma" option for such critical packages as grub? Or at least increase the number to something like 10-15 instead of the default 3? We can't afford to push packages like grub so quickly, this is a recipe for a disaster. Thank you.
Works. Both Fedora 36 and Windows 11 boot fine.
Works
No, not in the general case, but I will explain my thinking. Bodhi is also used for security updates, where, since Fedora has no embargoed build process, speed is also a concern. Generally bodhi updates sit for much longer: frankly, I think having a bug that people cared about caused this one to receive more attention during testing. For instance, see this previous Fedora 36 grub update that went stable due to time not karma, or this earlier grub2 update for Fedora 36 that went stable after 2 weeks and no karma at all, etc., etc.. The list goes on, but point is that it's normal to receive little if any karma, even on the bootloader.
As for "what if it breaks even more?"... well, that's tricky, isn't it? It's a risk on all changes, sure, but the reverse is also true. Especially in this case, where there's a high-profile bug we want fixed: which means that the longer we leave it unfixed, the longer ostree is broken and we can't chainload windows right. To be specific, adding to the profile of this bug is that it is believed to block a release criteria and was proposed as both Prioritized and Common. Obviously some people care a lot about getting this fixed quickly.
I hear your concerns about not wanting to play fast and loose with critpath packages. So far, I'm not aware of problems that have stemmed from the standard +3/2 weeks policy, though if that does start causing issues we can of course revisit. Given most of my updates are going to time already, I don't see the value in raising to 10-15 as you suggest: this seems like it would guarantee they all go to time.
works
Let me clarify, I'm not saying grub updates should sit here for 2 weeks. I'm just saying they should sit here at least for some reasonable minimum amount of time (my personal choice would be 2-3 days at least). Or they can be pushed faster, if the maintainer is certain and the feedback is great, but it should be a conscious decision, i.e. a manual push. Windows boot was broken because there was automatic push enabled, and the first negative comment came just 1 hour too late. By disabling automatic push or raising the limits high enough, you can be sure it's not triggered in just a few hours and you can use your own judgement whether it's the right time to push it now or wait a day longer.
Also, regarding karma feedback, it's a little misleading. It's true that overall we don't have enough testers providing karma (your examples, however, were from when Fedora 36 was pre-release, we have even fewer of them at that times). But that doesn't mean they don't get tested. We have hundreds, maybe thousands, maybe more people running with updates-testing enabled. They just don't submit karma. But if something important breaks for them, you can be sure some of them will file a bug. So while it seems like the Bodhi feedback is quiet, there's a lot of value in waiting some time, because there will certainly be some feedback, if the update is broken. But not immediately, people are distributed all over the globe, and they don't update every day. In a few days, there will be some bug reports, if things break horribly.
works
works
One option I do want to draw your attention to in bodhi is "Require bugs", which "If checked, this will require that positive feedback be given on all associated bugs before the update can pass to stable. If your update has no associated bugs, this option has no effect.". We use this option (because it's default) - this means that the only reason the update advanced as quickly as it did was that someone asserted testing of the attached Windows bug. So the idea that Windows would somehow be more broken seems vanishingly unlikely to me.
No, please do not inject more manual steps into our workflows. There are too many context switches already. If bodhi adds the ability to default to a minimum 24h in testing (for example), I would definitely consider enabling that. (I think it probably should be global, but now we're getting out of scope.)
And we get them, and fix them, as in this update :) Bugs will happen, no matter how much testing we throw at updates. That's just software.
I don't think so, because the same is true of released Fedoras. But the point is not to argue about specific updates: the point is that, as I think we both know, the amount of karma received is wildly variable and cannot be depended on. (You might recall the grub2 CVE update to post-release Fedora 35, where only you (thanks!) and one other person provided karma and it sat for the full 2 weeks.)
More generally, it seems to me that your issue is with the quality of feedback being provided by other testers on bodhi and the speed with which they provide it. The guidance for +1 is "Is the update generally functional?" - which I take to mean that the tester asserts that, according to their testing, they believe the package to function in an expected manner consistent with the release criteria. I am honestly not seeing a way it could mean anything else, nor can I vet the quality of testing based on the information they currently provide (and I don't think it's reasonable for me to do so).
Works
no regressions noted
Resolves Windows 11 not booting with GRUB2 on two Dell laptops for me.
This update has been pushed to stable.
@fundies , can you verify that your use case is still working with this update?
After installing this grub2 version, running "grub2-install /dev/sda", and "grub2-mkconfig -o /boot/grub2/grub.conf" (on a BIOS machine), then installing a new kernel, the boot options "rhgb quiet" were used upon booting it even though I had removed them from /etc/default/grub. This only happens with the latest kernel installed after grub2, the older kernels omit "rhgb quiet" as expected. Anyone else seeing this?
Posted comment in #2113883.
https://bugzilla.redhat.com/show_bug.cgi?id=2118287
Changing to -1. AWS cloud images are broken with this package installed. Reinstalling the kernel with this grub2 version renders EC2 instances unbootable (they drop to emergency mode).
Broke ARMv7 too
Hi bojan, pbrobinson - I don't have enough information for that feedback to be actionable. Maybe you could open bugzillas, or mention which existing ones apply?
I already commented in relevant bug, which is linked here.
FEDORA-2022-a3480ad0d3 new update attempting to address these problems.
@rharwood "More generally, it seems to me that your issue is with the quality of feedback being provided by other testers on bodhi and the speed with which they provide it. The guidance for +1 is "Is the update generally functional?" - which I take to mean that the tester asserts that, according to their testing, they believe the package to function in an expected manner consistent with the release criteria. I am honestly not seeing a way it could mean anything else, nor can I vet the quality of testing based on the information they currently provide (and I don't think it's reasonable for me to do so)."
That's not really it, no. It's more about context. I don't dual boot with Windows or run cloud instances. So if I had tested one of the broken updates I could, with a clear conscience, have given a thumbs-up to "Is the update generally functional?", because for me it was. But the bugs were still there.
Unless a package is extremely trivial, no single tester is exercising every codepath in it. So you can get three quick +1s and yet still have a huge bug. This happened here. It has also happened to other packages before; it can happen to the kernel (if an update broke wireless networking entirely, but got three quick +1s from people who all use ethernet, and autopush was enabled, we'd be in trouble).
This is why @kparal is saying it's a good idea to disable autopush and wait a little for potential negative reports to appear.
One thing I can think of is we could put a delay on autopush. It could be set not to happen until 24 or 48 hours after the update hit updates-testing, and be cancelled if any negative karma appeared during that period. This wouldn't require maintainers to do anything manually in the 'success case'.
Yeah, delay on autopush is what I was imagining.
I hear what you're saying about quick +1s, but I don't get more than 3 karma basically ever - the fix for the update @kparal is lamenting as a huge problem to be avoided only got 4, so...