Provide the option to panic upon the kernel becoming tainted

[raja-grewal]

This pull request provides the option to panic upon the kernel becoming tainted.

After first testing, this can be used to enforce strict user-defined kernel operation and security at runtime. The exact subset selected is up to the user but in the presented default I have included panicking upon using out of specification hardware, bad page states, severe firmware bugs, and kernel live patching.

Given this is a comment-only PR, I do not see anything controversial with this as all it does is tell users that such a feature is available if they are first willing to test there are no problems prior to them being used as hardening features.

Please see references inside the commits and also the kernel docs.

Changes

There are no changes to the functionality of the codebase.

Provide the disabled by default option:

panic_on_taint=0x8824

Mandatory Checklist

• Legal agreements accepted. By contributing to this organisation, you acknowledge you have read, understood, and agree to be bound by these these agreements:

Terms of Service, Privacy Policy, Cookie Policy, E-Sign Consent, DMCA, Imprint

Optional Checklist

The following items are optional but might be requested in certain cases.

• I have tested it locally
• I have reviewed and updated any documentation if relevant
• I am providing new code and test(s) for it

[raja-grewal]

Note this also connects to #328. Using panic_on_taint we can also extend the panicking to also include loading of proprietary, out-of-tree, or unsigned modules which will then cause an immediate reboot.

[ArrayBolt3]

This unfortunately conflicts with tirdad, which uses kernel live patching to replace some TCP ISN functions with more secure alternatives. This means enabling this option on a default installation of Kicksecure will result in an immediate panic on next boot.

Looking at all available options:

• Proprietary module was loaded; this one is scary for obvious reasons. We can leave it in a comment but not in the recommended option itself. (This is what's already done in this PR.)
• Module was force loaded; maybe we should recommend the user enable this one? Force-loading modules could cause a dangerous condition, and I wouldn't be surprised if someone did this to their system without knowing any better.
• Kernel is running on an out of specification system; out-of-specification is well defined by one of the documents you linked (https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html), the definition seems sane to me. Recommending this the user enable this seems like a good idea, maybe we should even enable it by default.
• Module was force unloaded; this one is probably also good to recommend. People shouldn't do that.
• Processor reported an MCE; this one is probably good to recommend. People who use ECC memory might want to turn it off since correctable ECC errors may generate an MCE, but those users are probably rare and errors even in ECC memory are probably rare too.
• Bad page referenced or some unexpected page flags we should definitely recommend that the user enable this. This looks useful enough I'd say maybe we should enable it by default.
• Taint requested by userspace application; I guess we can recommend this? Applications probably won't use this unless there's a good reason to, and according to https://stackoverflow.com/questions/69319348/how-do-i-programmatically-taint-the-linux-kernel it seems like it probably something only the root user can do.
• Kernel died recently (OOPS or BUG); this is another one we might want to enable by default rather than just recommending. It's essentially another form of panic_on_oops but maybe better, at the very least defense-in-depth.
• ACPI table overridden by user; this one we also might want to consider enabling by default. There is a usecase for overriding ACPI tables manually, but it's very niche, and from what I understand ACPI code is very highly privileged and powerful.
• Kernel issued warning; this is another one we might enable by default. Another variant of panic_on_warn essentially.
• Staging driver was loaded; this one is kind of weird. Does Debian ship any staging drivers in the default kernel? If no, I'd say we should consider enabling this by default, but if so, we probably don't want to even recommend this to the user.
• Workaround for bug in platform firmware applied; this one is good to recommend but probably not to enable by default. Hardware with firmware bugs is pretty common sadly.
• Externally-built module was loaded: document in a comment, but don't recommend or enable by default, as it conflicts with tirdad.
• Unsigned module was loaded; same as "Externally-built module was loaded".
• Soft lockup occurred; probably document but don't recommend, since VirtualBox on Windows with Hyper-V enabled may be slow enough to result in benign kernel soft lockups.
• Kernel has been live-patched; document but don't recommend or enable, as it conflicts with tirdad.
• Auxiliary taint; this one we might want to recommend. In Debian's kernel, it kicks in when one of two things happen (according to https://codesearch.debian.net/search?q=TAINT_AUX&literal=1):
  • fanotify with permission checks are used (apparently this can allow an application to deadlock the rest of the system). This is mainly done by antivirus software according to the patch itself. There's an example of how to use the API in the fanotify manpage. This seems like something someone might legitimately want to use.
  • BTRFS RAID5/6 is used. This is also something people have legitimate use cases for.
• Kernel was built with the struct randomization plugin; I personally wouldn't even document this. Struct randomization is a pretty neat security feature that unfortunately requires the user to compile a kernel for it to be useful. Debian's kernel isn't built with struct randomization enabled (and it would be pointless to do so). Users who use this feature should not have the kernel panic because of it, and most users won't use this feature.
• An in-kernel test has been run; we might want to enable this by default. I don't think anyone should be running a kernel in production that ran through (part of) a unit test, since unit tests might cause all sorts of fun havoc.
• Userspace used a mutating debug operation in fwctl; this one I'd say we should enable by default. I wouldn't expect most applications to ever do this unless they were being used for hardware debugging (in which case the user should install debug-misc), but I can imagine malware using it.

Thoughts on the above?

[raja-grewal]

Great overview of potential options and yes you are definitely correct about kernel live patching.

The important thing is I am currently only recommending this only as an optional since every user needs to test whether the settings are compatible with their existing setup. Thus for the time being I am hoping to have this included and then at a later stage begin enabling things by default.

That aside, before going through your comments there is one thing that I think is a good idea. For the same reason we do not currently double enable panic on oopses and warnings, I do not think doing it with this setting should be done other. This will make disabling and debugging even more difficult since here we are dealing with a bitmask

• Proprietary module was loaded; agreed should be optional (already done).
• Module was force loaded; agreed should be optional.
• Kernel is running on an out of specification system; agreed should be by default (already done).
• Module was force unloaded; agreed should be optional.
• Processor reported an MCE; agreed should be optional.
• Bad page referenced or some unexpected page flags; agreed should be by default (already done).
• Taint requested by userspace application; agreed should be optional.
• Kernel died recently (OOPS or BUG); disagree as already done elsewhere.
• ACPI table overridden by user; agreed should be optional but not sure if be by default.
• Kernel issued warning; disagree as already done elsewhere.
• Staging driver was loaded; agreed should be optional.
• Workaround for bug in platform firmware applied; disagree and should be by default (already done) as the specification is a severe bug, not something more manageable.
• Externally-built module was loaded; agreed should be optional (already done).
• Unsigned module was loaded; agreed should be optional (already done).
• Soft lockup occurred; agreed should be optional.
• Kernel has been live-patched; agreed should not be allowed (will be corrected).
• Auxiliary taint; agreed should be optional.
• Kernel was built with the struct randomization plugin; agreed should be optional.
• An in-kernel test has been run; agreed should be by default (will be done).
• Userspace used a mutating debug operation in fwctl; agreed should be by default (will be done).

Let me me know what you think.

[ArrayBolt3]

For the same reason we do not currently double enable panic on oopses and warnings, I do not think doing it with this setting should be done other. This will make disabling and debugging even more difficult since here we are dealing with a bitmask

Good point, agreed.

• Workaround for bug in platform firmware applied; disagree and should be by default (already done) as the specification is a severe bug, not something more manageable.

Checked the kernel code, I agree with you now. I thought this would cause the system to panic any time the kernel did something that warranted a [Firmware Bug] message, but after some grepping I discovered that FW_BUG messages are oftentimes set without enabling TAINT_FIRMWARE_WORKAROUND, and that TAINT_FIRMWARE_WORKAROUND is used in some instances that look pretty bad to me (i.e. EFI calls corrupting IRQ flags).

Unsigned module was loaded; agreed should be optional (already done).

This should be a "should not be allowed" for the same reason that panic on live-patch-induced taint should not be allowed, IMO.

ACPI table overridden by user; agreed should be optional but not sure if be by default.

At least on my laptop (a Clevo machine), ACPI code can be used to write directly to devices like the embedded controller, which could in theory allow one to overwrite arbitrary memory or even brick (some parts of) the hardware. I suspect most laptops are like this since the ACPI tables contain what are effectively driver components for some (many?) devices. If we're concerned about the use of MSRs to write arbitrary memory, I think disallowing custom ACPI code would also be a good idea.

[raja-grewal]

This should be a "should not be allowed" for the same reason that panic on live-patch-induced taint should not be allowed, IMO.

Agreed, my mistake. It has been updated.

I think disallowing custom ACPI code would also be a good idea.

Yes that makes sense, agreed. It has been updated.

[ArrayBolt3]

Merged into my arraybolt3/trixie branch. I did make one change in the process that was non-trivial:

-## After confirming stability this enforces strict user-defined kernel operation and security at runtime.
+## After confirming stability this reduces attack surface.

This was to avoid users misunderstanding, and thinking that panic_on_taint enabled some sort of "formally verified" mode or similar ("enforces string user-defined kernel operation" was a bit stronger than I thought was accurate). Let me know if that's OK.

[raja-grewal]

Yep I agree that your rewording is better so as not to give the presentation of a something more formal. Looks good to me!