Repository navigation
Redundant kernel args #199
Description
Activity
Saving space in kernel command line, reducing command line options, avoiding redundancy would be good.
We might comment these options out and document that inside the settings file instead out deleting these to avoid having this topic come up again.
Waiting a few days before I look into this and welcoming other comments or review.
Yes this can be done if desired. Note this situation was discussed earlier in Issue #177 (comment).
The primary reason the redundant kernel parameters were kept in place was for potential future proofing. If the default
mitigations=auto,nosmtsettings were to change and become more lenient in the future, our hardcoded settings would remain fixed and be applied regardless.However, the concern regarding limited number of characters for kernel arguments was not considered by me. So, if this is a genuine concern, we can easily remove the parameters. Perhaps instead of removing the lines entirely just comment them out so users can verify the correct mitigations are in place if required.
Should I add a note explicitly saying that these are redundant?
Yes. That would be nice so anyone else looking into this knows it and no need to rehash this discussion. Always good that way.
As a rough example, one way to highlight a downside of not using fine-grained settings can be shown with the
l1tfmitigation.Upon closer inspection, one potential limitation regarding solely using
mitigations=auto,nosmtexists if did not to also manually includel1d_flush=on.If you look closely at the kernel parameters:
mitigations= [X86,PPC,S390,ARM64] Control optional mitigations for CPU vulnerabilities. This is a set of curated, arch-independent options, each of which is an aggregation of existing arch-specific options. auto,nosmt Mitigate all CPU vulnerabilities, disabling SMT if needed. This is for users who always want to be fully mitigated, even if it means losing SMT. Equivalent to: l1tf=flush,nosmt [X86] mds=full,nosmt [X86] tsx_async_abort=full,nosmt [X86] mmio_stale_data=full,nosmt [X86] retbleed=auto,nosmt [X86]Now if you focus on the L1 Terminal Fault (L1TF) mitigation parameter:
l1tf= [X86] Control mitigation of the L1TF vulnerability on affected CPUs The kernel PTE inversion protection is unconditionally enabled and cannot be disabled. full Provides all available mitigations for the L1TF vulnerability. Disables SMT and enables all mitigations in the hypervisors, i.e. unconditional L1D flush. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled. full,force Same as 'full', but disables SMT and L1D flush runtime control. Implies the 'nosmt=force' command line option. (i.e. sysfs control of SMT is disabled.) flush Leaves SMT enabled and enables the default hypervisor mitigation, i.e. conditional L1D flush. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled. flush,nosmt Disables SMT and enables the default hypervisor mitigation. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled. flush,nowarn Same as 'flush', but hypervisors will not warn when a VM is started in a potentially insecure configuration. off Disables hypervisor mitigations and doesn't emit any warnings. It also drops the swap size and available RAM limit restriction on both hypervisor and bare metal. Default is 'flush'.Notice how the more 'secure' option is using
l1tf=full,forcerather thanl1tf=flush,nosmt. If were to simply usemitigations=auto,nosmtand not simultaneously manually applyl1d_flush=onit would be problematic. Why the latter is not also included inmitigations=auto,nosmtI am not sure.Overall, unless we are in actual need of additional characters, I think for the time being we should leave the kernel arguments as they are without modification. Maybe consider closing PR #200 and potentially reopening it at future date if required.
Well the current default is
l1d_flush=onand I am not removing it anyways. I am not advocating for removing everything and only keepingmitigations=auto,nosmt, I am advocating for removing the ones that are known to be redundant.Personally, I think outsourcing responsibility to
mitigations=auto,nosmtis likely not the best option at this point in time.You still have not addressed this:
If the default
mitigations=auto,nosmtsettings were to change and become more lenient in the future, our hardcoded settings would remain fixed and be applied regardless.One could make a strong case that mitigating against CPU vulnerabilities is one of the most important features of a 'hardening' project.
Currently, we have the best of both worlds,
mitigations=auto,nosmtenforces many mitigations and potential future mitigations that maintainers of this repository might miss. This is then combined with our hardcoded settings that ensure strict compliance to the highest known standards which in turn makes us resilient to any potential future weakening ofmitigations=auto,nosmt.Unless we are currently in need for more characters for kernel arguments, I do not see opening up this potential exposure is necessary?
If the default mitigations=auto,nosmt settings were to change and become more lenient in the future, our hardcoded settings would remain fixed and be applied regardless.
Well on this point, we are supposed to check what the kernel does as time goes on anyways.
Unless we are currently in need for more characters for kernel arguments, I do not see opening up this potential exposure is necessary?
Another point is that it makes management easier. Having a gigantic list of kargs does make it more painful to deal with when you need to do debugging and stuff.
I did have a quick talk with Madaidan yesterday and he just said he personally isn't sure if the whole hard coding kargs matters that much.
Well on this point, we are supposed to check what the kernel does as time goes on anyways.
Personally, I am not a fan of putting additional burden on volunteer contributors when it can be avoided. I would much rather implement a through general solution that is robust moving forward in time, as opposed to a very acceptable solution that requires somewhat constant vigilance.
Another point is that it makes management easier. Having a gigantic list of kargs does make it more painful to deal with when you need to do debugging and stuff.
I overall very strongly agree with everything you stated in this point. In fact, I don’t think it is possible for anyone to actually disagree with this point.
Overall, in light of all of the above discussion, I will leave it to @adrelanos to decide what approach he considers superior.
Useful would be a feature request against the kernel:
mitigations=strictshould always enable the most secure kernel mitigation settings.If such a feature were to exist, then no such work downstream in security-misc would be needed.
Lower maintenance effort is preferable.
Should I add a note explicitly saying that these are redundant?
This for sure is actionable. I wouldn't mind more comments saying "this options may be superfluous because...". Or 1 longer comment explaining this and then just a tag "## potentially-superfluous".
Then as a follow-up step the potentially superfluous kernel parameters could even be moved to a separate file. Thereby making it easier for those concerned or impacted by the length of the kernel command line to easily delete these.
- Reacted by Patrick Schleizer
Hi everyone just a quick update on this issue that I thought would be useful given I increasingly noticed many other Linux distributions heavily rely on
mitigations=auto,nosmtfor many implementing CPU mitigations.Firstly, let us take a look at what
mitigations=autoactually does using the kernel docs where it essentially states that using this parameter is "Equivalent to: (default behavior)". Now if we enhance this to also disable SMT usingmitigations=auto,nosmtwhat we have a some of further tightening to five mitigations.Now what I could be noticing incorrectly but to seems to be the definition is that in most cases this still applies the "default" mitigations for most vulnerabilities. The question is that whether all the "defaults" are the maximum amount of hardening possible.
It is again my opinion that they are not in many cases and we can perform further hardening by using more fine-grained controls as we have done in our configs which appear to be working without any major issues for quite sometime.
More recently, I have spotted even more discrepancies mainly using the kernel docs, below I have complied a list of the "default" using
mitigations=auto,nosmt(top) and compared them to the most strict settings (bottom).Spectre Variant 2:
spectre_v2=auto
spectre_v2=onSpectre Variant 3 (Meltdown):
(on)
kpti=1Spectre Variant 4 (Speculative Store Bypass) [1]:
spec_store_bypass_disable=autossbd=kernel
spec_store_bypass_disable=onssbd=force-onL1TF - L1 Terminal Fault:
l1tf=flush,nosmtkvm-intel.vmentry_l1d_flush=cond
l1tf=full,forcekvm-intel.vmentry_l1d_flush=alwaysiTLB Multihit:
kvm.nx_huge_pages=auto
kvm.nx_huge_pages=forceL1D Flushing:
(off)
l1d_flush=onCross-Thread Return Address Predictions:
(off)
kvm.mitigate_smt_rsb=1GDS - Gather Data Sampling:
(on)
gather_data_sampling=forceIndirect Target Selection (ITS):
indirect_target_selection=on
indirect_target_selection=forceVMScape:
vmscape=ibpb
vmscape=forceObviously some of these defaults on top using just
mitigations=auto,nosmtare not in line with the strictest possible settings shown below.Therefore, while once again I understand ease of using a single kernel boot parameter, it's sole use would again not be in line with the projects objective of maxmising hardening by default.
Or have I made some sort of error and have we been doing it wrong in this whole time?
Let me know what you think!
Edit:
As per #333, the mitigation for Meltdown for ARM64 CPUs is also not applied to the maximum setting on all cores.Is there a way to actually test the effect of using
mitigations=auto,nosmtversus not usingmitigations=auto,nosmt?Idea:
'''1.''' Copy the status of vulnerabilities folder while using
mitigations=auto,nosmt.cp -r /sys/devices/system/cpu/vulnerabilities ~/with'''2.''' Disable use of
mitigations=auto,nosmt.(Note to other readers: Don't forget
sudo update-grub,rebootand confirm if the kernel parameter change has been actually applied by usingcat /proc/cmdline.)'''3.''' Make another copy.
cp -r /sys/devices/system/cpu/vulnerabilities ~/without'''4.''' Compare the vulnerabilities folder.
meld ~/with ~/withoutUsing that method, can you find any effect?
As long as the last kernel parameter wins, keeping
mitigations=auto,nosmtat the top might be best?Would be useful to point that out explicitly so we don't move
mitigations=auto,nosmtfurther down in the future.related:
https://unix.stackexchange.com/questions/544224/evaluation-order-of-duplicated-kernel-parametersIs there a way to actually test the effect of using
mitigations=auto,nosmtversus not usingmitigations=auto,nosmt?Idea:
Regrading this experiment it is very important to note that what you are actually testing is using
mitigations=auto,nosmtand then by it's removal this against the kernel default ofmitigations=auto.These two are obviously not equivalent since SMT will be re-enabled again and so the relevant mitigations will be less strict.
Another problem with this approach relevant to me is that I do not have enough CPUs to cover the range of all the mitigations and so am unable to comprehensively find out the consequences of such a test thoroughly.
Therefore, while it is good idea, we are not actually comparing apples-to-apples and even if that were so I do not have enough CPUs to make the test valid myself.
As long as the last kernel parameter wins, keeping mitigations=auto,nosmt at the top might be best?
Regarding keeping the parameter at the top, I also think this is a good idea given how ubiquitous it's use is across so many projects. However, in the PR I have suggested to comment it out since it's use is well and truly redundant and so there will be no confusion as to the fact it is only still present for illustration of what not to use when others are browsing the configuration.
I however soon make a comment warning about how it should not be enabled and especially not while also being moved down the list.
These two are obviously not equivalent since SMT will be re-enabled again and so the relevant mitigations will be less strict.
Well, almost, because all of the other parameters will still be enabled, and theoretically one of those will disable SMT, right?
Regardless, I do agree this is probably quite tricky to test perfectly, since the "automatic" mitigations will depend on what CPU you're using and thus it will be unclear if a particular mitigation is enabled because our settings enable it, or because the kernel enables it. What would be more interesting is testing
mitigations=offvsmitigations=auto,nosmtwith all our other options set, since (presumably) that will mean only mitigations we explicitly enable will be enabled, and thus we'll get to see if we missed anything. (https://docs.kernel.org/admin-guide/hw-vuln/index.html is probably a useful reference to make sure we got everything, we're probably using that already but figured I'd mention it just in case.)I do not agree that we should disable
mitigations=auto,nosmtthough. It is probably dangerous to move down the list, but without it, if a new vulnerability comes out and the kernel adds a mitigation for it, we might not get it if we miss it and don't usemitigations=auto,nosmt. If we do use it, we'll probably get that mitigation, even if we could (and in the future should) set a stronger mitigation on it. I don't think it's too likely we and everyone else will miss a new CPU vulnerability showing up, but if we can harden the system against our own possible ignorance, we should.
From the standpoint of how much longer our command line can be...
The kernel parameters documentation for Linux states:
The number of kernel parameters is not limited, but the length of the complete command line (parameters including spaces etc.) is limited to a fixed number of characters. This limit depends on the architecture and is between 256 and 4096 characters. It is defined in the file ./include/uapi/asm-generic/setup.h as COMMAND_LINE_SIZE.
Based on experimentation and studying the Linux headers, the maximum command line length for amd64 systems is 2048 bytes, not a full 4096 as I was hoping. (There was conflicting information online, and it was unclear if the 2048 byte limit only applied to 32-bit systems or both 32-bit and 64-bit systems when checking the headers.) On a system installed from a Kicksecure 18 ISO with disk encryption enabled, our current kernel command line is 1110 bytes long, thus we've used a little over half of the available space. Getting rid of
mitigations=auto,nosmtwould save 23 bytes, not negligible, but not worthwhile at the moment IMO. I'd say we should keep this at the start of our list of mitigations.(Additional fun fact, the bit about the number of kernel parameters not being limited only applies to parameters the kernel actually understands. For everything else that gets passed to init, there's a limit of 32 parameters, at least on Debian. Learned this the hard way, took me about half an hour to figure out why the kernel was panicking when my command line wasn't anywhere close to 2048 bytes long :P)
Well, almost, because all of the other parameters will still be enabled, and theoretically one of those will disable SMT, right?
I am assuming in this experiment that they are all off and we are just using
mitigations=autoversusmitigations=auto,nosmtand nothing else.I do not agree that we should disable mitigations=auto,nosmt though.
Note that
mitigations=autois the kernel default and so is redundant, this then combined with our usage ofnosmt=forceis then identical to settingmitigations=auto,nosmt.There is no future mitigation at the default level we are going to miss with this approach. I am not really proposing to commenting it out it for purposes of saving characters, rather, on the simple principle that it serves no purpose.
https://lore.kernel.org/lkml/20251013143444.3999-1-david.kaplan@amd.com/
https://www.phoronix.com/news/Linux-Dynamic-MitigationsImportantly, note that a very recent RFC that has just a few days ago proposed to massively shake up on how CPU vulnerability mitigations work at the kernel level by making them able to be toggled on and off without reboot.
This is means than an elevation of privileged to root or even sudo can disable all of them. This would be akin to the functionality of
kexecwhich can replace the running kernel with another, something which we obviously disable.Personally, I disagree with this proposal but at the same time the understand argument that if root is achieved there is little point. Regardless, it will be very interesting to see how this conversation develop moving forward.
If this goes ahead and is merged, I am curios if this functionality will become possible by default or alternatively will (in my opinion) more sensibly require a compile time flag for those running a kernel that actually requires such "dynamic" use of mitigation.
Personally, I disagree with this proposal but at the same time the understand argument that if root is achieved there is little point.
For systems that don't use Secure Boot or kernel lockdown, sure. For systems that do... this could be a problem since lockdown mode doesn't allow even root the ability to read arbitrary kernel memory AFAIK (
/proc/kcoredoesn't exist), but if root can just do the equivalent ofmitigations=off, then it can leak whatever it wants via a side-channel attack. New CPUs are produced today with vulnerabilities that are only avoided by kernel mitigations, so this is a real threat in those scenarios.I wonder if anyone else has brought that up already, or if someone should bring that up...
Edit: Looks like no one brought that up, so I sent an email into that thread mentioning lockdown mode and suggesting that a mechanism be added to prevent further mitigations changes once the proper mitigations are set up.
Edit 2: Link to email: https://lore.kernel.org/lkml/20251014231039.6d23008f@kf-m2g5/
Reacted by raja-grewal and Patrick SchleizerThere is no future mitigation at the default level we are going to miss with this approach. I am not really proposing to commenting it out it for purposes of saving characters, rather, on the simple principle that it serves no purpose.
@raja-grewal Good point, I didn't notice that. In that instance removing
mitigations=auto,nosmtsounds like a good idea.Reacted by raja-grewalNote that
mitigations=autois the kernel default and so is redundant, this then combined with our usage ofnosmt=forceis then identical to settingmitigations=auto,nosmt.Are we sure that is is really the same? That's hard or impossible to know for sure without looking at the source code.
This argument should therefore still be applicable:
I do not agree that we should disable mitigations=auto,nosmt though. It is probably dangerous to move down the list, but without it, if a new vulnerability comes out and the kernel adds a mitigation for it, we might not get it if we miss it and don't use mitigations=auto,nosmt. If we do use it, we'll probably get that mitigation, even if we could (and in the future should) set a stronger mitigation on it. I don't think it's too likely we and everyone else will miss a new CPU vulnerability showing up, but if we can harden the system against our own possible ignorance, we should.
Another argument is that
mitigations=auto,nosmtis probably more widely in use and tested as it's the maximum security setting that one would set based on reading the kernel manual.In doubt, wasting a bit kernel command line space using
mitigations=auto,nosmtseems safer than not using it. This solution will be good enough until the kernel completely revises how how to enable maximum mitigation more easily?Apologies for the delay.
You make a good points that in terms of auditing that people would more likely expect this parameter to be present which is also why I wanted to highlight the it's used was flawed in the first place. Regardless, you are correct that until the kernel completely revises how to enable maximum mitigations more clearly we are left in the current scenario.
We can either comment it out as it does not functionally do anything other than in my personal opinion basically spread quite corrosive misinformation. Or as you suggest, simply leave it in place as is and wait for a better solution from upstream.
Therefore, after thinking about this I understand your perspective and respect your decision to keep it active and in place. If your opinion remains the same, should I make the appropriate changes to the PR? Any thoughts @ArrayBolt3?
As an aside, note that even the attack vector controls introduced in kernel 6.17 appear to inherit the same application of default mitigations. A good future solution would be as you suggested in #199 (comment) to have a sort of
mitigations=strict,nosmtoption.Sorry to accidentally leave this for so long, been caught up in a lot of other things.
@raja-grewal Yes, I think keeping the parameter in place would be a good idea. It may be that other settings and kernel configuration that exists make this parameter redundant and even misinformative, but it would only take one bad refactor in the kernel to change how things currently work and possibly leave us in a vulnerable state without the parameter. OpenSSH has had one such bad refactor, a critical PGP library used by Thunderbird had another just recently, I don't trust the kernel to not have the same mistake made with it.
Ok no worries, understood.
I get both of your reasoning and respect your judgments and so have changed it back. Therefore, usage of
mitigations=auto,nosmtcontinues from when I initially set it previously.Let me know whether you guys find this reversion better than my current recommendation to now explicitly stop using
mitigations=auto,nosmt.That does look good to me. I'll do a review of that PR now and most likely merge it in. Thank you!
According to the kernel's documentation:
auto,nosmt:
Mitigate all CPU vulnerabilities, disabling SMT if needed. This is for users who always want to be fully mitigated, even if it means losing SMT. Equivalent to:
Why are these other args being explicitly set in
/etc/default/grub.dif mitigations=auto,nosmt is already being set?There is a limit on number of characters that can be in the kernel args as well, and the kernel args we set will just get longer and longer over time. I don't think it is a good idea to waste the precious characters on these redundant args. Either we use
mitigations=auto,nosmt(which should be the default anyways), or explicitly spell out which args to set so the user can easily customize them. There is really no reason to have both.