The Linux kernel stable release model started in 2005, when it was determined that the existing kernel development model (a new release every 2-3 months) was not meeting the needs of most users. Users wanted bugfixes made during those 2-3 months, and Linux distributions found it difficult to keep kernels up to date without feedback from the kernel community. In general, attempts to keep individual kernels secure and with the latest bugfixes was a large and confusing effort by lots of different individuals.
Stable kernel releases are based directly on Linus Torvalds' releases, and are released every week or so, depending on various external factors (time of year, available patches, maintainer workload, etc.). The numbering of the stable releases starts with the number of the kernel release, and an additional number is added to the end of it. For example, the 4.4 kernel is released by Linus, and then the stable kernel releases based on this kernel are numbered 4.4.1, 4.4.2, 4.4.3, and so on. This sequence is usually shortened with the number 4.4.y when referring to a stable kernel release tree. Each stable kernel release tree is maintained by a single kernel developer, who is responsible for picking the needed patches for the release and managing the review/release process.
Stable kernels are maintained for the length of the current development cycle. After Linus releases a new kernel, the previous stable kernel release tree is stopped and users must move to the newer released kernel.
Long-term stable kernels
After a year of this new stable release process, it was determined that many different users of Linux wanted a kernel to be supported for longer than just a few months. In response, the Long Term Supported (LTS) kernel release was created, with the first LTS kernel (2.6.16) released in 2006. Since then, a new LTS kernel has been selected once a year and kernel community maintains that kernel for a minimum of 2 years.
At the time of this writing, the LTS kernels are the 4.4.y, 4.9.y, 4.14.y, 4.19.y, 5.4.y, and 5.10.y releases. A new kernel is released weekly. Due to the needs of some users and distributions, a few additional older kernels are maintained by kernel developers at a slower release cycle. Information about all long-term stable kernels, who is in charge of them, and how long they will be maintained, can be found on the kernel.org releases page.
LTS kernel releases average 6-8 patches accepted per day, while the normal stable kernel releases contain 10-15 patches per day. The number of patches fluctuates per release given the current time of the corresponding development kernel release, and other external variables. The older a LTS kernel is, the less patches are applicable to it as many recent bugfixes are not relevant to older kernels. However, the older a kernel is, the harder it is to backport the changes that are needed to be applied, due to the changes in the codebase. So while there might be a lower number of overall patches being applied, the effort involved in maintaining a LTS kernel is greater than maintaining the normal stable kernel.
Stable kernel patch rules
The rules for what can be added to a stable kernel release have remained almost identical since its introduction and are summarized below:
- Must be obviously correct and tested.
- Must not be bigger than 100 lines.
- Must fix only one thing.
- Must fix something that has been reported to be an issue.
- Can be a new device id or quirk for hardware, but not add major new functionality.
- Must already be merged into Linus Torvalds' tree.
The last rule, "Must already be merged into Linus Torvalds' tree", prevents the kernel community from losing fixes. The community never wants a fix to go into a stable kernel release that is not already in Linus Torvalds' tree, so that anyone who upgrades should never see a regression. This prevents many problems that other projects who maintain a stable and development branch can have.
The Linux kernel community has promised its userbase that no upgrade will ever break anything that is currently working in a previous release. That promise still holds true today. Regressions do happen, but those are the highest priority bugs and are either quickly fixed, or the change that caused the regression is quickly reverted from the Linux kernel tree.
This promise holds true for both the incremental stable kernel updates, as well as the larger major updates that happen every three months. However, the kernel community can only make this promise for the code that is merged into the Linux kernel tree. Any code that is merged into a device's kernel that is not in the kernel.org releases is unknown and interactions with it can never be planned for, or even considered.
Devices based on Linux that have large patch sets can have major issues when updating to newer kernels, because of the large number of changes between each release (10-14 thousand changes per release). SoC patchsets are especially known to have issues with updating to newer kernels due to their large size and heavy modification of architecture specific, and sometimes core, kernel code. As a result, most SoC vendors are starting to standardize on using the LTS releases for their devices, enabling those devices to receive bug and security updates directly from the Linux kernel community.
When doing kernel releases, the Linux kernel community almost never declares specific changes as security fixes. This is due to the basic problem of the difficulty in determining if a bugfix is a security fix or not at the time of creation. Also, many bugfixes are only determined to be security related after much time has passed, so the kernel community strongly recommends always taking all bugfixes that are released.
When security problems are reported to the kernel community, they are fixed as soon as possible and pushed out publically to the development tree and the stable releases. As described above, the changes are almost never described as a "security fix", but rather look like any other bugfix for the kernel. This is done to allow affected parties the ability to update their systems before the reporter of the problem announces it.
For details on reporting security bugs to the kernel community to get them resolved and fixed as soon as possible, refer to Security bugs in The Linux kernel user's and administrator's guide at www.kernel.org.
Because security bugs are not announced to the public by the kernel team, CVE numbers for Linux kernel-related issues are usually released weeks, months, and sometimes years after the fix was merged into the stable and development branches.
Keeping a secure system
When deploying a device that uses Linux, it is strongly recommended that all LTS kernel updates be taken by the manufacturer and pushed out to their users after proper testing shows the update works well. This has several advantages:
- Releases have been reviewed by the kernel developers as a whole, not in individual parts.
- It is hard, if not impossible, to determine which patches fix "security" issues and which do not. Almost every LTS release contains at least one known security fix, and many yet "unknown".
- If testing shows a problem, the kernel developer community will react quickly to resolve the issue.
- Attempts to filter out only the changes you run will result in a kernel tree that is impossible to merge correctly with future upstream releases.