verify-bugs-and-bisect-regressions.rst

   1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
   2.. [see the bottom of this file for redistribution information]
   3
   4=========================================
   5How to verify bugs and bisect regressions
   6=========================================
   7
   8This document describes how to check if some Linux kernel problem occurs in code
   9currently supported by developers -- to then explain how to locate the change
  10causing the issue, if it is a regression (e.g. did not happen with earlier
  11versions).
  12
  13The text aims at people running kernels from mainstream Linux distributions on
  14commodity hardware who want to report a kernel bug to the upstream Linux
  15developers. Despite this intent, the instructions work just as well for users
  16who are already familiar with building their own kernels: they help avoid
  17mistakes occasionally made even by experienced developers.
  18
  19..
  20   Note: if you see this note, you are reading the text's source file. You
  21   might want to switch to a rendered version: it makes it a lot easier to
  22   read and navigate this document -- especially when you want to look something
  23   up in the reference section, then jump back to where you left off.
  24..
  25   Find the latest rendered version of this text here:
  26   https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
  27
  28The essence of the process (aka 'TL;DR')
  29========================================
  30
  31*[If you are new to building or bisecting Linux, ignore this section and head
  32over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes
  33the same commands as this section while describing them in brief fashion. The
  34steps are nevertheless easy to follow and together with accompanying entries
  35in a reference section mention many alternatives, pitfalls, and additional
  36aspects, all of which might be essential in your present case.]*
  37
  38**In case you want to check if a bug is present in code currently supported by
  39developers**, execute just the *preparations* and *segment 1*; while doing so,
  40consider the newest Linux kernel you regularly use to be the 'working' kernel.
  41In the following example that's assumed to be 6.0, which is why its sources
  42will be used to prepare the .config file.
  43
  44**In case you face a regression**, follow the steps at least till the end of
  45*segment 2*. Then you can submit a preliminary report -- or continue with
  46*segment 3*, which describes how to perform a bisection needed for a
  47full-fledged regression report. In the following example 6.0.13 is assumed to be
  48the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.0
  49will be considered the 'good' release and used to prepare the .config file.
  50
  51* **Preparations**: set up everything to build your own kernels::
  52
  53    # * Remove any software that depends on externally maintained kernel modules
  54    #   or builds any automatically during bootup.
  55    # * Ensure Secure Boot permits booting self-compiled Linux kernels.
  56    # * If you are not already running the 'working' kernel, reboot into it.
  57    # * Install compilers and everything else needed for building Linux.
  58    # * Ensure to have 15 Gigabyte free space in your home directory.
  59    git clone -o mainline --no-checkout \
  60      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
  61    cd ~/linux/
  62    git remote add -t master stable \
  63      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
  64    git switch --detach v6.0
  65    # * Hint: if you used an existing clone, ensure no stale .config is around.
  66    make olddefconfig
  67    # * Ensure the former command picked the .config of the 'working' kernel.
  68    # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
  69    #   VPNs, mount network shares, and briefly try the feature that is broken.
  70    yes '' | make localmodconfig
  71    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
  72    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
  73    # * Note, when short on storage space, check the guide for an alternative:
  74    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
  75      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
  76    # * Hint: at this point you might want to adjust the build configuration;
  77    #   you'll have to, if you are running Debian.
  78    make olddefconfig
  79    cp .config ~/kernel-config-working
  80
  81* **Segment 1**: build a kernel from the latest mainline codebase.
  82
  83  This among others checks if the problem was fixed already and which developers
  84  later need to be told about the problem; in case of a regression, this rules
  85  out a .config change as root of the problem.
  86
  87  a) Checking out latest mainline code::
  88
  89       cd ~/linux/
  90       git switch --discard-changes --detach mainline/master
  91
  92  b) Build, install, and boot a kernel::
  93
  94       cp ~/kernel-config-working .config
  95       make olddefconfig
  96       make -j $(nproc --all)
  97       # * Make sure there is enough disk space to hold another kernel:
  98       df -h /boot/ /lib/modules/
  99       # * Note: on Arch Linux, its derivatives and a few other distributions
 100       #   the following commands will do nothing at all or only part of the
 101       #   job. See the step-by-step guide for further details.
 102       sudo make modules_install
 103       command -v installkernel && sudo make install
 104       # * Check how much space your self-built kernel actually needs, which
 105       #   enables you to make better estimates later:
 106       du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
 107       du -sh /lib/modules/$(make -s kernelrelease)/
 108       # * Hint: the output of the following command will help you pick the
 109       #   right kernel from the boot menu:
 110       make -s kernelrelease | tee -a ~/kernels-built
 111       reboot
 112       # * Once booted, ensure you are running the kernel you just built by
 113       #   checking if the output of the next two commands matches:
 114       tail -n 1 ~/kernels-built
 115       uname -r
 116       cat /proc/sys/kernel/tainted
 117
 118  c) Check if the problem occurs with this kernel as well.
 119
 120* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
 121
 122  This among others verifies the trimmed .config file actually works well, as
 123  bisecting with it otherwise would be a waste of time:
 124
 125  a) Start by checking out the sources of the 'good' version::
 126
 127       cd ~/linux/
 128       git switch --discard-changes --detach v6.0
 129
 130  b) Build, install, and boot a kernel as described earlier in *segment 1,
 131     section b* -- just feel free to skip the 'du' commands, as you have a rough
 132     estimate already.
 133
 134  c) Ensure the feature that regressed with the 'broken' kernel actually works
 135     with this one.
 136
 137* **Segment 3**: perform and validate the bisection.
 138
 139  a) Retrieve the sources for your 'bad' version::
 140
 141       git remote set-branches --add stable linux-6.1.y
 142       git fetch stable
 143
 144  b) Initialize the bisection::
 145
 146       cd ~/linux/
 147       git bisect start
 148       git bisect good v6.0
 149       git bisect bad v6.1.5
 150
 151  c) Build, install, and boot a kernel as described earlier in *segment 1,
 152     section b*.
 153
 154     In case building or booting the kernel fails for unrelated reasons, run
 155     ``git bisect skip``. In all other outcomes, check if the regressed feature
 156     works with the newly built kernel. If it does, tell Git by executing
 157     ``git bisect good``; if it does not, run ``git bisect bad`` instead.
 158
 159     All three commands will make Git check out another commit; then re-execute
 160     this step (e.g. build, install, boot, and test a kernel to then tell Git
 161     the outcome). Do so again and again until Git shows which commit broke
 162     things. If you run short of disk space during this process, check the
 163     section 'Complementary tasks: cleanup during and after the process'
 164     below.
 165
 166  d) Once your finished the bisection, put a few things away::
 167
 168       cd ~/linux/
 169       git bisect log > ~/bisect-log
 170       cp .config ~/bisection-config-culprit
 171       git bisect reset
 172
 173  e) Try to verify the bisection result::
 174
 175       git switch --discard-changes --detach mainline/master
 176       git revert --no-edit cafec0cacaca0
 177       cp ~/kernel-config-working .config
 178       ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
 179
 180    This is optional, as some commits are impossible to revert. But if the
 181    second command worked flawlessly, build, install, and boot one more kernel
 182    kernel; just this time skip the first command copying the base .config file
 183    over, as that already has been taken care off.
 184
 185* **Complementary tasks**: cleanup during and after the process.
 186
 187  a) To avoid running out of disk space during a bisection, you might need to
 188     remove some kernels you built earlier. You most likely want to keep those
 189     you built during segment 1 and 2 around for a while, but you will most
 190     likely no longer need kernels tested during the actual bisection
 191     (Segment 3 c). You can list them in build order using::
 192
 193       ls -ltr /lib/modules/*-local*
 194
 195    To then for example erase a kernel that identifies itself as
 196    '6.0-rc1-local-gcafec0cacaca0', use this::
 197
 198       sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
 199       sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
 200       # * Note, on some distributions kernel-install is missing
 201       #   or does only part of the job.
 202
 203  b) If you performed a bisection and successfully validated the result, feel
 204     free to remove all kernels built during the actual bisection (Segment 3 c);
 205     the kernels you built earlier and later you might want to keep around for
 206     a week or two.
 207
 208* **Optional task**: test a debug patch or a proposed fix later::
 209
 210    git fetch mainline
 211    git switch --discard-changes --detach mainline/master
 212    git apply /tmp/foobars-proposed-fix-v1.patch
 213    cp ~/kernel-config-working .config
 214    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
 215
 216  Build, install, and boot a kernel as described in *segment 1, section b* --
 217  but this time omit the first command copying the build configuration over,
 218  as that has been taken care of already.
 219
 220.. _introguide_bissbs:
 221
 222Step-by-step guide on how to verify bugs and bisect regressions
 223===============================================================
 224
 225This guide describes how to set up your own Linux kernels for investigating bugs
 226or regressions you intend to report. How far you want to follow the instructions
 227depends on your issue:
 228
 229Execute all steps till the end of *segment 1* to **verify if your kernel problem
 230is present in code supported by Linux kernel developers**. If it is, you are all
 231set to report the bug -- unless it did not happen with earlier kernel versions,
 232as then your want to at least continue with *segment 2* to **check if the issue
 233qualifies as regression** which receive priority treatment. Depending on the
 234outcome you then are ready to report a bug or submit a preliminary regression
 235report; instead of the latter your could also head straight on and follow
 236*segment 3* to **perform a bisection** for a full-fledged regression report
 237developers are obliged to act upon.
 238
 239 :ref:`Preparations: set up everything to build your own kernels <introprep_bissbs>`.
 240
 241 :ref:`Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>`.
 242
 243 :ref:`Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>`.
 244
 245 :ref:`Segment 3: perform a bisection and validate the result <introbisect_bissbs>`.
 246
 247 :ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
 248
 249 :ref:`Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
 250
 251The steps in each segment illustrate the important aspects of the process, while
 252a comprehensive reference section holds additional details for almost all of the
 253steps. The reference section sometimes also outlines alternative approaches,
 254pitfalls, as well as problems that might occur at the particular step -- and how
 255to get things rolling again.
 256
 257For further details on how to report Linux kernel issues or regressions check
 258out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
 259with this document. It among others explains why you need to verify bugs with
 260the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6),
 261even if you face a problem with a kernel from a 'stable/longterm' series
 262(say 6.0.13).
 263
 264For users facing a regression that document also explains why sending a
 265preliminary report after segment 2 might be wise, as the regression and its
 266culprit might be known already. For further details on what actually qualifies
 267as a regression check out Documentation/admin-guide/reporting-regressions.rst.
 268
 269If you run into any problems while following this guide or have ideas how to
 270improve it, :ref:`please let the kernel developers know <submit_improvements_vbbr>`.
 271
 272.. _introprep_bissbs:
 273
 274Preparations: set up everything to build your own kernels
 275---------------------------------------------------------
 276
 277The following steps lay the groundwork for all further tasks.
 278
 279Note: the instructions assume you are building and testing on the same
 280machine; if you want to compile the kernel on another system, check
 281:ref:`Build kernels on a different machine <buildhost_bis>` below.
 282
 283.. _backup_bissbs:
 284
 285* Create a fresh backup and put system repair and restore tools at hand, just
 286  to be prepared for the unlikely case of something going sideways.
 287
 288  [:ref:`details <backup_bisref>`]
 289
 290.. _vanilla_bissbs:
 291
 292* Remove all software that depends on externally developed kernel drivers or
 293  builds them automatically. That includes but is not limited to DKMS, openZFS,
 294  VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
 295
 296  [:ref:`details <vanilla_bisref>`]
 297
 298.. _secureboot_bissbs:
 299
 300* On platforms with 'Secure Boot' or similar solutions, prepare everything to
 301  ensure the system will permit your self-compiled kernel to boot. The
 302  quickest and easiest way to achieve this on commodity x86 systems is to
 303  disable such techniques in the BIOS setup utility; alternatively, remove
 304  their restrictions through a process initiated by
 305  ``mokutil --disable-validation``.
 306
 307  [:ref:`details <secureboot_bisref>`]
 308
 309.. _rangecheck_bissbs:
 310
 311* Determine the kernel versions considered 'good' and 'bad' throughout this
 312  guide:
 313
 314  * Do you follow this guide to verify if a bug is present in the code the
 315    primary developers care for? Then consider the version of the newest kernel
 316    you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2).
 317
 318  * Do you face a regression, e.g. something broke or works worse after
 319    switching to a newer kernel version? In that case it depends on the version
 320    range during which the problem appeared:
 321
 322    * Something regressed when updating from a stable/longterm release
 323      (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a
 324      stable/longterm version based on one (say 6.1.5)? Then consider the
 325      mainline release your working kernel is based on to be the 'good'
 326      version (e.g. 6.0) and the first version to be broken as the 'bad' one
 327      (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed
 328      that 6.0 is fine; this hypothesis will be checked in segment 2.
 329
 330    * Something regressed when switching from one mainline version (say 6.0) to
 331      a later one (like 6.1-rc1) or a stable/longterm release based on it
 332      (say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'good' and
 333      the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'.
 334
 335    * Something regressed when updating within a stable/longterm series (say
 336      from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and 'bad'
 337      (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series.
 338
 339  *Note, do not confuse 'good' version with 'working' kernel; the latter term
 340  throughout this guide will refer to the last kernel that has been working
 341  fine.*
 342
 343  [:ref:`details <rangecheck_bisref>`]
 344
 345.. _bootworking_bissbs:
 346
 347* Boot into the 'working' kernel and briefly use the apparently broken feature.
 348
 349  [:ref:`details <bootworking_bisref>`]
 350
 351.. _diskspace_bissbs:
 352
 353* Ensure to have enough free space for building Linux. 15 Gigabyte in your home
 354  directory should typically suffice. If you have less available, be sure to pay
 355  attention to later steps about retrieving the Linux sources and handling of
 356  debug symbols: both explain approaches reducing the amount of space, which
 357  should allow you to master these tasks with about 4 Gigabytes free space.
 358
 359  [:ref:`details <diskspace_bisref>`]
 360
 361.. _buildrequires_bissbs:
 362
 363* Install all software required to build a Linux kernel. Often you will need:
 364  'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
 365  'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
 366  reference section shows how to quickly install those on various popular Linux
 367  distributions.
 368
 369  [:ref:`details <buildrequires_bisref>`]
 370
 371.. _sources_bissbs:
 372
 373* Retrieve the mainline Linux sources; then change into the directory holding
 374  them, as all further commands in this guide are meant to be executed from
 375  there.
 376
 377  *Note, the following describe how to retrieve the sources using a full
 378  mainline clone, which downloads about 2,75 GByte as of early 2024. The*
 379  :ref:`reference section describes two alternatives <sources_bisref>` *:
 380  one downloads less than 500 MByte, the other works better with unreliable
 381  internet connections.*
 382
 383  Execute the following command to retrieve a fresh mainline codebase while
 384  preparing things to add branches for stable/longterm series later::
 385
 386    git clone -o mainline --no-checkout \
 387      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
 388    cd ~/linux/
 389    git remote add -t master stable \
 390      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
 391
 392  [:ref:`details <sources_bisref>`]
 393
 394.. _stablesources_bissbs:
 395
 396* Is one of the versions you earlier established as 'good' or 'bad' a stable or
 397  longterm release (say 6.1.5)? Then download the code for the series it belongs
 398  to ('linux-6.1.y' in this example)::
 399
 400    git remote set-branches --add stable linux-6.1.y
 401    git fetch stable
 402
 403.. _oldconfig_bissbs:
 404
 405* Start preparing a kernel build configuration (the '.config' file).
 406
 407  Before doing so, ensure you are still running the 'working' kernel an earlier
 408  step told you to boot; if you are unsure, check the current kernelrelease
 409  identifier using ``uname -r``.
 410
 411  Afterwards check out the source code for the version earlier established as
 412  'good'. In the following example command this is assumed to be 6.0; note that
 413  the version number in this and all later Git commands needs to be prefixed
 414  with a 'v'::
 415
 416    git switch --discard-changes --detach v6.0
 417
 418  Now create a build configuration file::
 419
 420    make olddefconfig
 421
 422  The kernel build scripts then will try to locate the build configuration file
 423  for the running kernel and then adjust it for the needs of the kernel sources
 424  you checked out. While doing so, it will print a few lines you need to check.
 425
 426  Look out for a line starting with '# using defaults found in'. It should be
 427  followed by a path to a file in '/boot/' that contains the release identifier
 428  of your currently working kernel. If the line instead continues with something
 429  like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find
 430  the .config file for your running kernel -- in which case you have to put one
 431  there manually, as explained in the reference section.
 432
 433  In case you can not find such a line, look for one containing '# configuration
 434  written to .config'. If that's the case you have a stale build configuration
 435  lying around. Unless you intend to use it, delete it; afterwards run
 436  'make olddefconfig' again and check if it now picked up the right config file
 437  as base.
 438
 439  [:ref:`details <oldconfig_bisref>`]
 440
 441.. _localmodconfig_bissbs:
 442
 443* Disable any kernel modules apparently superfluous for your setup. This is
 444  optional, but especially wise for bisections, as it speeds up the build
 445  process enormously -- at least unless the .config file picked up in the
 446  previous step was already tailored to your and your hardware needs, in which
 447  case you should skip this step.
 448
 449  To prepare the trimming, connect external hardware you occasionally use (USB
 450  keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted
 451  since you started that guide, ensure that you tried using the feature causing
 452  trouble since you started the system. Only then trim your .config::
 453
 454     yes '' | make localmodconfig
 455
 456  There is a catch to this, as the 'apparently' in initial sentence of this step
 457  and the preparation instructions already hinted at:
 458
 459  The 'localmodconfig' target easily disables kernel modules for features only
 460  used occasionally -- like modules for external peripherals not yet connected
 461  since booting, virtualization software not yet utilized, VPN tunnels, and a
 462  few other things. That's because some tasks rely on kernel modules Linux only
 463  loads when you execute tasks like the aforementioned ones for the first time.
 464
 465  This drawback of localmodconfig is nothing you should lose sleep over, but
 466  something to keep in mind: if something is misbehaving with the kernels built
 467  during this guide, this is most likely the reason. You can reduce or nearly
 468  eliminate the risk with tricks outlined in the reference section; but when
 469  building a kernel just for quick testing purposes this is usually not worth
 470  spending much effort on, as long as it boots and allows to properly test the
 471  feature that causes trouble.
 472
 473  [:ref:`details <localmodconfig_bisref>`]
 474
 475.. _tagging_bissbs:
 476
 477* Ensure all the kernels you will build are clearly identifiable using a special
 478  tag and a unique version number::
 479
 480    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
 481    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
 482
 483  [:ref:`details <tagging_bisref>`]
 484
 485.. _debugsymbols_bissbs:
 486
 487* Decide how to handle debug symbols.
 488
 489  In the context of this document it is often wise to enable them, as there is a
 490  decent chance you will need to decode a stack trace from a 'panic', 'Oops',
 491  'warning', or 'BUG'::
 492
 493    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
 494      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
 495
 496  But if you are extremely short on storage space, you might want to disable
 497  debug symbols instead::
 498
 499    ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
 500      -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
 501
 502  [:ref:`details <debugsymbols_bisref>`]
 503
 504.. _configmods_bissbs:
 505
 506* Check if you may want or need to adjust some other kernel configuration
 507  options:
 508
 509  * Are you running Debian? Then you want to avoid known problems by performing
 510    additional adjustments explained in the reference section.
 511
 512    [:ref:`details <configmods_distros_bisref>`].
 513
 514  * If you want to influence other aspects of the configuration, do so now using
 515    your preferred tool. Note, to use make targets like 'menuconfig' or
 516    'nconfig', you will need to install the development files of ncurses; for
 517    'xconfig' you likewise need the Qt5 or Qt6 headers.
 518
 519    [:ref:`details <configmods_individual_bisref>`].
 520
 521.. _saveconfig_bissbs:
 522
 523* Reprocess the .config after the latest adjustments and store it in a safe
 524  place::
 525
 526     make olddefconfig
 527     cp .config ~/kernel-config-working
 528
 529  [:ref:`details <saveconfig_bisref>`]
 530
 531.. _introlatestcheck_bissbs:
 532
 533Segment 1: try to reproduce the problem with the latest codebase
 534----------------------------------------------------------------
 535
 536The following steps verify if the problem occurs with the code currently
 537supported by developers. In case you face a regression, it also checks that the
 538problem is not caused by some .config change, as reporting the issue then would
 539be a waste of time. [:ref:`details <introlatestcheck_bisref>`]
 540
 541.. _checkoutmaster_bissbs:
 542
 543* Check out the latest Linux codebase.
 544
 545  * Are your 'good' and 'bad' versions from the same stable or longterm series?
 546    Then check the `front page of kernel.org <https://kernel.org/>`_: if it
 547    lists a release from that series without an '[EOL]' tag, checkout the series
 548    latest version ('linux-6.1.y' in the following example)::
 549
 550      cd ~/linux/
 551      git switch --discard-changes --detach stable/linux-6.1.y
 552
 553    Your series is unsupported, if is not listed or carrying a 'end of life'
 554    tag. In that case you might want to check if a successor series (say
 555    linux-6.2.y) or mainline (see next point) fix the bug.
 556
 557  * In all other cases, run::
 558
 559      cd ~/linux/
 560      git switch --discard-changes --detach mainline/master
 561
 562  [:ref:`details <checkoutmaster_bisref>`]
 563
 564.. _build_bissbs:
 565
 566* Build the image and the modules of your first kernel using the config file you
 567  prepared::
 568
 569    cp ~/kernel-config-working .config
 570    make olddefconfig
 571    make -j $(nproc --all)
 572
 573  If you want your kernel packaged up as deb, rpm, or tar file, see the
 574  reference section for alternatives, which obviously will require other
 575  steps to install as well.
 576
 577  [:ref:`details <build_bisref>`]
 578
 579.. _install_bissbs:
 580
 581* Install your newly built kernel.
 582
 583  Before doing so, consider checking if there is still enough space for it::
 584
 585    df -h /boot/ /lib/modules/
 586
 587  For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how
 588  much your kernels actually require will be determined later during this guide.
 589
 590  Now install the kernel's modules and its image, which will be stored in
 591  parallel to the your Linux distribution's kernels::
 592
 593    sudo make modules_install
 594    command -v installkernel && sudo make install
 595
 596  The second command ideally will take care of three steps required at this
 597  point: copying the kernel's image to /boot/, generating an initramfs, and
 598  adding an entry for both to the boot loader's configuration.
 599
 600  Sadly some distributions (among them Arch Linux, its derivatives, and many
 601  immutable Linux distributions) will perform none or only some of those tasks.
 602  You therefore want to check if all of them were taken care of and manually
 603  perform those that were not. The reference section provides further details on
 604  that; your distribution's documentation might help, too.
 605
 606  Once you figured out the steps needed at this point, consider writing them
 607  down: if you will build more kernels as described in segment 2 and 3, you will
 608  have to perform those again after executing ``command -v installkernel [...]``.
 609
 610  [:ref:`details <install_bisref>`]
 611
 612.. _storagespace_bissbs:
 613
 614* In case you plan to follow this guide further, check how much storage space
 615  the kernel, its modules, and other related files like the initramfs consume::
 616
 617    du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
 618    du -sh /lib/modules/$(make -s kernelrelease)/
 619
 620  Write down or remember those two values for later: they enable you to prevent
 621  running out of disk space accidentally during a bisection.
 622
 623  [:ref:`details <storagespace_bisref>`]
 624
 625.. _kernelrelease_bissbs:
 626
 627* Show and store the kernelrelease identifier of the kernel you just built::
 628
 629    make -s kernelrelease | tee -a ~/kernels-built
 630
 631  Remember the identifier momentarily, as it will help you pick the right kernel
 632  from the boot menu upon restarting.
 633
 634* Reboot into your newly built kernel. To ensure your actually started the one
 635  you just built, you might want to verify if the output of these commands
 636  matches::
 637
 638    tail -n 1 ~/kernels-built
 639    uname -r
 640
 641.. _tainted_bissbs:
 642
 643* Check if the kernel marked itself as 'tainted'::
 644
 645    cat /proc/sys/kernel/tainted
 646
 647  If that command does not return '0', check the reference section, as the cause
 648  for this might interfere with your testing.
 649
 650  [:ref:`details <tainted_bisref>`]
 651
 652.. _recheckbroken_bissbs:
 653
 654* Verify if your bug occurs with the newly built kernel. If it does not, check
 655  out the instructions in the reference section to ensure nothing went sideways
 656  during your tests.
 657
 658  [:ref:`details <recheckbroken_bisref>`]
 659
 660.. _recheckstablebroken_bissbs:
 661
 662* Did you just built a stable or longterm kernel? And were you able to reproduce
 663  the regression with it? Then you should test the latest mainline codebase as
 664  well, because the result determines which developers the bug must be submitted
 665  to.
 666
 667  To prepare that test, check out current mainline::
 668
 669    cd ~/linux/
 670    git switch --discard-changes --detach mainline/master
 671
 672  Now use the checked out code to build and install another kernel using the
 673  commands the earlier steps already described in more detail::
 674
 675    cp ~/kernel-config-working .config
 676    make olddefconfig
 677    make -j $(nproc --all)
 678    # * Check if the free space suffices holding another kernel:
 679    df -h /boot/ /lib/modules/
 680    sudo make modules_install
 681    command -v installkernel && sudo make install
 682    make -s kernelrelease | tee -a ~/kernels-built
 683    reboot
 684
 685  Confirm you booted the kernel you intended to start and check its tainted
 686  status::
 687
 688    tail -n 1 ~/kernels-built
 689    uname -r
 690    cat /proc/sys/kernel/tainted
 691
 692  Now verify if this kernel is showing the problem. If it does, then you need
 693  to report the bug to the primary developers; if it does not, report it to the
 694  stable team. See Documentation/admin-guide/reporting-issues.rst for details.
 695
 696  [:ref:`details <recheckstablebroken_bisref>`]
 697
 698Do you follow this guide to verify if a problem is present in the code
 699currently supported by Linux kernel developers? Then you are done at this
 700point. If you later want to remove the kernel you just built, check out
 701:ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.
 702
 703In case you face a regression, move on and execute at least the next segment
 704as well.
 705
 706.. _introworkingcheck_bissbs:
 707
 708Segment 2: check if the kernels you build work fine
 709---------------------------------------------------
 710
 711In case of a regression, you now want to ensure the trimmed configuration file
 712you created earlier works as expected; a bisection with the .config file
 713otherwise would be a waste of time. [:ref:`details <introworkingcheck_bisref>`]
 714
 715.. _recheckworking_bissbs:
 716
 717* Build your own variant of the 'working' kernel and check if the feature that
 718  regressed works as expected with it.
 719
 720  Start by checking out the sources for the version earlier established as
 721  'good' (once again assumed to be 6.0 here)::
 722
 723    cd ~/linux/
 724    git switch --discard-changes --detach v6.0
 725
 726  Now use the checked out code to configure, build, and install another kernel
 727  using the commands the previous subsection explained in more detail::
 728
 729    cp ~/kernel-config-working .config
 730    make olddefconfig
 731    make -j $(nproc --all)
 732    # * Check if the free space suffices holding another kernel:
 733    df -h /boot/ /lib/modules/
 734    sudo make modules_install
 735    command -v installkernel && sudo make install
 736    make -s kernelrelease | tee -a ~/kernels-built
 737    reboot
 738
 739  When the system booted, you may want to verify once again that the
 740  kernel you started is the one you just built::
 741
 742    tail -n 1 ~/kernels-built
 743    uname -r
 744
 745  Now check if this kernel works as expected; if not, consult the reference
 746  section for further instructions.
 747
 748  [:ref:`details <recheckworking_bisref>`]
 749
 750.. _introbisect_bissbs:
 751
 752Segment 3: perform the bisection and validate the result
 753--------------------------------------------------------
 754
 755With all the preparations and precaution builds taken care of, you are now ready
 756to begin the bisection. This will make you build quite a few kernels -- usually
 757about 15 in case you encountered a regression when updating to a newer series
 758(say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build
 759configuration created earlier this works a lot faster than many people assume:
 760overall on average it will often just take about 10 to 15 minutes to compile
 761each kernel on commodity x86 machines.
 762
 763.. _bisectstart_bissbs:
 764
 765* Start the bisection and tell Git about the versions earlier established as
 766  'good' (6.0 in the following example command) and 'bad' (6.1.5)::
 767
 768    cd ~/linux/
 769    git bisect start
 770    git bisect good v6.0
 771    git bisect bad v6.1.5
 772
 773  [:ref:`details <bisectstart_bisref>`]
 774
 775.. _bisectbuild_bissbs:
 776
 777* Now use the code Git checked out to build, install, and boot a kernel using
 778  the commands introduced earlier::
 779
 780    cp ~/kernel-config-working .config
 781    make olddefconfig
 782    make -j $(nproc --all)
 783    # * Check if the free space suffices holding another kernel:
 784    df -h /boot/ /lib/modules/
 785    sudo make modules_install
 786    command -v installkernel && sudo make install
 787    make -s kernelrelease | tee -a ~/kernels-built
 788    reboot
 789
 790  If compilation fails for some reason, run ``git bisect skip`` and restart
 791  executing the stack of commands from the beginning.
 792
 793  In case you skipped the 'test latest codebase' step in the guide, check its
 794  description as for why the 'df [...]' and 'make -s kernelrelease [...]'
 795  commands are here.
 796
 797  Important note: the latter command from this point on will print release
 798  identifiers that might look odd or wrong to you -- which they are not, as it's
 799  totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
 800  if you bisect between versions 6.1 and 6.2 for example.
 801
 802  [:ref:`details <bisectbuild_bisref>`]
 803
 804.. _bisecttest_bissbs:
 805
 806* Now check if the feature that regressed works in the kernel you just built.
 807
 808  You again might want to start by making sure the kernel you booted is the one
 809  you just built::
 810
 811    cd ~/linux/
 812    tail -n 1 ~/kernels-built
 813    uname -r
 814
 815  Now verify if the feature that regressed works at this kernel bisection point.
 816  If it does, run this::
 817
 818    git bisect good
 819
 820  If it does not, run this::
 821
 822    git bisect bad
 823
 824  Be sure about what you tell Git, as getting this wrong just once will send the
 825  rest of the bisection totally off course.
 826
 827  While the bisection is ongoing, Git will use the information you provided to
 828  find and check out another bisection point for you to test. While doing so, it
 829  will print something like 'Bisecting: 675 revisions left to test after this
 830  (roughly 10 steps)' to indicate how many further changes it expects to be
 831  tested. Now build and install another kernel using the instructions from the
 832  previous step; afterwards follow the instructions in this step again.
 833
 834  Repeat this again and again until you finish the bisection -- that's the case
 835  when Git after tagging a change as 'good' or 'bad' prints something like
 836  'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right
 837  afterwards it will show some details about the culprit including the patch
 838  description of the change. The latter might fill your terminal screen, so you
 839  might need to scroll up to see the message mentioning the culprit;
 840  alternatively, run ``git bisect log > ~/bisection-log``.
 841
 842  [:ref:`details <bisecttest_bisref>`]
 843
 844.. _bisectlog_bissbs:
 845
 846* Store Git's bisection log and the current .config file in a safe place before
 847  telling Git to reset the sources to the state before the bisection::
 848
 849    cd ~/linux/
 850    git bisect log > ~/bisection-log
 851    cp .config ~/bisection-config-culprit
 852    git bisect reset
 853
 854  [:ref:`details <bisectlog_bisref>`]
 855
 856.. _revert_bissbs:
 857
 858* Try reverting the culprit on top of latest mainline to see if this fixes your
 859  regression.
 860
 861  This is optional, as it might be impossible or hard to realize. The former is
 862  the case, if the bisection determined a merge commit as the culprit; the
 863  latter happens if other changes depend on the culprit. But if the revert
 864  succeeds, it is worth building another kernel, as it validates the result of
 865  a bisection, which can easily deroute; it furthermore will let kernel
 866  developers know, if they can resolve the regression with a quick revert.
 867
 868  Begin by checking out the latest codebase depending on the range you bisected:
 869
 870  * Did you face a regression within a stable/longterm series (say between
 871    6.0.13 and 6.0.15) that does not happen in mainline? Then check out the
 872    latest codebase for the affected series like this::
 873
 874      git fetch stable
 875      git switch --discard-changes --detach linux-6.0.y
 876
 877  * In all other cases check out latest mainline::
 878
 879      git fetch mainline
 880      git switch --discard-changes --detach mainline/master
 881
 882    If you bisected a regression within a stable/longterm series that also
 883    happens in mainline, there is one more thing to do: look up the mainline
 884    commit-id. To do so, use a command like ``git show abcdcafecabcd`` to
 885    view the patch description of the culprit. There will be a line near
 886    the top which looks like 'commit cafec0cacaca0 upstream.' or
 887    'Upstream commit cafec0cacaca0'; use that commit-id in the next command
 888    and not the one the bisection blamed.
 889
 890  Now try reverting the culprit by specifying its commit id::
 891
 892    git revert --no-edit cafec0cacaca0
 893
 894  If that fails, give up trying and move on to the next step; if it works,
 895  adjust the tag to facilitate the identification and prevent accidentally
 896  overwriting another kernel::
 897
 898    cp ~/kernel-config-working .config
 899    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
 900
 901  Build a kernel using the familiar command sequence, just without copying the
 902  the base .config over::
 903
 904    make olddefconfig &&
 905    make -j $(nproc --all)
 906    # * Check if the free space suffices holding another kernel:
 907    df -h /boot/ /lib/modules/
 908    sudo make modules_install
 909    command -v installkernel && sudo make install
 910    make -s kernelrelease | tee -a ~/kernels-built
 911    reboot
 912
 913  Now check one last time if the feature that made you perform a bisection works
 914  with that kernel: if everything went well, it should not show the regression.
 915
 916  [:ref:`details <revert_bisref>`]
 917
 918.. _introclosure_bissbs:
 919
 920Complementary tasks: cleanup during and after the bisection
 921-----------------------------------------------------------
 922
 923During and after following this guide you might want or need to remove some of
 924the kernels you installed: the boot menu otherwise will become confusing or
 925space might run out.
 926
 927.. _makeroom_bissbs:
 928
 929* To remove one of the kernels you installed, look up its 'kernelrelease'
 930  identifier. This guide stores them in '~/kernels-built', but the following
 931  command will print them as well::
 932
 933    ls -ltr /lib/modules/*-local*
 934
 935  You in most situations want to remove the oldest kernels built during the
 936  actual bisection (e.g. segment 3 of this guide). The two ones you created
 937  beforehand (e.g. to test the latest codebase and the version considered
 938  'good') might become handy to verify something later -- thus better keep them
 939  around, unless you are really short on storage space.
 940
 941  To remove the modules of a kernel with the kernelrelease identifier
 942  '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its
 943  modules::
 944
 945    sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
 946
 947  Afterwards try the following command::
 948
 949    sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
 950
 951  On quite a few distributions this will delete all other kernel files installed
 952  while also removing the kernel's entry from the boot menu. But on some
 953  distributions kernel-install does not exist or leaves boot-loader entries or
 954  kernel image and related files behind; in that case remove them as described
 955  in the reference section.
 956
 957  [:ref:`details <makeroom_bisref>`]
 958
 959.. _finishingtouch_bissbs:
 960
 961* Once you have finished the bisection, do not immediately remove anything you
 962  set up, as you might need a few things again. What is safe to remove depends
 963  on the outcome of the bisection:
 964
 965  * Could you initially reproduce the regression with the latest codebase and
 966    after the bisection were able to fix the problem by reverting the culprit on
 967    top of the latest codebase? Then you want to keep those two kernels around
 968    for a while, but safely remove all others with a '-local' in the release
 969    identifier.
 970
 971  * Did the bisection end on a merge-commit or seems questionable for other
 972    reasons? Then you want to keep as many kernels as possible around for a few
 973    days: it's pretty likely that you will be asked to recheck something.
 974
 975  * In other cases it likely is a good idea to keep the following kernels around
 976    for some time: the one built from the latest codebase, the one created from
 977    the version considered 'good', and the last three or four you compiled
 978    during the actual bisection process.
 979
 980  [:ref:`details <finishingtouch_bisref>`]
 981
 982.. _introoptional_bissbs:
 983
 984Optional: test reverts, patches, or later versions
 985--------------------------------------------------
 986
 987While or after reporting a bug, you might want or potentially will be asked to
 988test reverts, debug patches, proposed fixes, or other versions. In that case
 989follow these instructions.
 990
 991* Update your Git clone and check out the latest code.
 992
 993  * In case you want to test mainline, fetch its latest changes before checking
 994    its code out::
 995
 996      git fetch mainline
 997      git switch --discard-changes --detach mainline/master
 998
 999  * In case you want to test a stable or longterm kernel, first add the branch
1000    holding the series you are interested in (6.2 in the example), unless you
1001    already did so earlier::
1002
1003      git remote set-branches --add stable linux-6.2.y
1004
1005    Then fetch the latest changes and check out the latest version from the
1006    series::
1007
1008      git fetch stable
1009      git switch --discard-changes --detach stable/linux-6.2.y
1010
1011* Copy your kernel build configuration over::
1012
1013    cp ~/kernel-config-working .config
1014
1015* Your next step depends on what you want to do:
1016
1017  * In case you just want to test the latest codebase, head to the next step,
1018    you are already all set.
1019
1020  * In case you want to test if a revert fixes an issue, revert one or multiple
1021    changes by specifying their commit ids::
1022
1023      git revert --no-edit cafec0cacaca0
1024
1025    Now give that kernel a special tag to facilitates its identification and
1026    prevent accidentally overwriting another kernel::
1027
1028      ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'
1029
1030  * In case you want to test a patch, store the patch in a file like
1031    '/tmp/foobars-proposed-fix-v1.patch' and apply it like this::
1032
1033      git apply /tmp/foobars-proposed-fix-v1.patch
1034
1035    In case of multiple patches, repeat this step with the others.
1036
1037    Now give that kernel a special tag to facilitates its identification and
1038    prevent accidentally overwriting another kernel::
1039
1040    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'
1041
1042* Build a kernel using the familiar commands, just without copying the kernel
1043  build configuration over, as that has been taken care of already::
1044
1045    make olddefconfig &&
1046    make -j $(nproc --all)
1047    # * Check if the free space suffices holding another kernel:
1048    df -h /boot/ /lib/modules/
1049    sudo make modules_install
1050    command -v installkernel && sudo make install
1051    make -s kernelrelease | tee -a ~/kernels-built
1052    reboot
1053
1054* Now verify you booted the newly built kernel and check it.
1055
1056[:ref:`details <introoptional_bisref>`]
1057
1058.. _submit_improvements_vbbr:
1059
1060Conclusion
1061----------
1062
1063You have reached the end of the step-by-step guide.
1064
1065Did you run into trouble following any of the above steps not cleared up by the
1066reference section below? Did you spot errors? Or do you have ideas how to
1067improve the guide?
1068
1069If any of that applies, please take a moment and let the maintainer of this
1070document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while
1071CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is
1072vital to improve this text further, which is in everybody's interest, as it
1073will enable more people to master the task described here -- and hopefully also
1074improve similar guides inspired by this one.
1075
1076
1077Reference section for the step-by-step guide
1078============================================
1079
1080This section holds additional information for almost all the items in the above
1081step-by-step guide.
1082
1083Preparations for building your own kernels
1084------------------------------------------
1085
1086  *The steps in this section lay the groundwork for all further tests.*
1087  [:ref:`... <introprep_bissbs>`]
1088
1089The steps in all later sections of this guide depend on those described here.
1090
1091[:ref:`back to step-by-step guide <introprep_bissbs>`].
1092
1093.. _backup_bisref:
1094
1095Prepare for emergencies
1096~~~~~~~~~~~~~~~~~~~~~~~
1097
1098  *Create a fresh backup and put system repair and restore tools at hand.*
1099  [:ref:`... <backup_bissbs>`]
1100
1101Remember, you are dealing with computers, which sometimes do unexpected things
1102-- especially if you fiddle with crucial parts like the kernel of an operating
1103system. That's what you are about to do in this process. Hence, better prepare
1104for something going sideways, even if that should not happen.
1105
1106[:ref:`back to step-by-step guide <backup_bissbs>`]
1107
1108.. _vanilla_bisref:
1109
1110Remove anything related to externally maintained kernel modules
1111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112
1113  *Remove all software that depends on externally developed kernel drivers or
1114  builds them automatically.* [:ref:`...<vanilla_bissbs>`]
1115
1116Externally developed kernel modules can easily cause trouble during a bisection.
1117
1118But there is a more important reason why this guide contains this step: most
1119kernel developers will not care about reports about regressions occurring with
1120kernels that utilize such modules. That's because such kernels are not
1121considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst
1122explains in more detail.
1123
1124[:ref:`back to step-by-step guide <vanilla_bissbs>`]
1125
1126.. _secureboot_bisref:
1127
1128Deal with techniques like Secure Boot
1129~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1130
1131  *On platforms with 'Secure Boot' or similar techniques, prepare everything to
1132  ensure the system will permit your self-compiled kernel to boot later.*
1133  [:ref:`... <secureboot_bissbs>`]
1134
1135Many modern systems allow only certain operating systems to start; that's why
1136they reject booting self-compiled kernels by default.
1137
1138You ideally deal with this by making your platform trust your self-built kernels
1139with the help of a certificate. How to do that is not described
1140here, as it requires various steps that would take the text too far away from
1141its purpose; 'Documentation/admin-guide/module-signing.rst' and various web
1142sides already explain everything needed in more detail.
1143
1144Temporarily disabling solutions like Secure Boot is another way to make your own
1145Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup
1146utility; the required steps vary a lot between machines and therefore cannot be
1147described here.
1148
1149On mainstream x86 Linux distributions there is a third and universal option:
1150disable all Secure Boot restrictions for your Linux environment. You can
1151initiate this process by running ``mokutil --disable-validation``; this will
1152tell you to create a one-time password, which is safe to write down. Now
1153restart; right after your BIOS performed all self-tests the bootloader Shim will
1154show a blue box with a message 'Press any key to perform MOK management'. Hit
1155some key before the countdown exposes, which will open a menu. Choose 'Change
1156Secure Boot state'. Shim's 'MokManager' will now ask you to enter three
1157randomly chosen characters from the one-time password specified earlier. Once
1158you provided them, confirm you really want to disable the validation.
1159Afterwards, permit MokManager to reboot the machine.
1160
1161[:ref:`back to step-by-step guide <secureboot_bissbs>`]
1162
1163.. _bootworking_bisref:
1164
1165Boot the last kernel that was working
1166~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1167
1168  *Boot into the last working kernel and briefly recheck if the feature that
1169  regressed really works.* [:ref:`...<bootworking_bissbs>`]
1170
1171This will make later steps that cover creating and trimming the configuration do
1172the right thing.
1173
1174[:ref:`back to step-by-step guide <bootworking_bissbs>`]
1175
1176.. _diskspace_bisref:
1177
1178Space requirements
1179~~~~~~~~~~~~~~~~~~
1180
1181  *Ensure to have enough free space for building Linux.*
1182  [:ref:`... <diskspace_bissbs>`]
1183
1184The numbers mentioned are rough estimates with a big extra charge to be on the
1185safe side, so often you will need less.
1186
1187If you have space constraints, be sure to hay attention to the :ref:`step about
1188debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference
1189section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk
1190space by quite a few gigabytes.
1191
1192[:ref:`back to step-by-step guide <diskspace_bissbs>`]
1193
1194.. _rangecheck_bisref:
1195
1196Bisection range
1197~~~~~~~~~~~~~~~
1198
1199  *Determine the kernel versions considered 'good' and 'bad' throughout this
1200  guide.* [:ref:`...<rangecheck_bissbs>`]
1201
1202Establishing the range of commits to be checked is mostly straightforward,
1203except when a regression occurred when switching from a release of one stable
1204series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case
1205Git will need some hand holding, as there is no straight line of descent.
1206
1207That's because with the release of 6.0 mainline carried on to 6.1 while the
1208stable series 6.0.y branched to the side. It's therefore theoretically possible
1209that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a
1210commit that went into one of the 6.0.y releases, but never hit mainline or the
12116.1.y series. Thankfully that normally should not happen due to the way the
1212stable/longterm maintainers maintain the code. It's thus pretty safe to assume
12136.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel
1214will be built and tested in the segment '2' of this guide; Git would force you
1215to do this as well, if you tried bisecting between 6.0.13 and 6.1.15.
1216
1217[:ref:`back to step-by-step guide <rangecheck_bissbs>`]
1218
1219.. _buildrequires_bisref:
1220
1221Install build requirements
1222~~~~~~~~~~~~~~~~~~~~~~~~~~
1223
1224  *Install all software required to build a Linux kernel.*
1225  [:ref:`...<buildrequires_bissbs>`]
1226
1227The kernel is pretty stand-alone, but besides tools like the compiler you will
1228sometimes need a few libraries to build one. How to install everything needed
1229depends on your Linux distribution and the configuration of the kernel you are
1230about to build.
1231
1232Here are a few examples what you typically need on some mainstream
1233distributions:
1234
1235* Arch Linux and derivatives::
1236
1237    sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \
1238      pahole perl zlib ncurses qt6-base
1239
1240* Debian, Ubuntu, and derivatives::
1241
1242    sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \
1243      libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \
1244      libncurses-dev qt6-base-dev g++
1245
1246* Fedora and derivatives::
1247
1248    sudo dnf install binutils \
1249      /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \
1250      /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}
1251
1252* openSUSE and derivatives::
1253
1254    sudo zypper install bc binutils bison dwarves flex gcc git \
1255      kernel-install-tools libelf-devel make modutils openssl openssl-devel \
1256      perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel
1257
1258These commands install a few packages that are often, but not always needed. You
1259for example might want to skip installing the development headers for ncurses,
1260which you will only need in case you later might want to adjust the kernel build
1261configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit
1262the headers of Qt6 if you do not plan to adjust the .config using 'xconfig'.
1263
1264You furthermore might need additional libraries and their development headers
1265for tasks not covered in this guide -- for example when building utilities from
1266the kernel's tools/ directory.
1267
1268[:ref:`back to step-by-step guide <buildrequires_bissbs>`]
1269
1270.. _sources_bisref:
1271
1272Download the sources using Git
1273~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1274
1275  *Retrieve the Linux mainline sources.*
1276  [:ref:`...<sources_bissbs>`]
1277
1278The step-by-step guide outlines how to download the Linux sources using a full
1279Git clone of Linus' mainline repository. There is nothing more to say about
1280that -- but there are two alternatives ways to retrieve the sources that might
1281work better for you:
1282
1283* If you have an unreliable internet connection, consider
1284  :ref:`using a 'Git bundle'<sources_bundle_bisref>`.
1285
1286* If downloading the complete repository would take too long or requires too
1287  much storage space, consider :ref:`using a 'shallow
1288  clone'<sources_shallow_bisref>`.
1289
1290.. _sources_bundle_bisref:
1291
1292Downloading Linux mainline sources using a bundle
1293"""""""""""""""""""""""""""""""""""""""""""""""""
1294
1295Use the following commands to retrieve the Linux mainline sources using a
1296bundle::
1297
1298    wget -c \
1299      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
1300    git clone --no-checkout clone.bundle ~/linux/
1301    cd ~/linux/
1302    git remote remove origin
1303    git remote add mainline \
1304      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
1305    git fetch mainline
1306    git remote add -t master stable \
1307      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1308
1309In case the 'wget' command fails, just re-execute it, it will pick up where
1310it left off.
1311
1312[:ref:`back to step-by-step guide <sources_bissbs>`]
1313[:ref:`back to section intro <sources_bisref>`]
1314
1315.. _sources_shallow_bisref:
1316
1317Downloading Linux mainline sources using a shallow clone
1318~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1319
1320First, execute the following command to retrieve the latest mainline codebase::
1321
1322    git clone -o mainline --no-checkout --depth 1 -b master \
1323      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
1324    cd ~/linux/
1325    git remote add -t master stable \
1326      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1327
1328Now deepen your clone's history to the second predecessor of the mainline
1329release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would
1330be the first predecessor and 5.18 the second -- hence deepen the history up to
1331that version::
1332
1333    git fetch --shallow-exclude=v5.18 mainline
1334
1335Afterwards add the stable Git repository as remote and all required stable
1336branches as explained in the step-by-step guide.
1337
1338Note, shallow clones have a few peculiar characteristics:
1339
1340* For bisections the history needs to be deepened a few mainline versions
1341  farther than it seems necessary, as explained above already. That's because
1342  Git otherwise will be unable to revert or describe most of the commits within
1343  a range (say 6.1..6.2), as they are internally based on earlier kernels
1344  releases (like 6.0-rc2 or 5.19-rc3).
1345
1346* This document in most places uses ``git fetch`` with ``--shallow-exclude=``
1347  to specify the earliest version you care about (or to be precise: its git
1348  tag). You alternatively can use the parameter ``--shallow-since=`` to specify
1349  an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to
1350  define the depth of the history you want to download. When using them while
1351  bisecting mainline, ensure to deepen the history to at least 7 months before
1352  the release of the mainline release your 'good' kernel is based on.
1353
1354* Be warned, when deepening your clone you might encounter an error like
1355  'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.
1356  In that case run ``git repack -d`` and try again.
1357
1358[:ref:`back to step-by-step guide <sources_bissbs>`]
1359[:ref:`back to section intro <sources_bisref>`]
1360
1361.. _oldconfig_bisref:
1362
1363Start defining the build configuration for your kernel
1364~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1365
1366  *Start preparing a kernel build configuration (the '.config' file).*
1367  [:ref:`... <oldconfig_bissbs>`]
1368
1369*Note, this is the first of multiple steps in this guide that create or modify
1370build artifacts. The commands used in this guide store them right in the source
1371tree to keep things simple. In case you prefer storing the build artifacts
1372separately, create a directory like '~/linux-builddir/' and add the parameter
1373``O=~/linux-builddir/`` to all make calls used throughout this guide. You will
1374have to point other commands there as well -- among them the ``./scripts/config
1375[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to
1376locate the right build configuration.*
1377
1378Two things can easily go wrong when creating a .config file as advised:
1379
1380* The oldconfig target will use a .config file from your build directory, if
1381  one is already present there (e.g. '~/linux/.config'). That's totally fine if
1382  that's what you intend (see next step), but in all other cases you want to
1383  delete it. This for example is important in case you followed this guide
1384  further, but due to problems come back here to redo the configuration from
1385  scratch.
1386
1387* Sometimes olddefconfig is unable to locate the .config file for your running
1388  kernel and will use defaults, as briefly outlined in the guide. In that case
1389  check if your distribution ships the configuration somewhere and manually put
1390  it in the right place (e.g. '~/linux/.config') if it does. On distributions
1391  where /proc/config.gz exists this can be achieved using this command::
1392
1393    zcat /proc/config.gz > .config
1394
1395  Once you put it there, run ``make olddefconfig`` again to adjust it to the
1396  needs of the kernel about to be built.
1397
1398Note, the olddefconfig target will set any undefined build options to their
1399default value. If you prefer to set such configuration options manually, use
1400``make oldconfig`` instead. Then for each undefined configuration option you
1401will be asked how to proceed; in case you are unsure what to answer, simply hit
1402'enter' to apply the default value. Note though that for bisections you normally
1403want to go with the defaults, as you otherwise might enable a new feature that
1404causes a problem looking like regressions (for example due to security
1405restrictions).
1406
1407Occasionally odd things happen when trying to use a config file prepared for one
1408kernel (say 6.1) on an older mainline release -- especially if it is much older
1409(say 5.15). That's one of the reasons why the previous step in the guide told
1410you to boot the kernel where everything works. If you manually add a .config
1411file you thus want to ensure it's from the working kernel and not from a one
1412that shows the regression.
1413
1414In case you want to build kernels for another machine, locate its kernel build
1415configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy
1416that file to the build machine and store it as ~/linux/.config; afterwards run
1417``make olddefconfig`` to adjust it.
1418
1419[:ref:`back to step-by-step guide <oldconfig_bissbs>`]
1420
1421.. _localmodconfig_bisref:
1422
1423Trim the build configuration for your kernel
1424~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1425
1426  *Disable any kernel modules apparently superfluous for your setup.*
1427  [:ref:`... <localmodconfig_bissbs>`]
1428
1429As explained briefly in the step-by-step guide already: with localmodconfig it
1430can easily happen that your self-built kernels will lack modules for tasks you
1431did not perform at least once before utilizing this make target. That happens
1432when a task requires kernel modules which are only autoloaded when you execute
1433it for the first time. So when you never performed that task since starting your
1434kernel the modules will not have been loaded -- and from localmodconfig's point
1435of view look superfluous, which thus disables them to reduce the amount of code
1436to be compiled.
1437
1438You can try to avoid this by performing typical tasks that often will autoload
1439additional kernel modules: start a VM, establish VPN connections, loop-mount a
1440CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external
1441devices (2FA keys, headsets, webcams, ...) as well as storage devices with file
1442systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it
1443is hard to think of everything that might be needed -- even kernel developers
1444often forget one thing or another at this point.
1445
1446Do not let that risk bother you, especially when compiling a kernel only for
1447testing purposes: everything typically crucial will be there. And if you forget
1448something important you can turn on a missing feature manually later and quickly
1449run the commands again to compile and install a kernel that has everything you
1450need.
1451
1452But if you plan to build and use self-built kernels regularly, you might want to
1453reduce the risk by recording which modules your system loads over the course of
1454a few weeks. You can automate this with `modprobed-db
1455<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to
1456point localmodconfig to the list of modules modprobed-db noticed being used::
1457
1458  yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig
1459
1460That parameter also allows you to build trimmed kernels for another machine in
1461case you copied a suitable .config over to use as base (see previous step). Just
1462run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to
1463your build's host home directory. Then run these commands instead of the one the
1464step-by-step guide mentions::
1465
1466  yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig
1467
1468[:ref:`back to step-by-step guide <localmodconfig_bissbs>`]
1469
1470.. _tagging_bisref:
1471
1472Tag the kernels about to be build
1473~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1474
1475  *Ensure all the kernels you will build are clearly identifiable using a
1476  special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]
1477
1478This allows you to differentiate your distribution's kernels from those created
1479during this process, as the file or directories for the latter will contain
1480'-local' in the name; it also helps picking the right entry in the boot menu and
1481not lose track of you kernels, as their version numbers will look slightly
1482confusing during the bisection.
1483
1484[:ref:`back to step-by-step guide <tagging_bissbs>`]
1485
1486.. _debugsymbols_bisref:
1487
1488Decide to enable or disable debug symbols
1489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1490
1491  *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]
1492
1493Having debug symbols available can be important when your kernel throws a
1494'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
1495able to find the exact place where the problem occurred in the code. But
1496collecting and embedding the needed debug information takes time and consumes
1497quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
1498trimmed with localmodconfig consumed around 5 Gigabyte of space with debug
1499symbols, but less than 1 when they were disabled. The resulting kernel image and
1500modules are bigger as well, which increases storage requirements for /boot/ and
1501load times.
1502
1503In case you want a small kernel and are unlikely to decode a stack trace later,
1504you thus might want to disable debug symbols to avoid those downsides. If it
1505later turns out that you need them, just enable them as shown and rebuild the
1506kernel.
1507
1508You on the other hand definitely want to enable them for this process, if there
1509is a decent chance that you need to decode a stack trace later. The section
1510'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst
1511explains this process in more detail.
1512
1513[:ref:`back to step-by-step guide <debugsymbols_bissbs>`]
1514
1515.. _configmods_bisref:
1516
1517Adjust build configuration
1518~~~~~~~~~~~~~~~~~~~~~~~~~~
1519
1520  *Check if you may want or need to adjust some other kernel configuration
1521  options:*
1522
1523Depending on your needs you at this point might want or have to adjust some
1524kernel configuration options.
1525
1526.. _configmods_distros_bisref:
1527
1528Distro specific adjustments
1529"""""""""""""""""""""""""""
1530
1531  *Are you running* [:ref:`... <configmods_bissbs>`]
1532
1533The following sections help you to avoid build problems that are known to occur
1534when following this guide on a few commodity distributions.
1535
1536**Debian:**
1537
1538* Remove a stale reference to a certificate file that would cause your build to
1539  fail::
1540
1541   ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''
1542
1543  Alternatively, download the needed certificate and make that configuration
1544  option point to it, as `the Debian handbook explains in more detail
1545  <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_
1546  -- or generate your own, as explained in
1547  Documentation/admin-guide/module-signing.rst.
1548
1549[:ref:`back to step-by-step guide <configmods_bissbs>`]
1550
1551.. _configmods_individual_bisref:
1552
1553Individual adjustments
1554""""""""""""""""""""""
1555
1556  *If you want to influence the other aspects of the configuration, do so
1557  now.* [:ref:`... <configmods_bissbs>`]
1558
1559At this point you can use a command like ``make menuconfig`` or ``make nconfig``
1560to enable or disable certain features using a text-based user interface; to use
1561a graphical configuration utility, run ``make xconfig`` instead. Both of them
1562require development libraries from toolkits they are rely on (ncurses
1563respectively Qt5 or Qt6); an error message will tell you if something required
1564is missing.
1565
1566[:ref:`back to step-by-step guide <configmods_bissbs>`]
1567
1568.. _saveconfig_bisref:
1569
1570Put the .config file aside
1571~~~~~~~~~~~~~~~~~~~~~~~~~~
1572
1573  *Reprocess the .config after the latest changes and store it in a safe place.*
1574  [:ref:`... <saveconfig_bissbs>`]
1575
1576Put the .config you prepared aside, as you want to copy it back to the build
1577directory every time during this guide before you start building another
1578kernel. That's because going back and forth between different versions can alter
1579.config files in odd ways; those occasionally cause side effects that could
1580confuse testing or in some cases render the result of your bisection
1581meaningless.
1582
1583[:ref:`back to step-by-step guide <saveconfig_bissbs>`]
1584
1585.. _introlatestcheck_bisref:
1586
1587Try to reproduce the problem with the latest codebase
1588-----------------------------------------------------
1589
1590  *Verify the regression is not caused by some .config change and check if it
1591  still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]
1592
1593For some readers it might seem unnecessary to check the latest codebase at this
1594point, especially if you did that already with a kernel prepared by your
1595distributor or face a regression within a stable/longterm series. But it's
1596highly recommended for these reasons:
1597
1598* You will run into any problems caused by your setup before you actually begin
1599  a bisection. That will make it a lot easier to differentiate between 'this
1600  most likely is some problem in my setup' and 'this change needs to be skipped
1601  during the bisection, as the kernel sources at that stage contain an unrelated
1602  problem that causes building or booting to fail'.
1603
1604* These steps will rule out if your problem is caused by some change in the
1605  build configuration between the 'working' and the 'broken' kernel. This for
1606  example can happen when your distributor enabled an additional security
1607  feature in the newer kernel which was disabled or not yet supported by the
1608  older kernel. That security feature might get into the way of something you
1609  do -- in which case your problem from the perspective of the Linux kernel
1610  upstream developers is not a regression, as
1611  Documentation/admin-guide/reporting-regressions.rst explains in more detail.
1612  You thus would waste your time if you'd try to bisect this.
1613
1614* If the cause for your regression was already fixed in the latest mainline
1615  codebase, you'd perform the bisection for nothing. This holds true for a
1616  regression you encountered with a stable/longterm release as well, as they are
1617  often caused by problems in mainline changes that were backported -- in which
1618  case the problem will have to be fixed in mainline first. Maybe it already was
1619  fixed there and the fix is already in the process of being backported.
1620
1621* For regressions within a stable/longterm series it's furthermore crucial to
1622  know if the issue is specific to that series or also happens in the mainline
1623  kernel, as the report needs to be sent to different people:
1624
1625  * Regressions specific to a stable/longterm series are the stable team's
1626    responsibility; mainline Linux developers might or might not care.
1627
1628  * Regressions also happening in mainline are something the regular Linux
1629    developers and maintainers have to handle; the stable team does not care
1630    and does not need to be involved in the report, they just should be told
1631    to backport the fix once it's ready.
1632
1633  Your report might be ignored if you send it to the wrong party -- and even
1634  when you get a reply there is a decent chance that developers tell you to
1635  evaluate which of the two cases it is before they take a closer look.
1636
1637[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]
1638
1639.. _checkoutmaster_bisref:
1640
1641Check out the latest Linux codebase
1642~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1643
1644  *Check out the latest Linux codebase.*
1645  [:ref:`... <checkoutmaster_bissbs>`]
1646
1647In case you later want to recheck if an ever newer codebase might fix the
1648problem, remember to run that ``git fetch --shallow-exclude [...]`` command
1649again mentioned earlier to update your local Git repository.
1650
1651[:ref:`back to step-by-step guide <checkoutmaster_bissbs>`]
1652
1653.. _build_bisref:
1654
1655Build your kernel
1656~~~~~~~~~~~~~~~~~
1657
1658  *Build the image and the modules of your first kernel using the config file
1659  you prepared.* [:ref:`... <build_bissbs>`]
1660
1661A lot can go wrong at this stage, but the instructions below will help you help
1662yourself. Another subsection explains how to directly package your kernel up as
1663deb, rpm or tar file.
1664
1665Dealing with build errors
1666"""""""""""""""""""""""""
1667
1668When a build error occurs, it might be caused by some aspect of your machine's
1669setup that often can be fixed quickly; other times though the problem lies in
1670the code and can only be fixed by a developer. A close examination of the
1671failure messages coupled with some research on the internet will often tell you
1672which of the two it is. To perform such investigation, restart the build
1673process like this::
1674
1675  make V=1
1676
1677The ``V=1`` activates verbose output, which might be needed to see the actual
1678error. To make it easier to spot, this command also omits the ``-j $(nproc
1679--all)`` used earlier to utilize every CPU core in the system for the job -- but
1680this parallelism also results in some clutter when failures occur.
1681
1682After a few seconds the build process should run into the error again. Now try
1683to find the most crucial line describing the problem. Then search the internet
1684for the most important and non-generic section of that line (say 4 to 8 words);
1685avoid or remove anything that looks remotely system-specific, like your username
1686or local path names like ``/home/username/linux/``. First try your regular
1687internet search engine with that string, afterwards search Linux kernel mailing
1688lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.
1689
1690This most of the time will find something that will explain what is wrong; quite
1691often one of the hits will provide a solution for your problem, too. If you
1692do not find anything that matches your problem, try again from a different angle
1693by modifying your search terms or using another line from the error messages.
1694
1695In the end, most issues you run into have likely been encountered and
1696reported by others already. That includes issues where the cause is not your
1697system, but lies in the code. If you run into one of those, you might thus find
1698a solution (e.g. a patch) or workaround for your issue, too.
1699
1700Package your kernel up
1701""""""""""""""""""""""
1702
1703The step-by-step guide uses the default make targets (e.g. 'bzImage' and
1704'modules' on x86) to build the image and the modules of your kernel, which later
1705steps of the guide then install. You instead can also directly build everything
1706and directly package it up by using one of the following targets:
1707
1708* ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package
1709
1710* ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package
1711
1712* ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball
1713
1714This is just a selection of available make targets for this purpose, see
1715``make help`` for others. You can also use these targets after running
1716``make -j $(nproc --all)``, as they will pick up everything already built.
1717
1718If you employ the targets to generate deb or rpm packages, ignore the
1719step-by-step guide's instructions on installing and removing your kernel;
1720instead install and remove the packages using the package utility for the format
1721(e.g. dpkg and rpm) or a package management utility build on top of them (apt,
1722aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using
1723these two make targets are designed to work on various distributions utilizing
1724those formats, they thus will sometimes behave differently than your
1725distribution's kernel packages.
1726
1727[:ref:`back to step-by-step guide <build_bissbs>`]
1728
1729.. _install_bisref:
1730
1731Put the kernel in place
1732~~~~~~~~~~~~~~~~~~~~~~~
1733
1734  *Install the kernel you just built.* [:ref:`... <install_bissbs>`]
1735
1736What you need to do after executing the command in the step-by-step guide
1737depends on the existence and the implementation of ``/sbin/installkernel``
1738executable on your distribution.
1739
1740If installkernel is found, the kernel's build system will delegate the actual
1741installation of your kernel image to this executable, which then performs some
1742or all of these tasks:
1743
1744* On almost all Linux distributions installkernel will store your kernel's
1745  image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will
1746  put a 'System.map-<kernelrelease_id>' alongside it.
1747
1748* On most distributions installkernel will then generate an 'initramfs'
1749  (sometimes also called 'initrd'), which usually are stored as
1750  '/boot/initramfs-<kernelrelease_id>.img' or
1751  '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file
1752  for booting, hence ensure to execute the make target 'modules_install' first,
1753  as your distribution's initramfs generator otherwise will be unable to find
1754  the modules that go into the image.
1755
1756* On some distributions installkernel will then add an entry for your kernel
1757  to your bootloader's configuration.
1758
1759You have to take care of some or all of the tasks yourself, if your
1760distribution lacks an installkernel script or does only handle part of them.
1761Consult the distribution's documentation for details. If in doubt, install the
1762kernel manually::
1763
1764   sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
1765   sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)
1766
1767Now generate your initramfs using the tools your distribution provides for this
1768process. Afterwards add your kernel to your bootloader configuration and reboot.
1769
1770[:ref:`back to step-by-step guide <install_bissbs>`]
1771
1772.. _storagespace_bisref:
1773
1774Storage requirements per kernel
1775~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1776
1777  *Check how much storage space the kernel, its modules, and other related files
1778  like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]
1779
1780The kernels built during a bisection consume quite a bit of space in /boot/ and
1781/lib/modules/, especially if you enabled debug symbols. That makes it easy to
1782fill up volumes during a bisection -- and due to that even kernels which used to
1783work earlier might fail to boot. To prevent that you will need to know how much
1784space each installed kernel typically requires.
1785
1786Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in
1787the guide will match all files needed to boot your kernel -- but neither the
1788path nor the naming scheme are mandatory. On some distributions you thus will
1789need to look in different places.
1790
1791[:ref:`back to step-by-step guide <storagespace_bissbs>`]
1792
1793.. _tainted_bisref:
1794
1795Check if your newly built kernel considers itself 'tainted'
1796~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1797
1798  *Check if the kernel marked itself as 'tainted'.*
1799  [:ref:`... <tainted_bissbs>`]
1800
1801Linux marks itself as tainted when something happens that potentially leads to
1802follow-up errors that look totally unrelated. That is why developers might
1803ignore or react scantly to reports from tainted kernels -- unless of course the
1804kernel set the flag right when the reported bug occurred.
1805
1806That's why you want check why a kernel is tainted as explained in
1807Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own
1808interest, as your testing might be flawed otherwise.
1809
1810[:ref:`back to step-by-step guide <tainted_bissbs>`]
1811
1812.. _recheckbroken_bisref:
1813
1814Check the kernel built from a recent mainline codebase
1815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1816
1817  *Verify if your bug occurs with the newly built kernel.*
1818  [:ref:`... <recheckbroken_bissbs>`]
1819
1820There are a couple of reasons why your bug or regression might not show up with
1821the kernel you built from the latest codebase. These are the most frequent:
1822
1823* The bug was fixed meanwhile.
1824
1825* What you suspected to be a regression was caused by a change in the build
1826  configuration the provider of your kernel carried out.
1827
1828* Your problem might be a race condition that does not show up with your kernel;
1829  the trimmed build configuration, a different setting for debug symbols, the
1830  compiler used, and various other things can cause this.
1831
1832* In case you encountered the regression with a stable/longterm kernel it might
1833  be a problem that is specific to that series; the next step in this guide will
1834  check this.
1835
1836[:ref:`back to step-by-step guide <recheckbroken_bissbs>`]
1837
1838.. _recheckstablebroken_bisref:
1839
1840Check the kernel built from the latest stable/longterm codebase
1841~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1842
1843  *Are you facing a regression within a stable/longterm release, but failed to
1844  reproduce it with the kernel you just built using the latest mainline sources?
1845  Then check if the latest codebase for the particular series might already fix
1846  the problem.* [:ref:`... <recheckstablebroken_bissbs>`]
1847
1848If this kernel does not show the regression either, there most likely is no need
1849for a bisection.
1850
1851[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]
1852
1853.. _introworkingcheck_bisref:
1854
1855Ensure the 'good' version is really working well
1856------------------------------------------------
1857
1858  *Check if the kernels you build work fine.*
1859  [:ref:`... <introworkingcheck_bissbs>`]
1860
1861This section will reestablish a known working base. Skipping it might be
1862appealing, but is usually a bad idea, as it does something important:
1863
1864It will ensure the .config file you prepared earlier actually works as expected.
1865That is in your own interest, as trimming the configuration is not foolproof --
1866and you might be building and testing ten or more kernels for nothing before
1867starting to suspect something might be wrong with the build configuration.
1868
1869That alone is reason enough to spend the time on this, but not the only reason.
1870
1871Many readers of this guide normally run kernels that are patched, use add-on
1872modules, or both. Those kernels thus are not considered 'vanilla' -- therefore
1873it's possible that the thing that regressed might never have worked in vanilla
1874builds of the 'good' version in the first place.
1875
1876There is a third reason for those that noticed a regression between
1877stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will
1878ensure the kernel version you assumed to be 'good' earlier in the process (e.g.
18796.0) actually is working.
1880
1881[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]
1882
1883.. _recheckworking_bisref:
1884
1885Build your own version of the 'good' kernel
1886~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1887
1888  *Build your own variant of the working kernel and check if the feature that
1889  regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]
1890
1891In case the feature that broke with newer kernels does not work with your first
1892self-built kernel, find and resolve the cause before moving on. There are a
1893multitude of reasons why this might happen. Some ideas where to look:
1894
1895* Check the taint status and the output of ``dmesg``, maybe something unrelated
1896  went wrong.
1897
1898* Maybe localmodconfig did something odd and disabled the module required to
1899  test the feature? Then you might want to recreate a .config file based on the
1900  one from the last working kernel and skip trimming it down; manually disabling
1901  some features in the .config might work as well to reduce the build time.
1902
1903* Maybe it's not a kernel regression and something that is caused by some fluke,
1904  a broken initramfs (also known as initrd), new firmware files, or an updated
1905  userland software?
1906
1907* Maybe it was a feature added to your distributor's kernel which vanilla Linux
1908  at that point never supported?
1909
1910Note, if you found and fixed problems with the .config file, you want to use it
1911to build another kernel from the latest codebase, as your earlier tests with
1912mainline and the latest version from an affected stable/longterm series were
1913most likely flawed.
1914
1915[:ref:`back to step-by-step guide <recheckworking_bissbs>`]
1916
1917Perform a bisection and validate the result
1918-------------------------------------------
1919
1920  *With all the preparations and precaution builds taken care of, you are now
1921  ready to begin the bisection.* [:ref:`... <introbisect_bissbs>`]
1922
1923The steps in this segment perform and validate the bisection.
1924
1925[:ref:`back to step-by-step guide <introbisect_bissbs>`].
1926
1927.. _bisectstart_bisref:
1928
1929Start the bisection
1930~~~~~~~~~~~~~~~~~~~
1931
1932  *Start the bisection and tell Git about the versions earlier established as
1933  'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]
1934
1935This will start the bisection process; the last of the commands will make Git
1936check out a commit round about half-way between the 'good' and the 'bad' changes
1937for you to test.
1938
1939[:ref:`back to step-by-step guide <bisectstart_bissbs>`]
1940
1941.. _bisectbuild_bisref:
1942
1943Build a kernel from the bisection point
1944~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1945
1946  *Build, install, and boot a kernel from the code Git checked out using the
1947  same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]
1948
1949There are two things worth of note here:
1950
1951* Occasionally building the kernel will fail or it might not boot due some
1952  problem in the code at the bisection point. In that case run this command::
1953
1954    git bisect skip
1955
1956  Git will then check out another commit nearby which with a bit of luck should
1957  work better. Afterwards restart executing this step.
1958
1959* Those slightly odd looking version identifiers can happen during bisections,
1960  because the Linux kernel subsystems prepare their changes for a new mainline
1961  release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus
1962  base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then
1963  get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This
1964  leads to those slightly odd looking version identifiers coming up during
1965  bisections.
1966
1967[:ref:`back to step-by-step guide <bisectbuild_bissbs>`]
1968
1969.. _bisecttest_bisref:
1970
1971Bisection checkpoint
1972~~~~~~~~~~~~~~~~~~~~
1973
1974  *Check if the feature that regressed works in the kernel you just built.*
1975  [:ref:`... <bisecttest_bissbs>`]
1976
1977Ensure what you tell Git is accurate: getting it wrong just one time will bring
1978the rest of the bisection totally off course, hence all testing after that point
1979will be for nothing.
1980
1981[:ref:`back to step-by-step guide <bisecttest_bissbs>`]
1982
1983.. _bisectlog_bisref:
1984
1985Put the bisection log away
1986~~~~~~~~~~~~~~~~~~~~~~~~~~
1987
1988  *Store Git's bisection log and the current .config file in a safe place.*
1989  [:ref:`... <bisectlog_bissbs>`]
1990
1991As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will
1992render the end result of a bisection useless. In that case you'd normally have
1993to restart the bisection from scratch. The log can prevent that, as it might
1994allow someone to point out where a bisection likely went sideways -- and then
1995instead of testing ten or more kernels you might only have to build a few to
1996resolve things.
1997
1998The .config file is put aside, as there is a decent chance that developers might
1999ask for it after you report the regression.
2000
2001[:ref:`back to step-by-step guide <bisectlog_bissbs>`]
2002
2003.. _revert_bisref:
2004
2005Try reverting the culprit
2006~~~~~~~~~~~~~~~~~~~~~~~~~
2007
2008  *Try reverting the culprit on top of the latest codebase to see if this fixes
2009  your regression.* [:ref:`... <revert_bissbs>`]
2010
2011This is an optional step, but whenever possible one you should try: there is a
2012decent chance that developers will ask you to perform this step when you bring
2013the bisection result up. So give it a try, you are in the flow already, building
2014one more kernel shouldn't be a big deal at this point.
2015
2016The step-by-step guide covers everything relevant already except one slightly
2017rare thing: did you bisected a regression that also happened with mainline using
2018a stable/longterm series, but Git failed to revert the commit in mainline? Then
2019try to revert the culprit in the affected stable/longterm series -- and if that
2020succeeds, test that kernel version instead.
2021
2022[:ref:`back to step-by-step guide <revert_bissbs>`]
2023
2024Cleanup steps during and after following this guide
2025---------------------------------------------------
2026
2027  *During and after following this guide you might want or need to remove some
2028  of the kernels you installed.* [:ref:`... <introclosure_bissbs>`]
2029
2030The steps in this section describe clean-up procedures.
2031
2032[:ref:`back to step-by-step guide <introclosure_bissbs>`].
2033
2034.. _makeroom_bisref:
2035
2036Cleaning up during the bisection
2037~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2038
2039  *To remove one of the kernels you installed, look up its 'kernelrelease'
2040  identifier.* [:ref:`... <makeroom_bissbs>`]
2041
2042The kernels you install during this process are easy to remove later, as its
2043parts are only stored in two places and clearly identifiable. You thus do not
2044need to worry to mess up your machine when you install a kernel manually (and
2045thus bypass your distribution's packaging system): all parts of your kernels are
2046relatively easy to remove later.
2047
2048One of the two places is a directory in /lib/modules/, which holds the modules
2049for each installed kernel. This directory is named after the kernel's release
2050identifier; hence, to remove all modules for one of the kernels you built,
2051simply remove its modules directory in /lib/modules/.
2052
2053The other place is /boot/, where typically two up to five files will be placed
2054during installation of a kernel. All of them usually contain the release name in
2055their file name, but how many files and their exact names depend somewhat on
2056your distribution's installkernel executable and its initramfs generator. On
2057some distributions the ``kernel-install remove...`` command mentioned in the
2058step-by-step guide will delete all of these files for you while also removing
2059the menu entry for the kernel from your bootloader configuration. On others you
2060have to take care of these two tasks yourself. The following command should
2061interactively remove the three main files of a kernel with the release name
2062'6.0-rc1-local-gcafec0cacaca0'::
2063
2064  rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0
2065
2066Afterwards check for other files in /boot/ that have
2067'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.
2068Now remove the boot entry for the kernel from your bootloader's configuration;
2069the steps to do that vary quite a bit between Linux distributions.
2070
2071Note, be careful with wildcards like '*' when deleting files or directories
2072for kernels manually: you might accidentally remove files of a 6.0.13 kernel
2073when all you want is to remove 6.0 or 6.0.1.
2074
2075[:ref:`back to step-by-step guide <makeroom_bissbs>`]
2076
2077Cleaning up after the bisection
2078~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2079
2080.. _finishingtouch_bisref:
2081
2082  *Once you have finished the bisection, do not immediately remove anything
2083  you set up, as you might need a few things again.*
2084  [:ref:`... <finishingtouch_bissbs>`]
2085
2086When you are really short of storage space removing the kernels as described in
2087the step-by-step guide might not free as much space as you would like. In that
2088case consider running ``rm -rf ~/linux/*`` as well now. This will remove the
2089build artifacts and the Linux sources, but will leave the Git repository
2090(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the
2091sources back.
2092
2093Removing the repository as well would likely be unwise at this point: there
2094is a decent chance developers will ask you to build another kernel to
2095perform additional tests -- like testing a debug patch or a proposed fix.
2096Details on how to perform those can be found in the section :ref:`Optional
2097tasks: test reverts, patches, or later versions <introoptional_bissbs>`.
2098
2099Additional tests are also the reason why you want to keep the
2100~/kernel-config-working file around for a few weeks.
2101
2102[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]
2103
2104.. _introoptional_bisref:
2105
2106Test reverts, patches, or later versions
2107----------------------------------------
2108
2109  *While or after reporting a bug, you might want or potentially will be asked
2110  to test reverts, patches, proposed fixes, or other versions.*
2111  [:ref:`... <introoptional_bissbs>`]
2112
2113All the commands used in this section should be pretty straight forward, so
2114there is not much to add except one thing: when setting a kernel tag as
2115instructed, ensure it is not much longer than the one used in the example, as
2116problems will arise if the kernelrelease identifier exceeds 63 characters.
2117
2118[:ref:`back to step-by-step guide <introoptional_bissbs>`].
2119
2120
2121Additional information
2122======================
2123
2124.. _buildhost_bis:
2125
2126Build kernels on a different machine
2127------------------------------------
2128
2129To compile kernels on another system, slightly alter the step-by-step guide's
2130instructions:
2131
2132* Start following the guide on the machine where you want to install and test
2133  the kernels later.
2134
2135* After executing ':ref:`Boot into the working kernel and briefly use the
2136  apparently broken feature <bootworking_bissbs>`', save the list of loaded
2137  modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the
2138  build configuration for the running kernel (see ':ref:`Start defining the
2139  build configuration for your kernel <oldconfig_bisref>`' for hints on where
2140  to find it) and store it as '~/test-machine-config-working'. Transfer both
2141  files to the home directory of your build host.
2142
2143* Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough
2144  free space for building [...] <diskspace_bissbs>`').
2145
2146* When you reach ':ref:`Start preparing a kernel build configuration[...]
2147  <oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time,
2148  execute the following command to base your configuration on the one from the
2149  test machine's 'working' kernel::
2150
2151    cp ~/test-machine-config-working ~/linux/.config
2152
2153* During the next step to ':ref:`disable any apparently superfluous kernel
2154  modules <localmodconfig_bissbs>`' use the following command instead::
2155
2156    yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig
2157
2158* Continue the guide, but ignore the instructions outlining how to compile,
2159  install, and reboot into a kernel every time they come up. Instead build
2160  like this::
2161
2162    cp ~/kernel-config-working .config
2163    make olddefconfig &&
2164    make -j $(nproc --all) targz-pkg
2165
2166  This will generate a gzipped tar file whose name is printed in the last
2167  line shown; for example, a kernel with the kernelrelease identifier
2168  '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will
2169  be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.
2170
2171  Copy that file to your test machine's home directory.
2172
2173* Switch to the test machine to check if you have enough space to hold another
2174  kernel. Then extract the file you transferred::
2175
2176    sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /
2177
2178  Afterwards :ref:`generate the initramfs and add the kernel to your boot
2179  loader's configuration <install_bisref>`; on some distributions the following
2180  command will take care of both these tasks::
2181
2182    sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423
2183
2184  Now reboot and ensure you started the intended kernel.
2185
2186This approach even works when building for another architecture: just install
2187cross-compilers and add the appropriate parameters to every invocation of make
2188(e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).
2189
2190Additional reading material
2191---------------------------
2192
2193* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
2194  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
2195  in the Git documentation.
2196* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
2197  from kernel developer Nathan Chancellor.
2198* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
2199* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
2200
2201..
2202   end-of-content
2203..
2204   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
2205   you spot a typo or small mistake, feel free to let him know directly and
2206   he'll fix it. You are free to do the same in a mostly informal way if you
2207   want to contribute changes to the text -- but for copyright reasons please CC
2208   linux-doc@vger.kernel.org and 'sign-off' your contribution as
2209   Documentation/process/submitting-patches.rst explains in the section 'Sign
2210   your work - the Developer's Certificate of Origin'.
2211..
2212   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
2213   of the file. If you want to distribute this text under CC-BY-4.0 only,
2214   please use 'The Linux kernel development community' for author attribution
2215   and link this as source:
2216   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
2217
2218..
2219   Note: Only the content of this RST file as found in the Linux kernel sources
2220   is available under CC-BY-4.0, as versions of this text that were processed
2221   (for example by the kernel's build system) might contain content taken from
2222   files which use a more restrictive license.