faq.rst

  1.. SPDX-License-Identifier: GPL-2.0
  2
  3==========================
  4Frequently Asked Questions
  5==========================
  6
  7How is this different from Autotest, kselftest, and so on?
  8==========================================================
  9KUnit is a unit testing framework. Autotest, kselftest (and some others) are
 10not.
 11
 12A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is supposed to
 13test a single unit of code in isolation and hence the name *unit test*. A unit
 14test should be the finest granularity of testing and should allow all possible
 15code paths to be tested in the code under test. This is only possible if the
 16code under test is small and does not have any external dependencies outside of
 17the test's control like hardware.
 18
 19There are no testing frameworks currently available for the kernel that do not
 20require installing the kernel on a test machine or in a virtual machine. All
 21testing frameworks require tests to be written in userspace and run on the
 22kernel under test. This is true for Autotest, kselftest, and some others,
 23disqualifying any of them from being considered unit testing frameworks.
 24
 25Does KUnit support running on architectures other than UML?
 26===========================================================
 27
 28Yes, mostly.
 29
 30For the most part, the KUnit core framework (what we use to write the tests)
 31can compile to any architecture. It compiles like just another part of the
 32kernel and runs when the kernel boots, or when built as a module, when the
 33module is loaded.  However, there is infrastructure, like the KUnit Wrapper
 34(``tools/testing/kunit/kunit.py``) that might not support some architectures
 35(see :ref:`kunit-on-qemu`).
 36
 37In short, yes, you can run KUnit on other architectures, but it might require
 38more work than using KUnit on UML.
 39
 40For more information, see :ref:`kunit-on-non-uml`.
 41
 42.. _kinds-of-tests:
 43
 44What is the difference between a unit test and other kinds of tests?
 45====================================================================
 46Most existing tests for the Linux kernel would be categorized as an integration
 47test, or an end-to-end test.
 48
 49- A unit test is supposed to test a single unit of code in isolation. A unit
 50  test should be the finest granularity of testing and, as such, allows all
 51  possible code paths to be tested in the code under test. This is only possible
 52  if the code under test is small and does not have any external dependencies
 53  outside of the test's control like hardware.
 54- An integration test tests the interaction between a minimal set of components,
 55  usually just two or three. For example, someone might write an integration
 56  test to test the interaction between a driver and a piece of hardware, or to
 57  test the interaction between the userspace libraries the kernel provides and
 58  the kernel itself. However, one of these tests would probably not test the
 59  entire kernel along with hardware interactions and interactions with the
 60  userspace.
 61- An end-to-end test usually tests the entire system from the perspective of the
 62  code under test. For example, someone might write an end-to-end test for the
 63  kernel by installing a production configuration of the kernel on production
 64  hardware with a production userspace and then trying to exercise some behavior
 65  that depends on interactions between the hardware, the kernel, and userspace.
 66
 67KUnit is not working, what should I do?
 68=======================================
 69
 70Unfortunately, there are a number of things which can break, but here are some
 71things to try.
 72
 731. Run ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output``
 74   parameter. This might show details or error messages hidden by the kunit_tool
 75   parser.
 762. Instead of running ``kunit.py run``, try running ``kunit.py config``,
 77   ``kunit.py build``, and ``kunit.py exec`` independently. This can help track
 78   down where an issue is occurring. (If you think the parser is at fault, you
 79   can run it manually against ``stdin`` or a file with ``kunit.py parse``.)
 803. Running the UML kernel directly can often reveal issues or error messages,
 81   ``kunit_tool`` ignores. This should be as simple as running ``./vmlinux``
 82   after building the UML kernel (for example, by using ``kunit.py build``).
 83   Note that UML has some unusual requirements (such as the host having a tmpfs
 84   filesystem mounted), and has had issues in the past when built statically and
 85   the host has KASLR enabled. (On older host kernels, you may need to run
 86   ``setarch `uname -m` -R ./vmlinux`` to disable KASLR.)
 874. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
 88   (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
 89   around, so you can see what config was used after running ``kunit.py run``.
 90   It also preserves any config changes you might make, so you can
 91   enable/disable things with ``make ARCH=um menuconfig`` or similar, and then
 92   re-run kunit_tool.
 935. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This
 94   may help clean up any residual config items which could be causing problems.
 956. Finally, try running KUnit outside UML. KUnit and KUnit tests can be
 96   built into any kernel, or can be built as a module and loaded at runtime.
 97   Doing so should allow you to determine if UML is causing the issue you're
 98   seeing. When tests are built-in, they will execute when the kernel boots, and
 99   modules will automatically execute associated tests when loaded. Test results
100   can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
101   can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`.
102
103If none of the above tricks help, you are always welcome to email any issues to
104kunit-dev@googlegroups.com.