automated_testing.rst

  1.. SPDX-License-Identifier: GPL-2.0+
  2
  3=========================================
  4Automated testing of the DRM subsystem
  5=========================================
  6
  7Introduction
  8============
  9
 10Making sure that changes to the core or drivers don't introduce regressions can
 11be very time-consuming when lots of different hardware configurations need to
 12be tested. Moreover, it isn't practical for each person interested in this
 13testing to have to acquire and maintain what can be a considerable amount of
 14hardware.
 15
 16Also, it is desirable for developers to check for regressions in their code by
 17themselves, instead of relying on the maintainers to find them and then
 18reporting back.
 19
 20There are facilities in gitlab.freedesktop.org to automatically test Mesa that
 21can be used as well for testing the DRM subsystem. This document explains how
 22people interested in testing it can use this shared infrastructure to save
 23quite some time and effort.
 24
 25
 26Relevant files
 27==============
 28
 29drivers/gpu/drm/ci/gitlab-ci.yml
 30--------------------------------
 31
 32This is the root configuration file for GitLab CI. Among other less interesting
 33bits, it specifies the specific version of the scripts to be used. There are
 34some variables that can be modified to change the behavior of the pipeline:
 35
 36DRM_CI_PROJECT_PATH
 37    Repository that contains the Mesa software infrastructure for CI
 38
 39DRM_CI_COMMIT_SHA
 40    A particular revision to use from that repository
 41
 42UPSTREAM_REPO
 43    URL to git repository containing the target branch
 44
 45TARGET_BRANCH
 46    Branch to which this branch is to be merged into
 47
 48IGT_VERSION
 49    Revision of igt-gpu-tools being used, from
 50    https://gitlab.freedesktop.org/drm/igt-gpu-tools
 51
 52drivers/gpu/drm/ci/testlist.txt
 53-------------------------------
 54
 55IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
 56file, see below).
 57
 58drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
 59----------------------------------------------------------
 60
 61Lists the known failures for a given driver on a specific hardware revision.
 62
 63drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
 64-----------------------------------------------------------
 65
 66Lists the tests that for a given driver on a specific hardware revision are
 67known to behave unreliably. These tests won't cause a job to fail regardless of
 68the result. They will still be run.
 69
 70Each new flake entry must be associated with a link to the email reporting the
 71bug to the author of the affected driver or the relevant GitLab issue. The entry
 72must also include the board name or Device Tree name, the first kernel version
 73affected, the IGT version used for tests, and an approximation of the failure rate.
 74
 75They should be provided under the following format::
 76
 77  # Bug Report: $LORE_URL_OR_GITLAB_ISSUE
 78  # Board Name: broken-board.dtb
 79  # Linux Version: 6.6-rc1
 80  # IGT Version: 1.28-gd2af13d9f
 81  # Failure Rate: 100
 82  flaky-test
 83
 84Use the appropriate link below to create a GitLab issue:
 85amdgpu driver: https://gitlab.freedesktop.org/drm/amd/-/issues
 86i915 driver: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues
 87msm driver: https://gitlab.freedesktop.org/drm/msm/-/issues
 88xe driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues
 89
 90drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
 91-----------------------------------------------------------
 92
 93Lists the tests that won't be run for a given driver on a specific hardware
 94revision. These are usually tests that interfere with the running of the test
 95list due to hanging the machine, causing OOM, taking too long, etc.
 96
 97
 98How to enable automated testing on your tree
 99============================================
100
1011. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
102yet
103
1042. In your kernel repo's configuration (eg.
105https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
106CI/CD configuration file from .gitlab-ci.yml to
107drivers/gpu/drm/ci/gitlab-ci.yml.
108
1093. Request to be added to the drm/ci-ok group so that your user has the
110necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok
111
1124. Next time you push to this repository, you will see a CI pipeline being
113created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
114
1155. The various jobs will be run and when the pipeline is finished, all jobs
116should be green unless a regression has been found.
117
1186. Warnings in the pipeline indicate that lockdep
119(see Documentation/locking/lockdep-design.rst) issues have been detected
120during the tests.
121
122
123How to update test expectations
124===============================
125
126If your changes to the code fix any tests, you will have to remove one or more
127lines from one or more of the files in
128drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
129affected by the change.
130
131
132How to expand coverage
133======================
134
135If your code changes make it possible to run more tests (by solving reliability
136issues, for example), you can remove tests from the flakes and/or skips lists,
137and then the expected results if there are any known failures.
138
139If there is a need for updating the version of IGT being used (maybe you have
140added more tests to it), update the IGT_VERSION variable at the top of the
141gitlab-ci.yml file.
142
143
144How to test your changes to the scripts
145=======================================
146
147For testing changes to the scripts in the drm-ci repo, change the
148DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
149drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
150janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
151
152
153How to incorporate external fixes in your testing
154=================================================
155
156Often, regressions in other trees will prevent testing changes local to the
157tree under test. These fixes will be automatically merged in during the build
158jobs from a branch in the target tree that is named as
159${TARGET_BRANCH}-external-fixes.
160
161If the pipeline is not in a merge request and a branch with the same name
162exists in the local tree, commits from that branch will be merged in as well.
163
164
165How to deal with automated testing labs that may be down
166========================================================
167
168If a hardware farm is down and thus causing pipelines to fail that would
169otherwise pass, one can disable all jobs that would be submitted to that farm
170by editing the file at
171https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.