delay-accounting.rst

  1================
  2Delay accounting
  3================
  4
  5Tasks encounter delays in execution when they wait
  6for some kernel resource to become available e.g. a
  7runnable task may wait for a free CPU to run on.
  8
  9The per-task delay accounting functionality measures
 10the delays experienced by a task while
 11
 12a) waiting for a CPU (while being runnable)
 13b) completion of synchronous block I/O initiated by the task
 14c) swapping in pages
 15d) memory reclaim
 16e) thrashing
 17f) direct compact
 18g) write-protect copy
 19h) IRQ/SOFTIRQ
 20
 21and makes these statistics available to userspace through
 22the taskstats interface.
 23
 24Such delays provide feedback for setting a task's cpu priority,
 25io priority and rss limit values appropriately. Long delays for
 26important tasks could be a trigger for raising its corresponding priority.
 27
 28The functionality, through its use of the taskstats interface, also provides
 29delay statistics aggregated for all tasks (or threads) belonging to a
 30thread group (corresponding to a traditional Unix process). This is a commonly
 31needed aggregation that is more efficiently done by the kernel.
 32
 33Userspace utilities, particularly resource management applications, can also
 34aggregate delay statistics into arbitrary groups. To enable this, delay
 35statistics of a task are available both during its lifetime as well as on its
 36exit, ensuring continuous and complete monitoring can be done.
 37
 38
 39Interface
 40---------
 41
 42Delay accounting uses the taskstats interface which is described
 43in detail in a separate document in this directory. Taskstats returns a
 44generic data structure to userspace corresponding to per-pid and per-tgid
 45statistics. The delay accounting functionality populates specific fields of
 46this structure. See
 47
 48     include/uapi/linux/taskstats.h
 49
 50for a description of the fields pertaining to delay accounting.
 51It will generally be in the form of counters returning the cumulative
 52delay seen for cpu, sync block I/O, swapin, memory reclaim, thrash page
 53cache, direct compact, write-protect copy, IRQ/SOFTIRQ etc.
 54
 55Taking the difference of two successive readings of a given
 56counter (say cpu_delay_total) for a task will give the delay
 57experienced by the task waiting for the corresponding resource
 58in that interval.
 59
 60When a task exits, records containing the per-task statistics
 61are sent to userspace without requiring a command. If it is the last exiting
 62task of a thread group, the per-tgid statistics are also sent. More details
 63are given in the taskstats interface description.
 64
 65The getdelays.c userspace utility in tools/accounting directory allows simple
 66commands to be run and the corresponding delay statistics to be displayed. It
 67also serves as an example of using the taskstats interface.
 68
 69Usage
 70-----
 71
 72Compile the kernel with::
 73
 74	CONFIG_TASK_DELAY_ACCT=y
 75	CONFIG_TASKSTATS=y
 76
 77Delay accounting is disabled by default at boot up.
 78To enable, add::
 79
 80   delayacct
 81
 82to the kernel boot options. The rest of the instructions below assume this has
 83been done. Alternatively, use sysctl kernel.task_delayacct to switch the state
 84at runtime. Note however that only tasks started after enabling it will have
 85delayacct information.
 86
 87After the system has booted up, use a utility
 88similar to  getdelays.c to access the delays
 89seen by a given task or a task group (tgid).
 90The utility also allows a given command to be
 91executed and the corresponding delays to be
 92seen.
 93
 94General format of the getdelays command::
 95
 96	getdelays [-dilv] [-t tgid] [-p pid]
 97
 98Get delays, since system boot, for pid 10::
 99
100	# ./getdelays -d -p 10
101	(output similar to next case)
102
103Get sum and peak of delays, since system boot, for all pids with tgid 242::
104
105	bash-4.4# ./getdelays -d -t 242
106	print delayacct stats ON
107	TGID    242
108
109
110	CPU         count     real total  virtual total    delay total  delay average      delay max      delay min
111	               39      156000000      156576579        2111069          0.054ms     0.212296ms     0.031307ms
112	IO          count    delay total  delay average      delay max      delay min
113	                0              0          0.000ms     0.000000ms     0.000000ms
114	SWAP        count    delay total  delay average      delay max      delay min
115	                0              0          0.000ms     0.000000ms     0.000000ms
116	RECLAIM     count    delay total  delay average      delay max      delay min
117	                0              0          0.000ms     0.000000ms     0.000000ms
118	THRASHING   count    delay total  delay average      delay max      delay min
119	                0              0          0.000ms     0.000000ms     0.000000ms
120	COMPACT     count    delay total  delay average      delay max      delay min
121	                0              0          0.000ms     0.000000ms     0.000000ms
122	WPCOPY      count    delay total  delay average      delay max      delay min
123	              156       11215873          0.072ms     0.207403ms     0.033913ms
124	IRQ         count    delay total  delay average      delay max      delay min
125	                0              0          0.000ms     0.000000ms     0.000000ms
126
127Get IO accounting for pid 1, it works only with -p::
128
129	# ./getdelays -i -p 1
130	printing IO accounting
131	linuxrc: read=65536, write=0, cancelled_write=0
132
133The above command can be used with -v to get more debug information.
134
135After the system starts, use `delaytop` to get the system-wide delay information,
136which includes system-wide PSI information and Top-N high-latency tasks.
137Note: PSI support requires `CONFIG_PSI=y` and `psi=1` for full functionality.
138
139`delaytop` is an interactive tool for monitoring system pressure and task delays.
140It supports multiple sorting options, display modes, and real-time keyboard controls.
141
142Basic usage with default settings (sorts by CPU delay, shows top 20 tasks, refreshes every 2 seconds)::
143
144	bash# ./delaytop
145	System Pressure Information: (avg10/avg60vg300/total)
146	CPU some:       0.0%/   0.0%/   0.0%/  106137(ms)
147	CPU full:       0.0%/   0.0%/   0.0%/       0(ms)
148	Memory full:    0.0%/   0.0%/   0.0%/       0(ms)
149	Memory some:    0.0%/   0.0%/   0.0%/       0(ms)
150	IO full:        0.0%/   0.0%/   0.0%/    2240(ms)
151	IO some:        0.0%/   0.0%/   0.0%/    2783(ms)
152	IRQ full:       0.0%/   0.0%/   0.0%/       0(ms)
153	[o]sort [M]memverbose [q]quit
154	Top 20 processes (sorted by cpu delay):
155		PID      TGID  COMMAND           CPU(ms)   IO(ms)  IRQ(ms)  MEM(ms)
156	------------------------------------------------------------------------
157		110       110  kworker/15:0H-s   27.91     0.00     0.00     0.00
158		57        57  cpuhp/7            3.18     0.00     0.00     0.00
159		99        99  cpuhp/14           2.97     0.00     0.00     0.00
160		51        51  cpuhp/6            0.90     0.00     0.00     0.00
161		44        44  kworker/4:0H-sy    0.80     0.00     0.00     0.00
162		60        60  ksoftirqd/7        0.74     0.00     0.00     0.00
163		76        76  idle_inject/10     0.31     0.00     0.00     0.00
164		100       100  idle_inject/14     0.30     0.00     0.00     0.00
165		1309      1309  systemsettings     0.29     0.00     0.00     0.00
166		45        45  cpuhp/5            0.22     0.00     0.00     0.00
167		63        63  cpuhp/8            0.20     0.00     0.00     0.00
168		87        87  cpuhp/12           0.18     0.00     0.00     0.00
169		93        93  cpuhp/13           0.17     0.00     0.00     0.00
170		1265      1265  acpid              0.17     0.00     0.00     0.00
171		1552      1552  sshd               0.17     0.00     0.00     0.00
172		2584      2584  sddm-helper        0.16     0.00     0.00     0.00
173		1284      1284  rtkit-daemon       0.15     0.00     0.00     0.00
174		1326      1326  nde-netfilter      0.14     0.00     0.00     0.00
175		27        27  cpuhp/2            0.13     0.00     0.00     0.00
176		631       631  kworker/11:2-rc    0.11     0.00     0.00     0.00
177
178Interactive keyboard controls during runtime::
179
180	o - Select sort field (CPU, IO, IRQ, Memory, etc.)
181	M - Toggle display mode (Default/Memory Verbose)
182	q - Quit
183
184Available sort fields(use -s/--sort or interactive command)::
185
186	cpu(c)       - CPU delay
187	blkio(i)     - I/O delay
188	irq(q)       - IRQ delay
189	mem(m)       - Total memory delay
190	swapin(s)    - Swapin delay (memory verbose mode only)
191	freepages(r) - Freepages reclaim delay (memory verbose mode only)
192	thrashing(t) - Thrashing delay (memory verbose mode only)
193	compact(p)   - Compaction delay (memory verbose mode only)
194	wpcopy(w)    - Write page copy delay (memory verbose mode only)
195
196Advanced usage examples::
197
198	# ./delaytop -s blkio
199	Sorted by IO delay
200
201	# ./delaytop -s mem -M
202	Sorted by memory delay in memory verbose mode
203
204	# ./delaytop -p pid
205	Print delayacct stats
206
207	# ./delaytop -P num
208	Display the top N tasks
209
210	# ./delaytop -n num
211	Set delaytop refresh frequency (num times)
212
213	# ./delaytop -d secs
214	Specify refresh interval as secs