reclaim.rst

  1.. SPDX-License-Identifier: GPL-2.0
  2
  3=======================
  4DAMON-based Reclamation
  5=======================
  6
  7DAMON-based Reclamation (DAMON_RECLAIM) is a static kernel module that aimed to
  8be used for proactive and lightweight reclamation under light memory pressure.
  9It doesn't aim to replace the LRU-list based page_granularity reclamation, but
 10to be selectively used for different level of memory pressure and requirements.
 11
 12Where Proactive Reclamation is Required?
 13========================================
 14
 15On general memory over-committed systems, proactively reclaiming cold pages
 16helps saving memory and reducing latency spikes that incurred by the direct
 17reclaim of the process or CPU consumption of kswapd, while incurring only
 18minimal performance degradation [1]_ [2]_ .
 19
 20Free Pages Reporting [3]_ based memory over-commit virtualization systems are
 21good example of the cases.  In such systems, the guest VMs reports their free
 22memory to host, and the host reallocates the reported memory to other guests.
 23As a result, the memory of the systems are fully utilized.  However, the
 24guests could be not so memory-frugal, mainly because some kernel subsystems and
 25user-space applications are designed to use as much memory as available.  Then,
 26guests could report only small amount of memory as free to host, results in
 27memory utilization drop of the systems.  Running the proactive reclamation in
 28guests could mitigate this problem.
 29
 30How It Works?
 31=============
 32
 33DAMON_RECLAIM finds memory regions that didn't accessed for specific time
 34duration and page out.  To avoid it consuming too much CPU for the paging out
 35operation, a speed limit can be configured.  Under the speed limit, it pages
 36out memory regions that didn't accessed longer time first.  System
 37administrators can also configure under what situation this scheme should
 38automatically activated and deactivated with three memory pressure watermarks.
 39
 40Interface: Module Parameters
 41============================
 42
 43To use this feature, you should first ensure your system is running on a kernel
 44that is built with ``CONFIG_DAMON_RECLAIM=y``.
 45
 46To let sysadmins enable or disable it and tune for the given system,
 47DAMON_RECLAIM utilizes module parameters.  That is, you can put
 48``damon_reclaim.<parameter>=<value>`` on the kernel boot command line or write
 49proper values to ``/sys/module/damon_reclaim/parameters/<parameter>`` files.
 50
 51Below are the description of each parameter.
 52
 53enabled
 54-------
 55
 56Enable or disable DAMON_RECLAIM.
 57
 58You can enable DAMON_RCLAIM by setting the value of this parameter as ``Y``.
 59Setting it as ``N`` disables DAMON_RECLAIM.  Note that DAMON_RECLAIM could do
 60no real monitoring and reclamation due to the watermarks-based activation
 61condition.  Refer to below descriptions for the watermarks parameter for this.
 62
 63commit_inputs
 64-------------
 65
 66Make DAMON_RECLAIM reads the input parameters again, except ``enabled``.
 67
 68Input parameters that updated while DAMON_RECLAIM is running are not applied
 69by default.  Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
 70of parametrs except ``enabled`` again.  Once the re-reading is done, this
 71parameter is set as ``N``.  If invalid parameters are found while the
 72re-reading, DAMON_RECLAIM will be disabled.
 73
 74Once ``Y`` is written to this parameter, the user must not write to any
 75parameters until reading ``commit_inputs`` again returns ``N``.  If users
 76violate this rule, the kernel may exhibit undefined behavior.
 77
 78min_age
 79-------
 80
 81Time threshold for cold memory regions identification in microseconds.
 82
 83If a memory region is not accessed for this or longer time, DAMON_RECLAIM
 84identifies the region as cold, and reclaims it.
 85
 86120 seconds by default.
 87
 88quota_ms
 89--------
 90
 91Limit of time for the reclamation in milliseconds.
 92
 93DAMON_RECLAIM tries to use only up to this time within a time window
 94(quota_reset_interval_ms) for trying reclamation of cold pages.  This can be
 95used for limiting CPU consumption of DAMON_RECLAIM.  If the value is zero, the
 96limit is disabled.
 97
 9810 ms by default.
 99
100quota_sz
101--------
102
103Limit of size of memory for the reclamation in bytes.
104
105DAMON_RECLAIM charges amount of memory which it tried to reclaim within a time
106window (quota_reset_interval_ms) and makes no more than this limit is tried.
107This can be used for limiting consumption of CPU and IO.  If this value is
108zero, the limit is disabled.
109
110128 MiB by default.
111
112quota_reset_interval_ms
113-----------------------
114
115The time/size quota charge reset interval in milliseconds.
116
117The charget reset interval for the quota of time (quota_ms) and size
118(quota_sz).  That is, DAMON_RECLAIM does not try reclamation for more than
119quota_ms milliseconds or quota_sz bytes within quota_reset_interval_ms
120milliseconds.
121
1221 second by default.
123
124quota_mem_pressure_us
125---------------------
126
127Desired level of memory pressure-stall time in microseconds.
128
129While keeping the caps that set by other quotas, DAMON_RECLAIM automatically
130increases and decreases the effective level of the quota aiming this level of
131memory pressure is incurred.  System-wide ``some`` memory PSI in microseconds
132per quota reset interval (``quota_reset_interval_ms``) is collected and
133compared to this value to see if the aim is satisfied.  Value zero means
134disabling this auto-tuning feature.
135
136Disabled by default.
137
138quota_autotune_feedback
139-----------------------
140
141User-specifiable feedback for auto-tuning of the effective quota.
142
143While keeping the caps that set by other quotas, DAMON_RECLAIM automatically
144increases and decreases the effective level of the quota aiming receiving this
145feedback of value ``10,000`` from the user.  DAMON_RECLAIM assumes the feedback
146value and the quota are positively proportional.  Value zero means disabling
147this auto-tuning feature.
148
149Disabled by default.
150
151wmarks_interval
152---------------
153
154Minimal time to wait before checking the watermarks, when DAMON_RECLAIM is
155enabled but inactive due to its watermarks rule.
156
157wmarks_high
158-----------
159
160Free memory rate (per thousand) for the high watermark.
161
162If free memory of the system in bytes per thousand bytes is higher than this,
163DAMON_RECLAIM becomes inactive, so it does nothing but only periodically checks
164the watermarks.
165
166wmarks_mid
167----------
168
169Free memory rate (per thousand) for the middle watermark.
170
171If free memory of the system in bytes per thousand bytes is between this and
172the low watermark, DAMON_RECLAIM becomes active, so starts the monitoring and
173the reclaiming.
174
175wmarks_low
176----------
177
178Free memory rate (per thousand) for the low watermark.
179
180If free memory of the system in bytes per thousand bytes is lower than this,
181DAMON_RECLAIM becomes inactive, so it does nothing but periodically checks the
182watermarks.  In the case, the system falls back to the LRU-list based page
183granularity reclamation logic.
184
185sample_interval
186---------------
187
188Sampling interval for the monitoring in microseconds.
189
190The sampling interval of DAMON for the cold memory monitoring.  Please refer to
191the DAMON documentation (:doc:`usage`) for more detail.
192
193aggr_interval
194-------------
195
196Aggregation interval for the monitoring in microseconds.
197
198The aggregation interval of DAMON for the cold memory monitoring.  Please
199refer to the DAMON documentation (:doc:`usage`) for more detail.
200
201min_nr_regions
202--------------
203
204Minimum number of monitoring regions.
205
206The minimal number of monitoring regions of DAMON for the cold memory
207monitoring.  This can be used to set lower-bound of the monitoring quality.
208But, setting this too high could result in increased monitoring overhead.
209Please refer to the DAMON documentation (:doc:`usage`) for more detail.
210
211max_nr_regions
212--------------
213
214Maximum number of monitoring regions.
215
216The maximum number of monitoring regions of DAMON for the cold memory
217monitoring.  This can be used to set upper-bound of the monitoring overhead.
218However, setting this too low could result in bad monitoring quality.  Please
219refer to the DAMON documentation (:doc:`usage`) for more detail.
220
221monitor_region_start
222--------------------
223
224Start of target memory region in physical address.
225
226The start physical address of memory region that DAMON_RECLAIM will do work
227against.  That is, DAMON_RECLAIM will find cold memory regions in this region
228and reclaims.  By default, biggest System RAM is used as the region.
229
230monitor_region_end
231------------------
232
233End of target memory region in physical address.
234
235The end physical address of memory region that DAMON_RECLAIM will do work
236against.  That is, DAMON_RECLAIM will find cold memory regions in this region
237and reclaims.  By default, biggest System RAM is used as the region.
238
239addr_unit
240---------
241
242A scale factor for memory addresses and bytes.
243
244This parameter is for setting and getting the :ref:`address unit
245<damon_design_addr_unit>` parameter of the DAMON instance for DAMON_RECLAIM.
246
247``monitor_region_start`` and ``monitor_region_end`` should be provided in this
248unit.  For example, let's suppose ``addr_unit``, ``monitor_region_start`` and
249``monitor_region_end`` are set as ``1024``, ``0`` and ``10``, respectively.
250Then DAMON_RECLAIM will work for 10 KiB length of physical address range that
251starts from address zero (``[0 * 1024, 10 * 1024)`` in bytes).
252
253``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are also in
254this unit.  For example, let's suppose values of ``addr_unit``,
255``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are ``1024``,
256``42``, and ``32``, respectively.  Then it means DAMON_RECLAIM tried to reclaim
25742 KiB memory and successfully reclaimed 32 KiB memory in total.
258
259If unsure, use only the default value (``1``) and forget about this.
260
261skip_anon
262---------
263
264Skip anonymous pages reclamation.
265
266If this parameter is set as ``Y``, DAMON_RECLAIM does not reclaim anonymous
267pages.  By default, ``N``.
268
269
270kdamond_pid
271-----------
272
273PID of the DAMON thread.
274
275If DAMON_RECLAIM is enabled, this becomes the PID of the worker thread.  Else,
276-1.
277
278nr_reclaim_tried_regions
279------------------------
280
281Number of memory regions that tried to be reclaimed by DAMON_RECLAIM.
282
283bytes_reclaim_tried_regions
284---------------------------
285
286Total bytes of memory regions that tried to be reclaimed by DAMON_RECLAIM.
287
288nr_reclaimed_regions
289--------------------
290
291Number of memory regions that successfully be reclaimed by DAMON_RECLAIM.
292
293bytes_reclaimed_regions
294-----------------------
295
296Total bytes of memory regions that successfully be reclaimed by DAMON_RECLAIM.
297
298nr_quota_exceeds
299----------------
300
301Number of times that the time/space quota limits have exceeded.
302
303Example
304=======
305
306Below runtime example commands make DAMON_RECLAIM to find memory regions that
307not accessed for 30 seconds or more and pages out.  The reclamation is limited
308to be done only up to 1 GiB per second to avoid DAMON_RECLAIM consuming too
309much CPU time for the paging out operation.  It also asks DAMON_RECLAIM to do
310nothing if the system's free memory rate is more than 50%, but start the real
311works if it becomes lower than 40%.  If DAMON_RECLAIM doesn't make progress and
312therefore the free memory rate becomes lower than 20%, it asks DAMON_RECLAIM to
313do nothing again, so that we can fall back to the LRU-list based page
314granularity reclamation. ::
315
316    # cd /sys/module/damon_reclaim/parameters
317    # echo 30000000 > min_age
318    # echo $((1 * 1024 * 1024 * 1024)) > quota_sz
319    # echo 1000 > quota_reset_interval_ms
320    # echo 500 > wmarks_high
321    # echo 400 > wmarks_mid
322    # echo 200 > wmarks_low
323    # echo Y > enabled
324
325.. [1] https://research.google/pubs/pub48551/
326.. [2] https://lwn.net/Articles/787611/
327.. [3] https://www.kernel.org/doc/html/latest/mm/free_page_reporting.html