nfsd-io-modes.rst

  1.. SPDX-License-Identifier: GPL-2.0
  2
  3=============
  4NFSD IO MODES
  5=============
  6
  7Overview
  8========
  9
 10NFSD has historically always used buffered IO when servicing READ and
 11WRITE operations. BUFFERED is NFSD's default IO mode, but it is possible
 12to override that default to use either DONTCACHE or DIRECT IO modes.
 13
 14Experimental NFSD debugfs interfaces are available to allow the NFSD IO
 15mode used for READ and WRITE to be configured independently. See both:
 16
 17- /sys/kernel/debug/nfsd/io_cache_read
 18- /sys/kernel/debug/nfsd/io_cache_write
 19
 20The default value for both io_cache_read and io_cache_write reflects
 21NFSD's default IO mode (which is NFSD_IO_BUFFERED=0).
 22
 23Based on the configured settings, NFSD's IO will either be:
 24
 25- cached using page cache (NFSD_IO_BUFFERED=0)
 26- cached but removed from page cache on completion (NFSD_IO_DONTCACHE=1)
 27- not cached stable_how=NFS_UNSTABLE (NFSD_IO_DIRECT=2)
 28
 29To set an NFSD IO mode, write a supported value (0 - 2) to the
 30corresponding IO operation's debugfs interface, e.g.::
 31
 32  echo 2 > /sys/kernel/debug/nfsd/io_cache_read
 33  echo 2 > /sys/kernel/debug/nfsd/io_cache_write
 34
 35To check which IO mode NFSD is using for READ or WRITE, simply read the
 36corresponding IO operation's debugfs interface, e.g.::
 37
 38  cat /sys/kernel/debug/nfsd/io_cache_read
 39  cat /sys/kernel/debug/nfsd/io_cache_write
 40
 41If you experiment with NFSD's IO modes on a recent kernel and have
 42interesting results, please report them to linux-nfs@vger.kernel.org
 43
 44NFSD DONTCACHE
 45==============
 46
 47DONTCACHE offers a hybrid approach to servicing IO that aims to offer
 48the benefits of using DIRECT IO without any of the strict alignment
 49requirements that DIRECT IO imposes. To achieve this buffered IO is used
 50but the IO is flagged to "drop behind" (meaning associated pages are
 51dropped from the page cache) when IO completes.
 52
 53DONTCACHE aims to avoid what has proven to be a fairly significant
 54limition of Linux's memory management subsystem if/when large amounts of
 55data is infrequently accessed (e.g. read once _or_ written once but not
 56read until much later). Such use-cases are particularly problematic
 57because the page cache will eventually become a bottleneck to servicing
 58new IO requests.
 59
 60For more context on DONTCACHE, please see these Linux commit headers:
 61
 62- Overview:  9ad6344568cc3 ("mm/filemap: change filemap_create_folio()
 63  to take a struct kiocb")
 64- for READ:  8026e49bff9b1 ("mm/filemap: add read support for
 65  RWF_DONTCACHE")
 66- for WRITE: 974c5e6139db3 ("xfs: flag as supporting FOP_DONTCACHE")
 67
 68NFSD_IO_DONTCACHE will fall back to NFSD_IO_BUFFERED if the underlying
 69filesystem doesn't indicate support by setting FOP_DONTCACHE.
 70
 71NFSD DIRECT
 72===========
 73
 74DIRECT IO doesn't make use of the page cache, as such it is able to
 75avoid the Linux memory management's page reclaim scalability problems
 76without resorting to the hybrid use of page cache that DONTCACHE does.
 77
 78Some workloads benefit from NFSD avoiding the page cache, particularly
 79those with a working set that is significantly larger than available
 80system memory. The pathological worst-case workload that NFSD DIRECT has
 81proven to help most is: NFS client issuing large sequential IO to a file
 82that is 2-3 times larger than the NFS server's available system memory.
 83The reason for such improvement is NFSD DIRECT eliminates a lot of work
 84that the memory management subsystem would otherwise be required to
 85perform (e.g. page allocation, dirty writeback, page reclaim). When
 86using NFSD DIRECT, kswapd and kcompactd are no longer commanding CPU
 87time trying to find adequate free pages so that forward IO progress can
 88be made.
 89
 90The performance win associated with using NFSD DIRECT was previously
 91discussed on linux-nfs, see:
 92https://lore.kernel.org/linux-nfs/aEslwqa9iMeZjjlV@kernel.org/
 93
 94But in summary:
 95
 96- NFSD DIRECT can significantly reduce memory requirements
 97- NFSD DIRECT can reduce CPU load by avoiding costly page reclaim work
 98- NFSD DIRECT can offer more deterministic IO performance
 99
100As always, your mileage may vary and so it is important to carefully
101consider if/when it is beneficial to make use of NFSD DIRECT. When
102assessing comparative performance of your workload please be sure to log
103relevant performance metrics during testing (e.g. memory usage, cpu
104usage, IO performance). Using perf to collect perf data that may be used
105to generate a "flamegraph" for work Linux must perform on behalf of your
106test is a really meaningful way to compare the relative health of the
107system and how switching NFSD's IO mode changes what is observed.
108
109If NFSD_IO_DIRECT is specified by writing 2 (or 3 and 4 for WRITE) to
110NFSD's debugfs interfaces, ideally the IO will be aligned relative to
111the underlying block device's logical_block_size. Also the memory buffer
112used to store the READ or WRITE payload must be aligned relative to the
113underlying block device's dma_alignment.
114
115But NFSD DIRECT does handle misaligned IO in terms of O_DIRECT as best
116it can:
117
118Misaligned READ:
119    If NFSD_IO_DIRECT is used, expand any misaligned READ to the next
120    DIO-aligned block (on either end of the READ). The expanded READ is
121    verified to have proper offset/len (logical_block_size) and
122    dma_alignment checking.
123
124Misaligned WRITE:
125    If NFSD_IO_DIRECT is used, split any misaligned WRITE into a start,
126    middle and end as needed. The large middle segment is DIO-aligned
127    and the start and/or end are misaligned. Buffered IO is used for the
128    misaligned segments and O_DIRECT is used for the middle DIO-aligned
129    segment. DONTCACHE buffered IO is _not_ used for the misaligned
130    segments because using normal buffered IO offers significant RMW
131    performance benefit when handling streaming misaligned WRITEs.
132
133Tracing:
134    The nfsd_read_direct trace event shows how NFSD expands any
135    misaligned READ to the next DIO-aligned block (on either end of the
136    original READ, as needed).
137
138    This combination of trace events is useful for READs::
139
140      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_vector/enable
141      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_direct/enable
142      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_read_io_done/enable
143      echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_read/enable
144
145    The nfsd_write_direct trace event shows how NFSD splits a given
146    misaligned WRITE into a DIO-aligned middle segment.
147
148    This combination of trace events is useful for WRITEs::
149
150      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_opened/enable
151      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_direct/enable
152      echo 1 > /sys/kernel/tracing/events/nfsd/nfsd_write_io_done/enable
153      echo 1 > /sys/kernel/tracing/events/xfs/xfs_file_direct_write/enable