FILE
resctrl.rst
File preview with syntax highlighting.
linux-6.19.14/Documentation/filesystems/resctrl.rst
1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4===================================================== 5User Interface for Resource Control feature (resctrl) 6===================================================== 7 8:Copyright: |copy| 2016 Intel Corporation 9:Authors: - Fenghua Yu <fenghua.yu@intel.com> 10 - Tony Luck <tony.luck@intel.com> 11 - Vikas Shivappa <vikas.shivappa@intel.com> 12 13 14Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). 15AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). 16 17This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo 18flag bits: 19 20=============================================================== ================================ 21RDT (Resource Director Technology) Allocation "rdt_a" 22CAT (Cache Allocation Technology) "cat_l3", "cat_l2" 23CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" 24CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" 25MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" 26MBA (Memory Bandwidth Allocation) "mba" 27SMBA (Slow Memory Bandwidth Allocation) "" 28BMEC (Bandwidth Monitoring Event Configuration) "" 29ABMC (Assignable Bandwidth Monitoring Counters) "" 30SDCIAE (Smart Data Cache Injection Allocation Enforcement) "" 31=============================================================== ================================ 32 33Historically, new features were made visible by default in /proc/cpuinfo. This 34resulted in the feature flags becoming hard to parse by humans. Adding a new 35flag to /proc/cpuinfo should be avoided if user space can obtain information 36about the feature from resctrl's info directory. 37 38To use the feature mount the file system:: 39 40 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl 41 42mount options are: 43 44"cdp": 45 Enable code/data prioritization in L3 cache allocations. 46"cdpl2": 47 Enable code/data prioritization in L2 cache allocations. 48"mba_MBps": 49 Enable the MBA Software Controller(mba_sc) to specify MBA 50 bandwidth in MiBps 51"debug": 52 Make debug files accessible. Available debug files are annotated with 53 "Available only with debug option". 54 55L2 and L3 CDP are controlled separately. 56 57RDT features are orthogonal. A particular system may support only 58monitoring, only control, or both monitoring and control. Cache 59pseudo-locking is a unique way of using cache control to "pin" or 60"lock" data in the cache. Details can be found in 61"Cache Pseudo-Locking". 62 63 64The mount succeeds if either of allocation or monitoring is present, but 65only those files and directories supported by the system will be created. 66For more details on the behavior of the interface during monitoring 67and allocation, see the "Resource alloc and monitor groups" section. 68 69Info directory 70============== 71 72The 'info' directory contains information about the enabled 73resources. Each resource has its own subdirectory. The subdirectory 74names reflect the resource names. 75 76Most of the files in the resource's subdirectory are read-only, and 77describe properties of the resource. Resources that support global 78configuration options also include writable files that can be used 79to modify those settings. 80 81Each subdirectory contains the following files with respect to 82allocation: 83 84Cache resource(L3/L2) subdirectory contains the following files 85related to allocation: 86 87"num_closids": 88 The number of CLOSIDs which are valid for this 89 resource. The kernel uses the smallest number of 90 CLOSIDs of all enabled resources as limit. 91"cbm_mask": 92 The bitmask which is valid for this resource. 93 This mask is equivalent to 100%. 94"min_cbm_bits": 95 The minimum number of consecutive bits which 96 must be set when writing a mask. 97 98"shareable_bits": 99 Bitmask of shareable resource with other executing entities 100 (e.g. I/O). Applies to all instances of this resource. User 101 can use this when setting up exclusive cache partitions. 102 Note that some platforms support devices that have their 103 own settings for cache use which can over-ride these bits. 104 105 When "io_alloc" is enabled, a portion of each cache instance can 106 be configured for shared use between hardware and software. 107 "bit_usage" should be used to see which portions of each cache 108 instance is configured for hardware use via "io_alloc" feature 109 because every cache instance can have its "io_alloc" bitmask 110 configured independently via "io_alloc_cbm". 111 112"bit_usage": 113 Annotated capacity bitmasks showing how all 114 instances of the resource are used. The legend is: 115 116 "0": 117 Corresponding region is unused. When the system's 118 resources have been allocated and a "0" is found 119 in "bit_usage" it is a sign that resources are 120 wasted. 121 122 "H": 123 Corresponding region is used by hardware only 124 but available for software use. If a resource 125 has bits set in "shareable_bits" or "io_alloc_cbm" 126 but not all of these bits appear in the resource 127 groups' schemata then the bits appearing in 128 "shareable_bits" or "io_alloc_cbm" but no 129 resource group will be marked as "H". 130 "X": 131 Corresponding region is available for sharing and 132 used by hardware and software. These are the bits 133 that appear in "shareable_bits" or "io_alloc_cbm" 134 as well as a resource group's allocation. 135 "S": 136 Corresponding region is used by software 137 and available for sharing. 138 "E": 139 Corresponding region is used exclusively by 140 one resource group. No sharing allowed. 141 "P": 142 Corresponding region is pseudo-locked. No 143 sharing allowed. 144"sparse_masks": 145 Indicates if non-contiguous 1s value in CBM is supported. 146 147 "0": 148 Only contiguous 1s value in CBM is supported. 149 "1": 150 Non-contiguous 1s value in CBM is supported. 151 152"io_alloc": 153 "io_alloc" enables system software to configure the portion of 154 the cache allocated for I/O traffic. File may only exist if the 155 system supports this feature on some of its cache resources. 156 157 "disabled": 158 Resource supports "io_alloc" but the feature is disabled. 159 Portions of cache used for allocation of I/O traffic cannot 160 be configured. 161 "enabled": 162 Portions of cache used for allocation of I/O traffic 163 can be configured using "io_alloc_cbm". 164 "not supported": 165 Support not available for this resource. 166 167 The feature can be modified by writing to the interface, for example: 168 169 To enable:: 170 171 # echo 1 > /sys/fs/resctrl/info/L3/io_alloc 172 173 To disable:: 174 175 # echo 0 > /sys/fs/resctrl/info/L3/io_alloc 176 177 The underlying implementation may reduce resources available to 178 general (CPU) cache allocation. See architecture specific notes 179 below. Depending on usage requirements the feature can be enabled 180 or disabled. 181 182 On AMD systems, io_alloc feature is supported by the L3 Smart 183 Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for 184 io_alloc is the highest CLOSID supported by the resource. When 185 io_alloc is enabled, the highest CLOSID is dedicated to io_alloc and 186 no longer available for general (CPU) cache allocation. When CDP is 187 enabled, io_alloc routes I/O traffic using the highest CLOSID allocated 188 for the instruction cache (CDP_CODE), making this CLOSID no longer 189 available for general (CPU) cache allocation for both the CDP_CODE 190 and CDP_DATA resources. 191 192"io_alloc_cbm": 193 Capacity bitmasks that describe the portions of cache instances to 194 which I/O traffic from supported I/O devices are routed when "io_alloc" 195 is enabled. 196 197 CBMs are displayed in the following format: 198 199 <cache_id0>=<cbm>;<cache_id1>=<cbm>;... 200 201 Example:: 202 203 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 204 0=ffff;1=ffff 205 206 CBMs can be configured by writing to the interface. 207 208 Example:: 209 210 # echo 1=ff > /sys/fs/resctrl/info/L3/io_alloc_cbm 211 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 212 0=ffff;1=00ff 213 214 # echo "0=ff;1=f" > /sys/fs/resctrl/info/L3/io_alloc_cbm 215 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 216 0=00ff;1=000f 217 218 When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE 219 resources may reflect the same values. For example, values read from and 220 written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by 221 /sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa. 222 223Memory bandwidth(MB) subdirectory contains the following files 224with respect to allocation: 225 226"min_bandwidth": 227 The minimum memory bandwidth percentage which 228 user can request. 229 230"bandwidth_gran": 231 The granularity in which the memory bandwidth 232 percentage is allocated. The allocated 233 b/w percentage is rounded off to the next 234 control step available on the hardware. The 235 available bandwidth control steps are: 236 min_bandwidth + N * bandwidth_gran. 237 238"delay_linear": 239 Indicates if the delay scale is linear or 240 non-linear. This field is purely informational 241 only. 242 243"thread_throttle_mode": 244 Indicator on Intel systems of how tasks running on threads 245 of a physical core are throttled in cases where they 246 request different memory bandwidth percentages: 247 248 "max": 249 the smallest percentage is applied 250 to all threads 251 "per-thread": 252 bandwidth percentages are directly applied to 253 the threads running on the core 254 255If RDT monitoring is available there will be an "L3_MON" directory 256with the following files: 257 258"num_rmids": 259 The number of RMIDs available. This is the 260 upper bound for how many "CTRL_MON" + "MON" 261 groups can be created. 262 263"mon_features": 264 Lists the monitoring events if 265 monitoring is enabled for the resource. 266 Example:: 267 268 # cat /sys/fs/resctrl/info/L3_MON/mon_features 269 llc_occupancy 270 mbm_total_bytes 271 mbm_local_bytes 272 273 If the system supports Bandwidth Monitoring Event 274 Configuration (BMEC), then the bandwidth events will 275 be configurable. The output will be:: 276 277 # cat /sys/fs/resctrl/info/L3_MON/mon_features 278 llc_occupancy 279 mbm_total_bytes 280 mbm_total_bytes_config 281 mbm_local_bytes 282 mbm_local_bytes_config 283 284"mbm_total_bytes_config", "mbm_local_bytes_config": 285 Read/write files containing the configuration for the mbm_total_bytes 286 and mbm_local_bytes events, respectively, when the Bandwidth 287 Monitoring Event Configuration (BMEC) feature is supported. 288 The event configuration settings are domain specific and affect 289 all the CPUs in the domain. When either event configuration is 290 changed, the bandwidth counters for all RMIDs of both events 291 (mbm_total_bytes as well as mbm_local_bytes) are cleared for that 292 domain. The next read for every RMID will report "Unavailable" 293 and subsequent reads will report the valid value. 294 295 Following are the types of events supported: 296 297 ==== ======================================================== 298 Bits Description 299 ==== ======================================================== 300 6 Dirty Victims from the QOS domain to all types of memory 301 5 Reads to slow memory in the non-local NUMA domain 302 4 Reads to slow memory in the local NUMA domain 303 3 Non-temporal writes to non-local NUMA domain 304 2 Non-temporal writes to local NUMA domain 305 1 Reads to memory in the non-local NUMA domain 306 0 Reads to memory in the local NUMA domain 307 ==== ======================================================== 308 309 By default, the mbm_total_bytes configuration is set to 0x7f to count 310 all the event types and the mbm_local_bytes configuration is set to 311 0x15 to count all the local memory events. 312 313 Examples: 314 315 * To view the current configuration:: 316 :: 317 318 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 319 0=0x7f;1=0x7f;2=0x7f;3=0x7f 320 321 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 322 0=0x15;1=0x15;3=0x15;4=0x15 323 324 * To change the mbm_total_bytes to count only reads on domain 0, 325 the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary 326 (in hexadecimal 0x33): 327 :: 328 329 # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 330 331 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 332 0=0x33;1=0x7f;2=0x7f;3=0x7f 333 334 * To change the mbm_local_bytes to count all the slow memory reads on 335 domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b 336 in binary (in hexadecimal 0x30): 337 :: 338 339 # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 340 341 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 342 0=0x30;1=0x30;3=0x15;4=0x15 343 344"mbm_assign_mode": 345 The supported counter assignment modes. The enclosed brackets indicate which mode 346 is enabled. The MBM events associated with counters may reset when "mbm_assign_mode" 347 is changed. 348 :: 349 350 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 351 [mbm_event] 352 default 353 354 "mbm_event": 355 356 mbm_event mode allows users to assign a hardware counter to an RMID, event 357 pair and monitor the bandwidth usage as long as it is assigned. The hardware 358 continues to track the assigned counter until it is explicitly unassigned by 359 the user. Each event within a resctrl group can be assigned independently. 360 361 In this mode, a monitoring event can only accumulate data while it is backed 362 by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON 363 group to specify which of the events should have a counter assigned. The number 364 of counters available is described in the "num_mbm_cntrs" file. Changing the 365 mode may cause all counters on the resource to reset. 366 367 Moving to mbm_event counter assignment mode requires users to assign the counters 368 to the events. Otherwise, the MBM event counters will return 'Unassigned' when read. 369 370 The mode is beneficial for AMD platforms that support more CTRL_MON 371 and MON groups than available hardware counters. By default, this 372 feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth 373 Monitoring Counters) capability, ensuring counters remain assigned even 374 when the corresponding RMID is not actively used by any processor. 375 376 "default": 377 378 In default mode, resctrl assumes there is a hardware counter for each 379 event within every CTRL_MON and MON group. On AMD platforms, it is 380 recommended to use the mbm_event mode, if supported, to prevent reset of MBM 381 events between reads resulting from hardware re-allocating counters. This can 382 result in misleading values or display "Unavailable" if no counter is assigned 383 to the event. 384 385 * To enable "mbm_event" counter assignment mode: 386 :: 387 388 # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 389 390 * To enable "default" monitoring mode: 391 :: 392 393 # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 394 395"num_mbm_cntrs": 396 The maximum number of counters (total of available and assigned counters) in 397 each domain when the system supports mbm_event mode. 398 399 For example, on a system with maximum of 32 memory bandwidth monitoring 400 counters in each of its L3 domains: 401 :: 402 403 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 404 0=32;1=32 405 406"available_mbm_cntrs": 407 The number of counters available for assignment in each domain when mbm_event 408 mode is enabled on the system. 409 410 For example, on a system with 30 available [hardware] assignable counters 411 in each of its L3 domains: 412 :: 413 414 # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 415 0=30;1=30 416 417"event_configs": 418 Directory that exists when "mbm_event" counter assignment mode is supported. 419 Contains a sub-directory for each MBM event that can be assigned to a counter. 420 421 Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes. 422 Each MBM event's sub-directory contains a file named "event_filter" that is 423 used to view and modify which memory transactions the MBM event is configured 424 with. The file is accessible only when "mbm_event" counter assignment mode is 425 enabled. 426 427 List of memory transaction types supported: 428 429 ========================== ======================================================== 430 Name Description 431 ========================== ======================================================== 432 dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory 433 remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain 434 local_reads_slow_memory Reads to slow memory in the local NUMA domain 435 remote_non_temporal_writes Non-temporal writes to non-local NUMA domain 436 local_non_temporal_writes Non-temporal writes to local NUMA domain 437 remote_reads Reads to memory in the non-local NUMA domain 438 local_reads Reads to memory in the local NUMA domain 439 ========================== ======================================================== 440 441 For example:: 442 443 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 444 local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, 445 local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all 446 447 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 448 local_reads,local_non_temporal_writes,local_reads_slow_memory 449 450 Modify the event configuration by writing to the "event_filter" file within 451 the "event_configs" directory. The read/write "event_filter" file contains the 452 configuration of the event that reflects which memory transactions are counted by it. 453 454 For example:: 455 456 # echo "local_reads, local_non_temporal_writes" > 457 /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 458 459 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 460 local_reads,local_non_temporal_writes 461 462"mbm_assign_on_mkdir": 463 Exists when "mbm_event" counter assignment mode is supported. Accessible 464 only when "mbm_event" counter assignment mode is enabled. 465 466 Determines if a counter will automatically be assigned to an RMID, MBM event 467 pair when its associated monitor group is created via mkdir. Enabled by default 468 on boot, also when switched from "default" mode to "mbm_event" counter assignment 469 mode. Users can disable this capability by writing to the interface. 470 471 "0": 472 Auto assignment is disabled. 473 "1": 474 Auto assignment is enabled. 475 476 Example:: 477 478 # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 479 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 480 0 481 482"max_threshold_occupancy": 483 Read/write file provides the largest value (in 484 bytes) at which a previously used LLC_occupancy 485 counter can be considered for re-use. 486 487Finally, in the top level of the "info" directory there is a file 488named "last_cmd_status". This is reset with every "command" issued 489via the file system (making new directories or writing to any of the 490control files). If the command was successful, it will read as "ok". 491If the command failed, it will provide more information that can be 492conveyed in the error returns from file operations. E.g. 493:: 494 495 # echo L3:0=f7 > schemata 496 bash: echo: write error: Invalid argument 497 # cat info/last_cmd_status 498 mask f7 has non-consecutive 1-bits 499 500Resource alloc and monitor groups 501================================= 502 503Resource groups are represented as directories in the resctrl file 504system. The default group is the root directory which, immediately 505after mounting, owns all the tasks and cpus in the system and can make 506full use of all resources. 507 508On a system with RDT control features additional directories can be 509created in the root directory that specify different amounts of each 510resource (see "schemata" below). The root and these additional top level 511directories are referred to as "CTRL_MON" groups below. 512 513On a system with RDT monitoring the root directory and other top level 514directories contain a directory named "mon_groups" in which additional 515directories can be created to monitor subsets of tasks in the CTRL_MON 516group that is their ancestor. These are called "MON" groups in the rest 517of this document. 518 519Removing a directory will move all tasks and cpus owned by the group it 520represents to the parent. Removing one of the created CTRL_MON groups 521will automatically remove all MON groups below it. 522 523Moving MON group directories to a new parent CTRL_MON group is supported 524for the purpose of changing the resource allocations of a MON group 525without impacting its monitoring data or assigned tasks. This operation 526is not allowed for MON groups which monitor CPUs. No other move 527operation is currently allowed other than simply renaming a CTRL_MON or 528MON group. 529 530All groups contain the following files: 531 532"tasks": 533 Reading this file shows the list of all tasks that belong to 534 this group. Writing a task id to the file will add a task to the 535 group. Multiple tasks can be added by separating the task ids 536 with commas. Tasks will be assigned sequentially. Multiple 537 failures are not supported. A single failure encountered while 538 attempting to assign a task will cause the operation to abort and 539 already added tasks before the failure will remain in the group. 540 Failures will be logged to /sys/fs/resctrl/info/last_cmd_status. 541 542 If the group is a CTRL_MON group the task is removed from 543 whichever previous CTRL_MON group owned the task and also from 544 any MON group that owned the task. If the group is a MON group, 545 then the task must already belong to the CTRL_MON parent of this 546 group. The task is removed from any previous MON group. 547 548 549"cpus": 550 Reading this file shows a bitmask of the logical CPUs owned by 551 this group. Writing a mask to this file will add and remove 552 CPUs to/from this group. As with the tasks file a hierarchy is 553 maintained where MON groups may only include CPUs owned by the 554 parent CTRL_MON group. 555 When the resource group is in pseudo-locked mode this file will 556 only be readable, reflecting the CPUs associated with the 557 pseudo-locked region. 558 559 560"cpus_list": 561 Just like "cpus", only using ranges of CPUs instead of bitmasks. 562 563 564When control is enabled all CTRL_MON groups will also contain: 565 566"schemata": 567 A list of all the resources available to this group. 568 Each resource has its own line and format - see below for details. 569 570"size": 571 Mirrors the display of the "schemata" file to display the size in 572 bytes of each allocation instead of the bits representing the 573 allocation. 574 575"mode": 576 The "mode" of the resource group dictates the sharing of its 577 allocations. A "shareable" resource group allows sharing of its 578 allocations while an "exclusive" resource group does not. A 579 cache pseudo-locked region is created by first writing 580 "pseudo-locksetup" to the "mode" file before writing the cache 581 pseudo-locked region's schemata to the resource group's "schemata" 582 file. On successful pseudo-locked region creation the mode will 583 automatically change to "pseudo-locked". 584 585"ctrl_hw_id": 586 Available only with debug option. The identifier used by hardware 587 for the control group. On x86 this is the CLOSID. 588 589When monitoring is enabled all MON groups will also contain: 590 591"mon_data": 592 This contains a set of files organized by L3 domain and by 593 RDT event. E.g. on a system with two L3 domains there will 594 be subdirectories "mon_L3_00" and "mon_L3_01". Each of these 595 directories have one file per event (e.g. "llc_occupancy", 596 "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these 597 files provide a read out of the current value of the event for 598 all tasks in the group. In CTRL_MON groups these files provide 599 the sum for all tasks in the CTRL_MON group and all tasks in 600 MON groups. Please see example section for more details on usage. 601 On systems with Sub-NUMA Cluster (SNC) enabled there are extra 602 directories for each node (located within the "mon_L3_XX" directory 603 for the L3 cache they occupy). These are named "mon_sub_L3_YY" 604 where "YY" is the node number. 605 606 When the 'mbm_event' counter assignment mode is enabled, reading 607 an MBM event of a MON group returns 'Unassigned' if no hardware 608 counter is assigned to it. For CTRL_MON groups, 'Unassigned' is 609 returned if the MBM event does not have an assigned counter in the 610 CTRL_MON group nor in any of its associated MON groups. 611 612"mon_hw_id": 613 Available only with debug option. The identifier used by hardware 614 for the monitor group. On x86 this is the RMID. 615 616When monitoring is enabled all MON groups may also contain: 617 618"mbm_L3_assignments": 619 Exists when "mbm_event" counter assignment mode is supported and lists the 620 counter assignment states of the group. 621 622 The assignment list is displayed in the following format: 623 624 <Event>:<Domain ID>=<Assignment state>;<Domain ID>=<Assignment state> 625 626 Event: A valid MBM event in the 627 /sys/fs/resctrl/info/L3_MON/event_configs directory. 628 629 Domain ID: A valid domain ID. When writing, '*' applies the changes 630 to all the domains. 631 632 Assignment states: 633 634 _ : No counter assigned. 635 636 e : Counter assigned exclusively. 637 638 Example: 639 640 To display the counter assignment states for the default group. 641 :: 642 643 # cd /sys/fs/resctrl 644 # cat /sys/fs/resctrl/mbm_L3_assignments 645 mbm_total_bytes:0=e;1=e 646 mbm_local_bytes:0=e;1=e 647 648 Assignments can be modified by writing to the interface. 649 650 Examples: 651 652 To unassign the counter associated with the mbm_total_bytes event on domain 0: 653 :: 654 655 # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments 656 # cat /sys/fs/resctrl/mbm_L3_assignments 657 mbm_total_bytes:0=_;1=e 658 mbm_local_bytes:0=e;1=e 659 660 To unassign the counter associated with the mbm_total_bytes event on all the domains: 661 :: 662 663 # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments 664 # cat /sys/fs/resctrl/mbm_L3_assignments 665 mbm_total_bytes:0=_;1=_ 666 mbm_local_bytes:0=e;1=e 667 668 To assign a counter associated with the mbm_total_bytes event on all domains in 669 exclusive mode: 670 :: 671 672 # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments 673 # cat /sys/fs/resctrl/mbm_L3_assignments 674 mbm_total_bytes:0=e;1=e 675 mbm_local_bytes:0=e;1=e 676 677When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: 678 679"mba_MBps_event": 680 Reading this file shows which memory bandwidth event is used 681 as input to the software feedback loop that keeps memory bandwidth 682 below the value specified in the schemata file. Writing the 683 name of one of the supported memory bandwidth events found in 684 /sys/fs/resctrl/info/L3_MON/mon_features changes the input 685 event. 686 687Resource allocation rules 688------------------------- 689 690When a task is running the following rules define which resources are 691available to it: 692 6931) If the task is a member of a non-default group, then the schemata 694 for that group is used. 695 6962) Else if the task belongs to the default group, but is running on a 697 CPU that is assigned to some specific group, then the schemata for the 698 CPU's group is used. 699 7003) Otherwise the schemata for the default group is used. 701 702Resource monitoring rules 703------------------------- 7041) If a task is a member of a MON group, or non-default CTRL_MON group 705 then RDT events for the task will be reported in that group. 706 7072) If a task is a member of the default CTRL_MON group, but is running 708 on a CPU that is assigned to some specific group, then the RDT events 709 for the task will be reported in that group. 710 7113) Otherwise RDT events for the task will be reported in the root level 712 "mon_data" group. 713 714 715Notes on cache occupancy monitoring and control 716=============================================== 717When moving a task from one group to another you should remember that 718this only affects *new* cache allocations by the task. E.g. you may have 719a task in a monitor group showing 3 MB of cache occupancy. If you move 720to a new group and immediately check the occupancy of the old and new 721groups you will likely see that the old group is still showing 3 MB and 722the new group zero. When the task accesses locations still in cache from 723before the move, the h/w does not update any counters. On a busy system 724you will likely see the occupancy in the old group go down as cache lines 725are evicted and re-used while the occupancy in the new group rises as 726the task accesses memory and loads into the cache are counted based on 727membership in the new group. 728 729The same applies to cache allocation control. Moving a task to a group 730with a smaller cache partition will not evict any cache lines. The 731process may continue to use them from the old partition. 732 733Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) 734to identify a control group and a monitoring group respectively. Each of 735the resource groups are mapped to these IDs based on the kind of group. The 736number of CLOSid and RMID are limited by the hardware and hence the creation of 737a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID 738and creation of "MON" group may fail if we run out of RMIDs. 739 740max_threshold_occupancy - generic concepts 741------------------------------------------ 742 743Note that an RMID once freed may not be immediately available for use as 744the RMID is still tagged the cache lines of the previous user of RMID. 745Hence such RMIDs are placed on limbo list and checked back if the cache 746occupancy has gone down. If there is a time when system has a lot of 747limbo RMIDs but which are not ready to be used, user may see an -EBUSY 748during mkdir. 749 750max_threshold_occupancy is a user configurable value to determine the 751occupancy at which an RMID can be freed. 752 753The mon_llc_occupancy_limbo tracepoint gives the precise occupancy in bytes 754for a subset of RMID that are not immediately available for allocation. 755This can't be relied on to produce output every second, it may be necessary 756to attempt to create an empty monitor group to force an update. Output may 757only be produced if creation of a control or monitor group fails. 758 759Schemata files - general concepts 760--------------------------------- 761Each line in the file describes one resource. The line starts with 762the name of the resource, followed by specific values to be applied 763in each of the instances of that resource on the system. 764 765Cache IDs 766--------- 767On current generation systems there is one L3 cache per socket and L2 768caches are generally just shared by the hyperthreads on a core, but this 769isn't an architectural requirement. We could have multiple separate L3 770caches on a socket, multiple cores could share an L2 cache. So instead 771of using "socket" or "core" to define the set of logical cpus sharing 772a resource we use a "Cache ID". At a given cache level this will be a 773unique number across the whole system (but it isn't guaranteed to be a 774contiguous sequence, there may be gaps). To find the ID for each logical 775CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id 776 777Cache Bit Masks (CBM) 778--------------------- 779For cache resources we describe the portion of the cache that is available 780for allocation using a bitmask. The maximum value of the mask is defined 781by each cpu model (and may be different for different cache levels). It 782is found using CPUID, but is also provided in the "info" directory of 783the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware 784requires that these masks have all the '1' bits in a contiguous block. So 7850x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 786and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks 787if non-contiguous 1s value is supported. On a system with a 20-bit mask 788each bit represents 5% of the capacity of the cache. You could partition 789the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. 790 791Notes on Sub-NUMA Cluster mode 792============================== 793When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA 794nodes much more readily than between regular NUMA nodes since the CPUs 795on Sub-NUMA nodes share the same L3 cache and the system may report 796the NUMA distance between Sub-NUMA nodes with a lower value than used 797for regular NUMA nodes. 798 799The top-level monitoring files in each "mon_L3_XX" directory provide 800the sum of data across all SNC nodes sharing an L3 cache instance. 801Users who bind tasks to the CPUs of a specific Sub-NUMA node can read 802the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the 803"mon_sub_L3_YY" directories to get node local data. 804 805Memory bandwidth allocation is still performed at the L3 cache 806level. I.e. throttling controls are applied to all SNC nodes. 807 808L3 cache allocation bitmaps also apply to all SNC nodes. But note that 809the amount of L3 cache represented by each bit is divided by the number 810of SNC nodes per L3 cache. E.g. with a 100MB cache on a system with 10-bit 811allocation masks each bit normally represents 10MB. With SNC mode enabled 812with two SNC nodes per L3 cache, each bit only represents 5MB. 813 814Memory bandwidth Allocation and monitoring 815========================================== 816 817For Memory bandwidth resource, by default the user controls the resource 818by indicating the percentage of total memory bandwidth. 819 820The minimum bandwidth percentage value for each cpu model is predefined 821and can be looked up through "info/MB/min_bandwidth". The bandwidth 822granularity that is allocated is also dependent on the cpu model and can 823be looked up at "info/MB/bandwidth_gran". The available bandwidth 824control steps are: min_bw + N * bw_gran. Intermediate values are rounded 825to the next control step available on the hardware. 826 827The bandwidth throttling is a core specific mechanism on some of Intel 828SKUs. Using a high bandwidth and a low bandwidth setting on two threads 829sharing a core may result in both threads being throttled to use the 830low bandwidth (see "thread_throttle_mode"). 831 832The fact that Memory bandwidth allocation(MBA) may be a core 833specific mechanism where as memory bandwidth monitoring(MBM) is done at 834the package level may lead to confusion when users try to apply control 835via the MBA and then monitor the bandwidth to see if the controls are 836effective. Below are such scenarios: 837 8381. User may *not* see increase in actual bandwidth when percentage 839 values are increased: 840 841This can occur when aggregate L2 external bandwidth is more than L3 842external bandwidth. Consider an SKL SKU with 24 cores on a package and 843where L2 external is 10GBps (hence aggregate L2 external bandwidth is 844240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 845threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 846bandwidth of 100GBps although the percentage value specified is only 50% 847<< 100%. Hence increasing the bandwidth percentage will not yield any 848more bandwidth. This is because although the L2 external bandwidth still 849has capacity, the L3 external bandwidth is fully used. Also note that 850this would be dependent on number of cores the benchmark is run on. 851 8522. Same bandwidth percentage may mean different actual bandwidth 853 depending on # of threads: 854 855For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 856thread, with 10% bandwidth' can consume up to 10GBps and 40GBps although 857they have same percentage bandwidth of 10%. This is simply because as 858threads start using more cores in an rdtgroup, the actual bandwidth may 859increase or vary although user specified bandwidth percentage is same. 860 861In order to mitigate this and make the interface more user friendly, 862resctrl added support for specifying the bandwidth in MiBps as well. The 863kernel underneath would use a software feedback mechanism or a "Software 864Controller(mba_sc)" which reads the actual bandwidth using MBM counters 865and adjust the memory bandwidth percentages to ensure:: 866 867 "actual bandwidth < user specified bandwidth". 868 869By default, the schemata would take the bandwidth percentage values 870where as user can switch to the "MBA software controller" mode using 871a mount option 'mba_MBps'. The schemata format is specified in the below 872sections. 873 874L3 schemata file details (code and data prioritization disabled) 875---------------------------------------------------------------- 876With CDP disabled the L3 schemata format is:: 877 878 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 879 880L3 schemata file details (CDP enabled via mount option to resctrl) 881------------------------------------------------------------------ 882When CDP is enabled L3 control is split into two separate resources 883so you can specify independent masks for code and data like this:: 884 885 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 886 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 887 888L2 schemata file details 889------------------------ 890CDP is supported at L2 using the 'cdpl2' mount option. The schemata 891format is either:: 892 893 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 894 895or 896 897 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 898 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 899 900 901Memory bandwidth Allocation (default mode) 902------------------------------------------ 903 904Memory b/w domain is L3 cache. 905:: 906 907 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 908 909Memory bandwidth Allocation specified in MiBps 910---------------------------------------------- 911 912Memory bandwidth domain is L3 cache. 913:: 914 915 MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;... 916 917Slow Memory Bandwidth Allocation (SMBA) 918--------------------------------------- 919AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). 920CXL.memory is the only supported "slow" memory device. With the 921support of SMBA, the hardware enables bandwidth allocation on 922the slow memory devices. If there are multiple such devices in 923the system, the throttling logic groups all the slow sources 924together and applies the limit on them as a whole. 925 926The presence of SMBA (with CXL.memory) is independent of slow memory 927devices presence. If there are no such devices on the system, then 928configuring SMBA will have no impact on the performance of the system. 929 930The bandwidth domain for slow memory is L3 cache. Its schemata file 931is formatted as: 932:: 933 934 SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 935 936Reading/writing the schemata file 937--------------------------------- 938Reading the schemata file will show the state of all resources 939on all domains. When writing you only need to specify those values 940which you wish to change. E.g. 941:: 942 943 # cat schemata 944 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff 945 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 946 # echo "L3DATA:2=3c0;" > schemata 947 # cat schemata 948 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff 949 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 950 951Reading/writing the schemata file (on AMD systems) 952-------------------------------------------------- 953Reading the schemata file will show the current bandwidth limit on all 954domains. The allocated resources are in multiples of one eighth GB/s. 955When writing to the file, you need to specify what cache id you wish to 956configure the bandwidth limit. 957 958For example, to allocate 2GB/s limit on the first cache id: 959 960:: 961 962 # cat schemata 963 MB:0=2048;1=2048;2=2048;3=2048 964 L3:0=ffff;1=ffff;2=ffff;3=ffff 965 966 # echo "MB:1=16" > schemata 967 # cat schemata 968 MB:0=2048;1= 16;2=2048;3=2048 969 L3:0=ffff;1=ffff;2=ffff;3=ffff 970 971Reading/writing the schemata file (on AMD systems) with SMBA feature 972-------------------------------------------------------------------- 973Reading and writing the schemata file is the same as without SMBA in 974above section. 975 976For example, to allocate 8GB/s limit on the first cache id: 977 978:: 979 980 # cat schemata 981 SMBA:0=2048;1=2048;2=2048;3=2048 982 MB:0=2048;1=2048;2=2048;3=2048 983 L3:0=ffff;1=ffff;2=ffff;3=ffff 984 985 # echo "SMBA:1=64" > schemata 986 # cat schemata 987 SMBA:0=2048;1= 64;2=2048;3=2048 988 MB:0=2048;1=2048;2=2048;3=2048 989 L3:0=ffff;1=ffff;2=ffff;3=ffff 990 991Cache Pseudo-Locking 992==================== 993CAT enables a user to specify the amount of cache space that an 994application can fill. Cache pseudo-locking builds on the fact that a 995CPU can still read and write data pre-allocated outside its current 996allocated area on a cache hit. With cache pseudo-locking, data can be 997preloaded into a reserved portion of cache that no application can 998fill, and from that point on will only serve cache hits. The cache 999pseudo-locked memory is made accessible to user space where an 1000application can map it into its virtual address space and thus have 1001a region of memory with reduced average read latency. 1002 1003The creation of a cache pseudo-locked region is triggered by a request 1004from the user to do so that is accompanied by a schemata of the region 1005to be pseudo-locked. The cache pseudo-locked region is created as follows: 1006 1007- Create a CAT allocation CLOSNEW with a CBM matching the schemata 1008 from the user of the cache region that will contain the pseudo-locked 1009 memory. This region must not overlap with any current CAT allocation/CLOS 1010 on the system and no future overlap with this cache region is allowed 1011 while the pseudo-locked region exists. 1012- Create a contiguous region of memory of the same size as the cache 1013 region. 1014- Flush the cache, disable hardware prefetchers, disable preemption. 1015- Make CLOSNEW the active CLOS and touch the allocated memory to load 1016 it into the cache. 1017- Set the previous CLOS as active. 1018- At this point the closid CLOSNEW can be released - the cache 1019 pseudo-locked region is protected as long as its CBM does not appear in 1020 any CAT allocation. Even though the cache pseudo-locked region will from 1021 this point on not appear in any CBM of any CLOS an application running with 1022 any CLOS will be able to access the memory in the pseudo-locked region since 1023 the region continues to serve cache hits. 1024- The contiguous region of memory loaded into the cache is exposed to 1025 user-space as a character device. 1026 1027Cache pseudo-locking increases the probability that data will remain 1028in the cache via carefully configuring the CAT feature and controlling 1029application behavior. There is no guarantee that data is placed in 1030cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict 1031“locked” data from cache. Power management C-states may shrink or 1032power off cache. Deeper C-states will automatically be restricted on 1033pseudo-locked region creation. 1034 1035It is required that an application using a pseudo-locked region runs 1036with affinity to the cores (or a subset of the cores) associated 1037with the cache on which the pseudo-locked region resides. A sanity check 1038within the code will not allow an application to map pseudo-locked memory 1039unless it runs with affinity to cores associated with the cache on which the 1040pseudo-locked region resides. The sanity check is only done during the 1041initial mmap() handling, there is no enforcement afterwards and the 1042application self needs to ensure it remains affine to the correct cores. 1043 1044Pseudo-locking is accomplished in two stages: 1045 10461) During the first stage the system administrator allocates a portion 1047 of cache that should be dedicated to pseudo-locking. At this time an 1048 equivalent portion of memory is allocated, loaded into allocated 1049 cache portion, and exposed as a character device. 10502) During the second stage a user-space application maps (mmap()) the 1051 pseudo-locked memory into its address space. 1052 1053Cache Pseudo-Locking Interface 1054------------------------------ 1055A pseudo-locked region is created using the resctrl interface as follows: 1056 10571) Create a new resource group by creating a new directory in /sys/fs/resctrl. 10582) Change the new resource group's mode to "pseudo-locksetup" by writing 1059 "pseudo-locksetup" to the "mode" file. 10603) Write the schemata of the pseudo-locked region to the "schemata" file. All 1061 bits within the schemata should be "unused" according to the "bit_usage" 1062 file. 1063 1064On successful pseudo-locked region creation the "mode" file will contain 1065"pseudo-locked" and a new character device with the same name as the resource 1066group will exist in /dev/pseudo_lock. This character device can be mmap()'ed 1067by user space in order to obtain access to the pseudo-locked memory region. 1068 1069An example of cache pseudo-locked region creation and usage can be found below. 1070 1071Cache Pseudo-Locking Debugging Interface 1072---------------------------------------- 1073The pseudo-locking debugging interface is enabled by default (if 1074CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. 1075 1076There is no explicit way for the kernel to test if a provided memory 1077location is present in the cache. The pseudo-locking debugging interface uses 1078the tracing infrastructure to provide two ways to measure cache residency of 1079the pseudo-locked region: 1080 10811) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data 1082 from these measurements are best visualized using a hist trigger (see 1083 example below). In this test the pseudo-locked region is traversed at 1084 a stride of 32 bytes while hardware prefetchers and preemption 1085 are disabled. This also provides a substitute visualization of cache 1086 hits and misses. 10872) Cache hit and miss measurements using model specific precision counters if 1088 available. Depending on the levels of cache on the system the pseudo_lock_l2 1089 and pseudo_lock_l3 tracepoints are available. 1090 1091When a pseudo-locked region is created a new debugfs directory is created for 1092it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single 1093write-only file, pseudo_lock_measure, is present in this directory. The 1094measurement of the pseudo-locked region depends on the number written to this 1095debugfs file: 1096 10971: 1098 writing "1" to the pseudo_lock_measure file will trigger the latency 1099 measurement captured in the pseudo_lock_mem_latency tracepoint. See 1100 example below. 11012: 1102 writing "2" to the pseudo_lock_measure file will trigger the L2 cache 1103 residency (cache hits and misses) measurement captured in the 1104 pseudo_lock_l2 tracepoint. See example below. 11053: 1106 writing "3" to the pseudo_lock_measure file will trigger the L3 cache 1107 residency (cache hits and misses) measurement captured in the 1108 pseudo_lock_l3 tracepoint. 1109 1110All measurements are recorded with the tracing infrastructure. This requires 1111the relevant tracepoints to be enabled before the measurement is triggered. 1112 1113Example of latency debugging interface 1114~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1115In this example a pseudo-locked region named "newlock" was created. Here is 1116how we can measure the latency in cycles of reading from this region and 1117visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS 1118is set:: 1119 1120 # :> /sys/kernel/tracing/trace 1121 # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger 1122 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 1123 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 1124 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 1125 # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist 1126 1127 # event histogram 1128 # 1129 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] 1130 # 1131 1132 { latency: 456 } hitcount: 1 1133 { latency: 50 } hitcount: 83 1134 { latency: 36 } hitcount: 96 1135 { latency: 44 } hitcount: 174 1136 { latency: 48 } hitcount: 195 1137 { latency: 46 } hitcount: 262 1138 { latency: 42 } hitcount: 693 1139 { latency: 40 } hitcount: 3204 1140 { latency: 38 } hitcount: 3484 1141 1142 Totals: 1143 Hits: 8192 1144 Entries: 9 1145 Dropped: 0 1146 1147Example of cache hits/misses debugging 1148~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1149In this example a pseudo-locked region named "newlock" was created on the L2 1150cache of a platform. Here is how we can obtain details of the cache hits 1151and misses using the platform's precision counters. 1152:: 1153 1154 # :> /sys/kernel/tracing/trace 1155 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 1156 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 1157 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 1158 # cat /sys/kernel/tracing/trace 1159 1160 # tracer: nop 1161 # 1162 # _-----=> irqs-off 1163 # / _----=> need-resched 1164 # | / _---=> hardirq/softirq 1165 # || / _--=> preempt-depth 1166 # ||| / delay 1167 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 1168 # | | | |||| | | 1169 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 1170 1171 1172Examples for RDT allocation usage 1173~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1174 11751) Example 1 1176 1177On a two socket machine (one L3 cache per socket) with just four bits 1178for cache bit masks, minimum b/w of 10% with a memory bandwidth 1179granularity of 10%. 1180:: 1181 1182 # mount -t resctrl resctrl /sys/fs/resctrl 1183 # cd /sys/fs/resctrl 1184 # mkdir p0 p1 1185 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata 1186 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata 1187 1188The default resource group is unmodified, so we have access to all parts 1189of all caches (its schemata file reads "L3:0=f;1=f"). 1190 1191Tasks that are under the control of group "p0" may only allocate from the 1192"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1193Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1194 1195Similarly, tasks that are under the control of group "p0" may use a 1196maximum memory b/w of 50% on socket0 and 50% on socket 1. 1197Tasks in group "p1" may also use 50% memory b/w on both sockets. 1198Note that unlike cache masks, memory b/w cannot specify whether these 1199allocations can overlap or not. The allocations specifies the maximum 1200b/w that the group may be able to use and the system admin can configure 1201the b/w accordingly. 1202 1203If resctrl is using the software controller (mba_sc) then user can enter the 1204max b/w in MB rather than the percentage values. 1205:: 1206 1207 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata 1208 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata 1209 1210In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w 1211of 1024MB where as on socket 1 they would use 500MB. 1212 12132) Example 2 1214 1215Again two sockets, but this time with a more realistic 20-bit mask. 1216 1217Two real time tasks pid=1234 running on processor 0 and pid=5678 running on 1218processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy 1219neighbors, each of the two real-time tasks exclusively occupies one quarter 1220of L3 cache on socket 0. 1221:: 1222 1223 # mount -t resctrl resctrl /sys/fs/resctrl 1224 # cd /sys/fs/resctrl 1225 1226First we reset the schemata for the default group so that the "upper" 122750% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by 1228ordinary tasks:: 1229 1230 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata 1231 1232Next we make a resource group for our first real time task and give 1233it access to the "top" 25% of the cache on socket 0. 1234:: 1235 1236 # mkdir p0 1237 # echo "L3:0=f8000;1=fffff" > p0/schemata 1238 1239Finally we move our first real time task into this resource group. We 1240also use taskset(1) to ensure the task always runs on a dedicated CPU 1241on socket 0. Most uses of resource groups will also constrain which 1242processors tasks run on. 1243:: 1244 1245 # echo 1234 > p0/tasks 1246 # taskset -cp 1 1234 1247 1248Ditto for the second real time task (with the remaining 25% of cache):: 1249 1250 # mkdir p1 1251 # echo "L3:0=7c00;1=fffff" > p1/schemata 1252 # echo 5678 > p1/tasks 1253 # taskset -cp 2 5678 1254 1255For the same 2 socket system with memory b/w resource and CAT L3 the 1256schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is 125710): 1258 1259For our first real time task this would request 20% memory b/w on socket 0. 1260:: 1261 1262 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 1263 1264For our second real time task this would request an other 20% memory b/w 1265on socket 0. 1266:: 1267 1268 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 1269 12703) Example 3 1271 1272A single socket system which has real-time tasks running on core 4-7 and 1273non real-time workload assigned to core 0-3. The real-time tasks share text 1274and data, so a per task association is not required and due to interaction 1275with the kernel it's desired that the kernel on these cores shares L3 with 1276the tasks. 1277:: 1278 1279 # mount -t resctrl resctrl /sys/fs/resctrl 1280 # cd /sys/fs/resctrl 1281 1282First we reset the schemata for the default group so that the "upper" 128350% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 1284cannot be used by ordinary tasks:: 1285 1286 # echo "L3:0=3ff\nMB:0=50" > schemata 1287 1288Next we make a resource group for our real time cores and give it access 1289to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on 1290socket 0. 1291:: 1292 1293 # mkdir p0 1294 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata 1295 1296Finally we move core 4-7 over to the new group and make sure that the 1297kernel and the tasks running there get 50% of the cache. They should 1298also get 50% of memory bandwidth assuming that the cores 4-7 are SMT 1299siblings and only the real time threads are scheduled on the cores 4-7. 1300:: 1301 1302 # echo F0 > p0/cpus 1303 13044) Example 4 1305 1306The resource groups in previous examples were all in the default "shareable" 1307mode allowing sharing of their cache allocations. If one resource group 1308configures a cache allocation then nothing prevents another resource group 1309to overlap with that allocation. 1310 1311In this example a new exclusive resource group will be created on a L2 CAT 1312system with two L2 cache instances that can be configured with an 8-bit 1313capacity bitmask. The new exclusive resource group will be configured to use 131425% of each cache instance. 1315:: 1316 1317 # mount -t resctrl resctrl /sys/fs/resctrl/ 1318 # cd /sys/fs/resctrl 1319 1320First, we observe that the default group is configured to allocate to all L2 1321cache:: 1322 1323 # cat schemata 1324 L2:0=ff;1=ff 1325 1326We could attempt to create the new resource group at this point, but it will 1327fail because of the overlap with the schemata of the default group:: 1328 1329 # mkdir p0 1330 # echo 'L2:0=0x3;1=0x3' > p0/schemata 1331 # cat p0/mode 1332 shareable 1333 # echo exclusive > p0/mode 1334 -sh: echo: write error: Invalid argument 1335 # cat info/last_cmd_status 1336 schemata overlaps 1337 1338To ensure that there is no overlap with another resource group the default 1339resource group's schemata has to change, making it possible for the new 1340resource group to become exclusive. 1341:: 1342 1343 # echo 'L2:0=0xfc;1=0xfc' > schemata 1344 # echo exclusive > p0/mode 1345 # grep . p0/* 1346 p0/cpus:0 1347 p0/mode:exclusive 1348 p0/schemata:L2:0=03;1=03 1349 p0/size:L2:0=262144;1=262144 1350 1351A new resource group will on creation not overlap with an exclusive resource 1352group:: 1353 1354 # mkdir p1 1355 # grep . p1/* 1356 p1/cpus:0 1357 p1/mode:shareable 1358 p1/schemata:L2:0=fc;1=fc 1359 p1/size:L2:0=786432;1=786432 1360 1361The bit_usage will reflect how the cache is used:: 1362 1363 # cat info/L2/bit_usage 1364 0=SSSSSSEE;1=SSSSSSEE 1365 1366A resource group cannot be forced to overlap with an exclusive resource group:: 1367 1368 # echo 'L2:0=0x1;1=0x1' > p1/schemata 1369 -sh: echo: write error: Invalid argument 1370 # cat info/last_cmd_status 1371 overlaps with exclusive group 1372 1373Example of Cache Pseudo-Locking 1374~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1375Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked 1376region is exposed at /dev/pseudo_lock/newlock that can be provided to 1377application for argument to mmap(). 1378:: 1379 1380 # mount -t resctrl resctrl /sys/fs/resctrl/ 1381 # cd /sys/fs/resctrl 1382 1383Ensure that there are bits available that can be pseudo-locked, since only 1384unused bits can be pseudo-locked the bits to be pseudo-locked needs to be 1385removed from the default resource group's schemata:: 1386 1387 # cat info/L2/bit_usage 1388 0=SSSSSSSS;1=SSSSSSSS 1389 # echo 'L2:1=0xfc' > schemata 1390 # cat info/L2/bit_usage 1391 0=SSSSSSSS;1=SSSSSS00 1392 1393Create a new resource group that will be associated with the pseudo-locked 1394region, indicate that it will be used for a pseudo-locked region, and 1395configure the requested pseudo-locked region capacity bitmask:: 1396 1397 # mkdir newlock 1398 # echo pseudo-locksetup > newlock/mode 1399 # echo 'L2:1=0x3' > newlock/schemata 1400 1401On success the resource group's mode will change to pseudo-locked, the 1402bit_usage will reflect the pseudo-locked region, and the character device 1403exposing the pseudo-locked region will exist:: 1404 1405 # cat newlock/mode 1406 pseudo-locked 1407 # cat info/L2/bit_usage 1408 0=SSSSSSSS;1=SSSSSSPP 1409 # ls -l /dev/pseudo_lock/newlock 1410 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock 1411 1412:: 1413 1414 /* 1415 * Example code to access one page of pseudo-locked cache region 1416 * from user space. 1417 */ 1418 #define _GNU_SOURCE 1419 #include <fcntl.h> 1420 #include <sched.h> 1421 #include <stdio.h> 1422 #include <stdlib.h> 1423 #include <unistd.h> 1424 #include <sys/mman.h> 1425 1426 /* 1427 * It is required that the application runs with affinity to only 1428 * cores associated with the pseudo-locked region. Here the cpu 1429 * is hardcoded for convenience of example. 1430 */ 1431 static int cpuid = 2; 1432 1433 int main(int argc, char *argv[]) 1434 { 1435 cpu_set_t cpuset; 1436 long page_size; 1437 void *mapping; 1438 int dev_fd; 1439 int ret; 1440 1441 page_size = sysconf(_SC_PAGESIZE); 1442 1443 CPU_ZERO(&cpuset); 1444 CPU_SET(cpuid, &cpuset); 1445 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); 1446 if (ret < 0) { 1447 perror("sched_setaffinity"); 1448 exit(EXIT_FAILURE); 1449 } 1450 1451 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); 1452 if (dev_fd < 0) { 1453 perror("open"); 1454 exit(EXIT_FAILURE); 1455 } 1456 1457 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, 1458 dev_fd, 0); 1459 if (mapping == MAP_FAILED) { 1460 perror("mmap"); 1461 close(dev_fd); 1462 exit(EXIT_FAILURE); 1463 } 1464 1465 /* Application interacts with pseudo-locked memory @mapping */ 1466 1467 ret = munmap(mapping, page_size); 1468 if (ret < 0) { 1469 perror("munmap"); 1470 close(dev_fd); 1471 exit(EXIT_FAILURE); 1472 } 1473 1474 close(dev_fd); 1475 exit(EXIT_SUCCESS); 1476 } 1477 1478Locking between applications 1479---------------------------- 1480 1481Certain operations on the resctrl filesystem, composed of read/writes 1482to/from multiple files, must be atomic. 1483 1484As an example, the allocation of an exclusive reservation of L3 cache 1485involves: 1486 1487 1. Read the cbmmasks from each directory or the per-resource "bit_usage" 1488 2. Find a contiguous set of bits in the global CBM bitmask that is clear 1489 in any of the directory cbmmasks 1490 3. Create a new directory 1491 4. Set the bits found in step 2 to the new directory "schemata" file 1492 1493If two applications attempt to allocate space concurrently then they can 1494end up allocating the same bits so the reservations are shared instead of 1495exclusive. 1496 1497To coordinate atomic operations on the resctrlfs and to avoid the problem 1498above, the following locking procedure is recommended: 1499 1500Locking is based on flock, which is available in libc and also as a shell 1501script command 1502 1503Write lock: 1504 1505 A) Take flock(LOCK_EX) on /sys/fs/resctrl 1506 B) Read/write the directory structure. 1507 C) funlock 1508 1509Read lock: 1510 1511 A) Take flock(LOCK_SH) on /sys/fs/resctrl 1512 B) If success read the directory structure. 1513 C) funlock 1514 1515Example with bash:: 1516 1517 # Atomically read directory structure 1518 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl 1519 1520 # Read directory contents and create new subdirectory 1521 1522 $ cat create-dir.sh 1523 find /sys/fs/resctrl/ > output.txt 1524 mask = function-of(output.txt) 1525 mkdir /sys/fs/resctrl/newres/ 1526 echo mask > /sys/fs/resctrl/newres/schemata 1527 1528 $ flock /sys/fs/resctrl/ ./create-dir.sh 1529 1530Example with C:: 1531 1532 /* 1533 * Example code do take advisory locks 1534 * before accessing resctrl filesystem 1535 */ 1536 #include <sys/file.h> 1537 #include <stdlib.h> 1538 1539 void resctrl_take_shared_lock(int fd) 1540 { 1541 int ret; 1542 1543 /* take shared lock on resctrl filesystem */ 1544 ret = flock(fd, LOCK_SH); 1545 if (ret) { 1546 perror("flock"); 1547 exit(-1); 1548 } 1549 } 1550 1551 void resctrl_take_exclusive_lock(int fd) 1552 { 1553 int ret; 1554 1555 /* release lock on resctrl filesystem */ 1556 ret = flock(fd, LOCK_EX); 1557 if (ret) { 1558 perror("flock"); 1559 exit(-1); 1560 } 1561 } 1562 1563 void resctrl_release_lock(int fd) 1564 { 1565 int ret; 1566 1567 /* take shared lock on resctrl filesystem */ 1568 ret = flock(fd, LOCK_UN); 1569 if (ret) { 1570 perror("flock"); 1571 exit(-1); 1572 } 1573 } 1574 1575 void main(void) 1576 { 1577 int fd, ret; 1578 1579 fd = open("/sys/fs/resctrl", O_DIRECTORY); 1580 if (fd == -1) { 1581 perror("open"); 1582 exit(-1); 1583 } 1584 resctrl_take_shared_lock(fd); 1585 /* code to read directory contents */ 1586 resctrl_release_lock(fd); 1587 1588 resctrl_take_exclusive_lock(fd); 1589 /* code to read and write directory contents */ 1590 resctrl_release_lock(fd); 1591 } 1592 1593Examples for RDT Monitoring along with allocation usage 1594======================================================= 1595Reading monitored data 1596---------------------- 1597Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would 1598show the current snapshot of LLC occupancy of the corresponding MON 1599group or CTRL_MON group. 1600 1601 1602Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) 1603------------------------------------------------------------------------ 1604On a two socket machine (one L3 cache per socket) with just four bits 1605for cache bit masks:: 1606 1607 # mount -t resctrl resctrl /sys/fs/resctrl 1608 # cd /sys/fs/resctrl 1609 # mkdir p0 p1 1610 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata 1611 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata 1612 # echo 5678 > p1/tasks 1613 # echo 5679 > p1/tasks 1614 1615The default resource group is unmodified, so we have access to all parts 1616of all caches (its schemata file reads "L3:0=f;1=f"). 1617 1618Tasks that are under the control of group "p0" may only allocate from the 1619"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1620Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1621 1622Create monitor groups and assign a subset of tasks to each monitor group. 1623:: 1624 1625 # cd /sys/fs/resctrl/p1/mon_groups 1626 # mkdir m11 m12 1627 # echo 5678 > m11/tasks 1628 # echo 5679 > m12/tasks 1629 1630fetch data (data shown in bytes) 1631:: 1632 1633 # cat m11/mon_data/mon_L3_00/llc_occupancy 1634 16234000 1635 # cat m11/mon_data/mon_L3_01/llc_occupancy 1636 14789000 1637 # cat m12/mon_data/mon_L3_00/llc_occupancy 1638 16789000 1639 1640The parent ctrl_mon group shows the aggregated data. 1641:: 1642 1643 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1644 31234000 1645 1646Example 2 (Monitor a task from its creation) 1647-------------------------------------------- 1648On a two socket machine (one L3 cache per socket):: 1649 1650 # mount -t resctrl resctrl /sys/fs/resctrl 1651 # cd /sys/fs/resctrl 1652 # mkdir p0 p1 1653 1654An RMID is allocated to the group once its created and hence the <cmd> 1655below is monitored from its creation. 1656:: 1657 1658 # echo $$ > /sys/fs/resctrl/p1/tasks 1659 # <cmd> 1660 1661Fetch the data:: 1662 1663 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1664 31789000 1665 1666Example 3 (Monitor without CAT support or before creating CAT groups) 1667--------------------------------------------------------------------- 1668 1669Assume a system like HSW has only CQM and no CAT support. In this case 1670the resctrl will still mount but cannot create CTRL_MON directories. 1671But user can create different MON groups within the root group thereby 1672able to monitor all tasks including kernel threads. 1673 1674This can also be used to profile jobs cache size footprint before being 1675able to allocate them to different allocation groups. 1676:: 1677 1678 # mount -t resctrl resctrl /sys/fs/resctrl 1679 # cd /sys/fs/resctrl 1680 # mkdir mon_groups/m01 1681 # mkdir mon_groups/m02 1682 1683 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks 1684 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks 1685 1686Monitor the groups separately and also get per domain data. From the 1687below its apparent that the tasks are mostly doing work on 1688domain(socket) 0. 1689:: 1690 1691 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy 1692 31234000 1693 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy 1694 34555 1695 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy 1696 31234000 1697 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy 1698 32789 1699 1700 1701Example 4 (Monitor real time tasks) 1702----------------------------------- 1703 1704A single socket system which has real time tasks running on cores 4-7 1705and non real time tasks on other cpus. We want to monitor the cache 1706occupancy of the real time threads on these cores. 1707:: 1708 1709 # mount -t resctrl resctrl /sys/fs/resctrl 1710 # cd /sys/fs/resctrl 1711 # mkdir p1 1712 1713Move the cpus 4-7 over to p1:: 1714 1715 # echo f0 > p1/cpus 1716 1717View the llc occupancy snapshot:: 1718 1719 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy 1720 11234000 1721 1722 1723Examples on working with mbm_assign_mode 1724======================================== 1725 1726a. Check if MBM counter assignment mode is supported. 1727:: 1728 1729 # mount -t resctrl resctrl /sys/fs/resctrl/ 1730 1731 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1732 [mbm_event] 1733 default 1734 1735The "mbm_event" mode is detected and enabled. 1736 1737b. Check how many assignable counters are supported. 1738:: 1739 1740 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 1741 0=32;1=32 1742 1743c. Check how many assignable counters are available for assignment in each domain. 1744:: 1745 1746 # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 1747 0=30;1=30 1748 1749d. To list the default group's assign states. 1750:: 1751 1752 # cat /sys/fs/resctrl/mbm_L3_assignments 1753 mbm_total_bytes:0=e;1=e 1754 mbm_local_bytes:0=e;1=e 1755 1756e. To unassign the counter associated with the mbm_total_bytes event on domain 0. 1757:: 1758 1759 # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments 1760 # cat /sys/fs/resctrl/mbm_L3_assignments 1761 mbm_total_bytes:0=_;1=e 1762 mbm_local_bytes:0=e;1=e 1763 1764f. To unassign the counter associated with the mbm_total_bytes event on all domains. 1765:: 1766 1767 # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments 1768 # cat /sys/fs/resctrl/mbm_L3_assignment 1769 mbm_total_bytes:0=_;1=_ 1770 mbm_local_bytes:0=e;1=e 1771 1772g. To assign a counter associated with the mbm_total_bytes event on all domains in 1773exclusive mode. 1774:: 1775 1776 # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments 1777 # cat /sys/fs/resctrl/mbm_L3_assignments 1778 mbm_total_bytes:0=e;1=e 1779 mbm_local_bytes:0=e;1=e 1780 1781h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is 1782no change in reading the events with the assignment. 1783:: 1784 1785 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 1786 779247936 1787 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_total_bytes 1788 562324232 1789 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1790 212122123 1791 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1792 121212144 1793 1794i. Check the event configurations. 1795:: 1796 1797 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 1798 local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, 1799 local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all 1800 1801 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1802 local_reads,local_non_temporal_writes,local_reads_slow_memory 1803 1804j. Change the event configuration for mbm_local_bytes. 1805:: 1806 1807 # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > 1808 /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1809 1810 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1811 local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads 1812 1813k. Now read the local events again. The first read may come back with "Unavailable" 1814status. The subsequent read of mbm_local_bytes will display the current value. 1815:: 1816 1817 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1818 Unavailable 1819 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1820 2252323 1821 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1822 Unavailable 1823 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1824 1566565 1825 1826l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be 1827done using the following command. Note that switching the mbm_assign_mode may reset all 1828the MBM counters (and thus all MBM events) of all the resctrl groups. 1829:: 1830 1831 # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1832 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1833 mbm_event 1834 [default] 1835 1836m. Unmount the resctrl filesystem. 1837:: 1838 1839 # umount /sys/fs/resctrl/ 1840 1841Intel RDT Errata 1842================ 1843 1844Intel MBM Counters May Report System Memory Bandwidth Incorrectly 1845----------------------------------------------------------------- 1846 1847Errata SKX99 for Skylake server and BDF102 for Broadwell server. 1848 1849Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics 1850according to the assigned Resource Monitor ID (RMID) for that logical 1851core. The IA32_QM_CTR register (MSR 0xC8E), used to report these 1852metrics, may report incorrect system bandwidth for certain RMID values. 1853 1854Implication: Due to the errata, system memory bandwidth may not match 1855what is reported. 1856 1857Workaround: MBM total and local readings are corrected according to the 1858following correction factor table: 1859 1860+---------------+---------------+---------------+-----------------+ 1861|core count |rmid count |rmid threshold |correction factor| 1862+---------------+---------------+---------------+-----------------+ 1863|1 |8 |0 |1.000000 | 1864+---------------+---------------+---------------+-----------------+ 1865|2 |16 |0 |1.000000 | 1866+---------------+---------------+---------------+-----------------+ 1867|3 |24 |15 |0.969650 | 1868+---------------+---------------+---------------+-----------------+ 1869|4 |32 |0 |1.000000 | 1870+---------------+---------------+---------------+-----------------+ 1871|6 |48 |31 |0.969650 | 1872+---------------+---------------+---------------+-----------------+ 1873|7 |56 |47 |1.142857 | 1874+---------------+---------------+---------------+-----------------+ 1875|8 |64 |0 |1.000000 | 1876+---------------+---------------+---------------+-----------------+ 1877|9 |72 |63 |1.185115 | 1878+---------------+---------------+---------------+-----------------+ 1879|10 |80 |63 |1.066553 | 1880+---------------+---------------+---------------+-----------------+ 1881|11 |88 |79 |1.454545 | 1882+---------------+---------------+---------------+-----------------+ 1883|12 |96 |0 |1.000000 | 1884+---------------+---------------+---------------+-----------------+ 1885|13 |104 |95 |1.230769 | 1886+---------------+---------------+---------------+-----------------+ 1887|14 |112 |95 |1.142857 | 1888+---------------+---------------+---------------+-----------------+ 1889|15 |120 |95 |1.066667 | 1890+---------------+---------------+---------------+-----------------+ 1891|16 |128 |0 |1.000000 | 1892+---------------+---------------+---------------+-----------------+ 1893|17 |136 |127 |1.254863 | 1894+---------------+---------------+---------------+-----------------+ 1895|18 |144 |127 |1.185255 | 1896+---------------+---------------+---------------+-----------------+ 1897|19 |152 |0 |1.000000 | 1898+---------------+---------------+---------------+-----------------+ 1899|20 |160 |127 |1.066667 | 1900+---------------+---------------+---------------+-----------------+ 1901|21 |168 |0 |1.000000 | 1902+---------------+---------------+---------------+-----------------+ 1903|22 |176 |159 |1.454334 | 1904+---------------+---------------+---------------+-----------------+ 1905|23 |184 |0 |1.000000 | 1906+---------------+---------------+---------------+-----------------+ 1907|24 |192 |127 |0.969744 | 1908+---------------+---------------+---------------+-----------------+ 1909|25 |200 |191 |1.280246 | 1910+---------------+---------------+---------------+-----------------+ 1911|26 |208 |191 |1.230921 | 1912+---------------+---------------+---------------+-----------------+ 1913|27 |216 |0 |1.000000 | 1914+---------------+---------------+---------------+-----------------+ 1915|28 |224 |191 |1.143118 | 1916+---------------+---------------+---------------+-----------------+ 1917 1918If rmid > rmid threshold, MBM total and local values should be multiplied 1919by the correction factor. 1920 1921See: 1922 19231. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update: 1924http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html 1925 19262. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update: 1927http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf 1928 19293. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual: 1930https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html 1931 1932for further information.
~/
Read-only file preview.