main.rst

   1.. SPDX-License-Identifier: GPL-2.0
   2.. include:: <isonum.txt>
   3
   4==================================================
   5Reliability, Availability and Serviceability (RAS)
   6==================================================
   7
   8This documents different aspects of the RAS functionality present in the
   9kernel.
  10
  11RAS concepts
  12************
  13
  14Reliability, Availability and Serviceability (RAS) is a concept used on
  15servers meant to measure their robustness.
  16
  17Reliability
  18  is the probability that a system will produce correct outputs.
  19
  20  * Generally measured as Mean Time Between Failures (MTBF)
  21  * Enhanced by features that help to avoid, detect and repair hardware faults
  22
  23Availability
  24  is the probability that a system is operational at a given time
  25
  26  * Generally measured as a percentage of downtime per a period of time
  27  * Often uses mechanisms to detect and correct hardware faults in
  28    runtime;
  29
  30Serviceability (or maintainability)
  31  is the simplicity and speed with which a system can be repaired or
  32  maintained
  33
  34  * Generally measured on Mean Time Between Repair (MTBR)
  35
  36Improving RAS
  37-------------
  38
  39In order to reduce systems downtime, a system should be capable of detecting
  40hardware errors, and, when possible correcting them in runtime. It should
  41also provide mechanisms to detect hardware degradation, in order to warn
  42the system administrator to take the action of replacing a component before
  43it causes data loss or system downtime.
  44
  45Among the monitoring measures, the most usual ones include:
  46
  47* CPU – detect errors at instruction execution and at L1/L2/L3 caches;
  48* Memory – add error correction logic (ECC) to detect and correct errors;
  49* I/O – add CRC checksums for transferred data;
  50* Storage – RAID, journal file systems, checksums,
  51  Self-Monitoring, Analysis and Reporting Technology (SMART).
  52
  53By monitoring the number of occurrences of error detections, it is possible
  54to identify if the probability of hardware errors is increasing, and, on such
  55case, do a preventive maintenance to replace a degraded component while
  56those errors are correctable.
  57
  58Types of errors
  59---------------
  60
  61Most mechanisms used on modern systems use technologies like Hamming
  62Codes that allow error correction when the number of errors on a bit packet
  63is below a threshold. If the number of errors is above, those mechanisms
  64can indicate with a high degree of confidence that an error happened, but
  65they can't correct.
  66
  67Also, sometimes an error occur on a component that it is not used. For
  68example, a part of the memory that it is not currently allocated.
  69
  70That defines some categories of errors:
  71
  72* **Correctable Error (CE)** - the error detection mechanism detected and
  73  corrected the error. Such errors are usually not fatal, although some
  74  Kernel mechanisms allow the system administrator to consider them as fatal.
  75
  76* **Uncorrected Error (UE)** - the amount of errors happened above the error
  77  correction threshold, and the system was unable to auto-correct.
  78
  79* **Fatal Error** - when an UE error happens on a critical component of the
  80  system (for example, a piece of the Kernel got corrupted by an UE), the
  81  only reliable way to avoid data corruption is to hang or reboot the machine.
  82
  83* **Non-fatal Error** - when an UE error happens on an unused component,
  84  like a CPU in power down state or an unused memory bank, the system may
  85  still run, eventually replacing the affected hardware by a hot spare,
  86  if available.
  87
  88  Also, when an error happens on a userspace process, it is also possible to
  89  kill such process and let userspace restart it.
  90
  91The mechanism for handling non-fatal errors is usually complex and may
  92require the help of some userspace application, in order to apply the
  93policy desired by the system administrator.
  94
  95Identifying a bad hardware component
  96------------------------------------
  97
  98Just detecting a hardware flaw is usually not enough, as the system needs
  99to pinpoint to the minimal replaceable unit (MRU) that should be exchanged
 100to make the hardware reliable again.
 101
 102So, it requires not only error logging facilities, but also mechanisms that
 103will translate the error message to the silkscreen or component label for
 104the MRU.
 105
 106Typically, it is very complex for memory, as modern CPUs interlace memory
 107from different memory modules, in order to provide a better performance. The
 108DMI BIOS usually have a list of memory module labels, with can be obtained
 109using the ``dmidecode`` tool. For example, on a desktop machine, it shows::
 110
 111	Memory Device
 112		Total Width: 64 bits
 113		Data Width: 64 bits
 114		Size: 16384 MB
 115		Form Factor: SODIMM
 116		Set: None
 117		Locator: ChannelA-DIMM0
 118		Bank Locator: BANK 0
 119		Type: DDR4
 120		Type Detail: Synchronous
 121		Speed: 2133 MHz
 122		Rank: 2
 123		Configured Clock Speed: 2133 MHz
 124
 125On the above example, a DDR4 SO-DIMM memory module is located at the
 126system's memory labeled as "BANK 0", as given by the *bank locator* field.
 127Please notice that, on such system, the *total width* is equal to the
 128*data width*. It means that such memory module doesn't have error
 129detection/correction mechanisms.
 130
 131Unfortunately, not all systems use the same field to specify the memory
 132bank. On this example, from an older server, ``dmidecode`` shows::
 133
 134	Memory Device
 135		Array Handle: 0x1000
 136		Error Information Handle: Not Provided
 137		Total Width: 72 bits
 138		Data Width: 64 bits
 139		Size: 8192 MB
 140		Form Factor: DIMM
 141		Set: 1
 142		Locator: DIMM_A1
 143		Bank Locator: Not Specified
 144		Type: DDR3
 145		Type Detail: Synchronous Registered (Buffered)
 146		Speed: 1600 MHz
 147		Rank: 2
 148		Configured Clock Speed: 1600 MHz
 149
 150There, the DDR3 RDIMM memory module is located at the system's memory labeled
 151as "DIMM_A1", as given by the *locator* field. Please notice that this
 152memory module has 64 bits of *data width* and 72 bits of *total width*. So,
 153it has 8 extra bits to be used by error detection and correction mechanisms.
 154Such kind of memory is called Error-correcting code memory (ECC memory).
 155
 156To make things even worse, it is not uncommon that systems with different
 157labels on their system's board to use exactly the same BIOS, meaning that
 158the labels provided by the BIOS won't match the real ones.
 159
 160ECC memory
 161----------
 162
 163As mentioned in the previous section, ECC memory has extra bits to be
 164used for error correction. In the above example, a memory module has
 16564 bits of *data width*, and 72 bits of *total width*.  The extra 8
 166bits which are used for the error detection and correction mechanisms
 167are referred to as the *syndrome*\ [#f1]_\ [#f2]_.
 168
 169So, when the cpu requests the memory controller to write a word with
 170*data width*, the memory controller calculates the *syndrome* in real time,
 171using Hamming code, or some other error correction code, like SECDED+,
 172producing a code with *total width* size. Such code is then written
 173on the memory modules.
 174
 175At read, the *total width* bits code is converted back, using the same
 176ECC code used on write, producing a word with *data width* and a *syndrome*.
 177The word with *data width* is sent to the CPU, even when errors happen.
 178
 179The memory controller also looks at the *syndrome* in order to check if
 180there was an error, and if the ECC code was able to fix such error.
 181If the error was corrected, a Corrected Error (CE) happened. If not, an
 182Uncorrected Error (UE) happened.
 183
 184The information about the CE/UE errors is stored on some special registers
 185at the memory controller and can be accessed by reading such registers,
 186either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64
 187bit CPUs, such errors can also be retrieved via the Machine Check
 188Architecture (MCA)\ [#f3]_.
 189
 190.. [#f1] Please notice that several memory controllers allow operation on a
 191  mode called "Lock-Step", where it groups two memory modules together,
 192  doing 128-bit reads/writes. That gives 16 bits for error correction, with
 193  significantly improves the error correction mechanism, at the expense
 194  that, when an error happens, there's no way to know what memory module is
 195  to blame. So, it has to blame both memory modules.
 196
 197.. [#f2] Some memory controllers also allow using memory in mirror mode.
 198  On such mode, the same data is written to two memory modules. At read,
 199  the system checks both memory modules, in order to check if both provide
 200  identical data. On such configuration, when an error happens, there's no
 201  way to know what memory module is to blame. So, it has to blame both
 202  memory modules (or 4 memory modules, if the system is also on Lock-step
 203  mode).
 204
 205.. [#f3] For more details about the Machine Check Architecture (MCA),
 206  please read Documentation/arch/x86/x86_64/machinecheck.rst at the Kernel tree.
 207
 208EDAC - Error Detection And Correction
 209*************************************
 210
 211.. note::
 212
 213   "bluesmoke" was the name for this device driver subsystem when it
 214   was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net.
 215   That site is mostly archaic now and can be used only for historical
 216   purposes.
 217
 218   When the subsystem was pushed upstream for the first time, on
 219   Kernel 2.6.16, it was renamed to ``EDAC``.
 220
 221Purpose
 222-------
 223
 224The ``edac`` kernel module's goal is to detect and report hardware errors
 225that occur within the computer system running under linux.
 226
 227Memory
 228------
 229
 230Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
 231primary errors being harvested. These types of errors are harvested by
 232the ``edac_mc`` device.
 233
 234Detecting CE events, then harvesting those events and reporting them,
 235**can** but must not necessarily be a predictor of future UE events. With
 236CE events only, the system can and will continue to operate as no data
 237has been damaged yet.
 238
 239However, preventive maintenance and proactive part replacement of memory
 240modules exhibiting CEs can reduce the likelihood of the dreaded UE events
 241and system panics.
 242
 243Other hardware elements
 244-----------------------
 245
 246A new feature for EDAC, the ``edac_device`` class of device, was added in
 247the 2.6.23 version of the kernel.
 248
 249This new device type allows for non-memory type of ECC hardware detectors
 250to have their states harvested and presented to userspace via the sysfs
 251interface.
 252
 253Some architectures have ECC detectors for L1, L2 and L3 caches,
 254along with DMA engines, fabric switches, main data path switches,
 255interconnections, and various other hardware data paths. If the hardware
 256reports it, then an edac_device device probably can be constructed to
 257harvest and present that to userspace.
 258
 259
 260PCI bus scanning
 261----------------
 262
 263In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
 264in order to determine if errors are occurring during data transfers.
 265
 266The presence of PCI Parity errors must be examined with a grain of salt.
 267There are several add-in adapters that do **not** follow the PCI specification
 268with regards to Parity generation and reporting. The specification says
 269the vendor should tie the parity status bits to 0 if they do not intend
 270to generate parity.  Some vendors do not do this, and thus the parity bit
 271can "float" giving false positives.
 272
 273There is a PCI device attribute located in sysfs that is checked by
 274the EDAC PCI scanning code. If that attribute is set, PCI parity/error
 275scanning is skipped for that device. The attribute is::
 276
 277	broken_parity_status
 278
 279and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for
 280PCI devices.
 281
 282
 283Versioning
 284----------
 285
 286EDAC is composed of a "core" module (``edac_core.ko``) and several Memory
 287Controller (MC) driver modules. On a given system, the CORE is loaded
 288and one MC driver will be loaded. Both the CORE and the MC driver (or
 289``edac_device`` driver) have individual versions that reflect current
 290release level of their respective modules.
 291
 292Thus, to "report" on what version a system is running, one must report
 293both the CORE's and the MC driver's versions.
 294
 295
 296Loading
 297-------
 298
 299If ``edac`` was statically linked with the kernel then no loading
 300is necessary. If ``edac`` was built as modules then simply modprobe
 301the ``edac`` pieces that you need. You should be able to modprobe
 302hardware-specific modules and have the dependencies load the necessary
 303core modules.
 304
 305Example::
 306
 307	$ modprobe amd76x_edac
 308
 309loads both the ``amd76x_edac.ko`` memory controller module and the
 310``edac_mc.ko`` core module.
 311
 312
 313Sysfs interface
 314---------------
 315
 316EDAC presents a ``sysfs`` interface for control and reporting purposes. It
 317lives in the /sys/devices/system/edac directory.
 318
 319Within this directory there currently reside 2 components:
 320
 321	======= ==============================
 322	mc	memory controller(s) system
 323	pci	PCI control and status system
 324	======= ==============================
 325
 326
 327
 328Memory Controller (mc) Model
 329----------------------------
 330
 331Each ``mc`` device controls a set of memory modules [#f4]_. These modules
 332are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``).
 333There can be multiple csrows and multiple channels.
 334
 335.. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely
 336  used to refer to a memory module, although there are other memory
 337  packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI
 338  specification (Version 2.7) defines a memory module in the Common
 339  Platform Error Record (CPER) section to be an SMBIOS Memory Device
 340  (Type 17). Along this document, and inside the EDAC subsystem, the term
 341  "dimm" is used for all memory modules, even when they use a
 342  different kind of packaging.
 343
 344Memory controllers allow for several csrows, with 8 csrows being a
 345typical value. Yet, the actual number of csrows depends on the layout of
 346a given motherboard, memory controller and memory module characteristics.
 347
 348Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems)
 349data transfers to/from the CPU from/to memory. Some newer chipsets allow
 350for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory
 351controllers. The following example will assume 2 channels:
 352
 353	+------------+-----------------------+
 354	| CS Rows    |       Channels        |
 355	+------------+-----------+-----------+
 356	|            |  ``ch0``  |  ``ch1``  |
 357	+============+===========+===========+
 358	|            |**DIMM_A0**|**DIMM_B0**|
 359	+------------+-----------+-----------+
 360	| ``csrow0`` |   rank0   |   rank0   |
 361	+------------+-----------+-----------+
 362	| ``csrow1`` |   rank1   |   rank1   |
 363	+------------+-----------+-----------+
 364	|            |**DIMM_A1**|**DIMM_B1**|
 365	+------------+-----------+-----------+
 366	| ``csrow2`` |    rank0  |  rank0    |
 367	+------------+-----------+-----------+
 368	| ``csrow3`` |    rank1  |  rank1    |
 369	+------------+-----------+-----------+
 370
 371In the above example, there are 4 physical slots on the motherboard
 372for memory DIMMs:
 373
 374	+---------+---------+
 375	| DIMM_A0 | DIMM_B0 |
 376	+---------+---------+
 377	| DIMM_A1 | DIMM_B1 |
 378	+---------+---------+
 379
 380Labels for these slots are usually silk-screened on the motherboard.
 381Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are
 382channel 1. Notice that there are two csrows possible on a physical DIMM.
 383These csrows are allocated their csrow assignment based on the slot into
 384which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
 385Channel, the csrows cross both DIMMs.
 386
 387Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
 388In the example above 2 dual ranked DIMMs are similarly placed. Thus,
 389both csrow0 and csrow1 are populated. On the other hand, when 2 single
 390ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will
 391have just one csrow (csrow0) and csrow1 will be empty. The pattern
 392repeats itself for csrow2 and csrow3. Also note that some memory
 393controllers don't have any logic to identify the memory module, see
 394``rankX`` directories below.
 395
 396The representation of the above is reflected in the directory
 397tree in EDAC's sysfs interface. Starting in directory
 398``/sys/devices/system/edac/mc``, each memory controller will be
 399represented by its own ``mcX`` directory, where ``X`` is the
 400index of the MC::
 401
 402	..../edac/mc/
 403		   |
 404		   |->mc0
 405		   |->mc1
 406		   |->mc2
 407		   ....
 408
 409Within each of the ``mcX`` directory are several EDAC control and
 410attribute files.
 411
 412``mcX`` directories
 413-------------------
 414
 415In ``mcX`` directories are EDAC control and attribute files for
 416this ``X`` instance of the memory controllers.
 417
 418For a description of the sysfs API, please see:
 419
 420	Documentation/ABI/testing/sysfs-devices-edac
 421
 422
 423``dimmX`` or ``rankX`` directories
 424----------------------------------
 425
 426The recommended way to use the EDAC subsystem is to look at the information
 427provided by the ``dimmX`` or ``rankX`` directories [#f5]_.
 428
 429A typical EDAC system has the following structure under
 430``/sys/devices/system/edac/``\ [#f6]_::
 431
 432	/sys/devices/system/edac/
 433	├── mc
 434	│   ├── mc0
 435	│   │   ├── ce_count
 436	│   │   ├── ce_noinfo_count
 437	│   │   ├── dimm0
 438	│   │   │   ├── dimm_ce_count
 439	│   │   │   ├── dimm_dev_type
 440	│   │   │   ├── dimm_edac_mode
 441	│   │   │   ├── dimm_label
 442	│   │   │   ├── dimm_location
 443	│   │   │   ├── dimm_mem_type
 444	│   │   │   ├── dimm_ue_count
 445	│   │   │   ├── size
 446	│   │   │   └── uevent
 447	│   │   ├── max_location
 448	│   │   ├── mc_name
 449	│   │   ├── reset_counters
 450	│   │   ├── seconds_since_reset
 451	│   │   ├── size_mb
 452	│   │   ├── ue_count
 453	│   │   ├── ue_noinfo_count
 454	│   │   └── uevent
 455	│   ├── mc1
 456	│   │   ├── ce_count
 457	│   │   ├── ce_noinfo_count
 458	│   │   ├── dimm0
 459	│   │   │   ├── dimm_ce_count
 460	│   │   │   ├── dimm_dev_type
 461	│   │   │   ├── dimm_edac_mode
 462	│   │   │   ├── dimm_label
 463	│   │   │   ├── dimm_location
 464	│   │   │   ├── dimm_mem_type
 465	│   │   │   ├── dimm_ue_count
 466	│   │   │   ├── size
 467	│   │   │   └── uevent
 468	│   │   ├── max_location
 469	│   │   ├── mc_name
 470	│   │   ├── reset_counters
 471	│   │   ├── seconds_since_reset
 472	│   │   ├── size_mb
 473	│   │   ├── ue_count
 474	│   │   ├── ue_noinfo_count
 475	│   │   └── uevent
 476	│   └── uevent
 477	└── uevent
 478
 479In the ``dimmX`` directories are EDAC control and attribute files for
 480this ``X`` memory module:
 481
 482- ``size`` - Total memory managed by this csrow attribute file
 483
 484	This attribute file displays, in count of megabytes, the memory
 485	that this csrow contains.
 486
 487- ``dimm_ue_count`` - Uncorrectable Errors count attribute file
 488
 489	This attribute file displays the total count of uncorrectable
 490	errors that have occurred on this DIMM. If panic_on_ue is set
 491	this counter will not have a chance to increment, since EDAC
 492	will panic the system.
 493
 494- ``dimm_ce_count`` - Correctable Errors count attribute file
 495
 496	This attribute file displays the total count of correctable
 497	errors that have occurred on this DIMM. This count is very
 498	important to examine. CEs provide early indications that a
 499	DIMM is beginning to fail. This count field should be
 500	monitored for non-zero values and report such information
 501	to the system administrator.
 502
 503- ``dimm_dev_type``  - Device type attribute file
 504
 505	This attribute file will display what type of DRAM device is
 506	being utilized on this DIMM.
 507	Examples:
 508
 509		- x1
 510		- x2
 511		- x4
 512		- x8
 513
 514- ``dimm_edac_mode`` - EDAC Mode of operation attribute file
 515
 516	This attribute file will display what type of Error detection
 517	and correction is being utilized.
 518
 519- ``dimm_label`` - memory module label control file
 520
 521	This control file allows this DIMM to have a label assigned
 522	to it. With this label in the module, when errors occur
 523	the output can provide the DIMM label in the system log.
 524	This becomes vital for panic events to isolate the
 525	cause of the UE event.
 526
 527	DIMM Labels must be assigned after booting, with information
 528	that correctly identifies the physical slot with its
 529	silk screen label. This information is currently very
 530	motherboard specific and determination of this information
 531	must occur in userland at this time.
 532
 533- ``dimm_location`` - location of the memory module
 534
 535	The location can have up to 3 levels, and describe how the
 536	memory controller identifies the location of a memory module.
 537	Depending on the type of memory and memory controller, it
 538	can be:
 539
 540		- *csrow* and *channel* - used when the memory controller
 541		  doesn't identify a single DIMM - e. g. in ``rankX`` dir;
 542		- *branch*, *channel*, *slot* - typically used on FB-DIMM memory
 543		  controllers;
 544		- *channel*, *slot* - used on Nehalem and newer Intel drivers.
 545
 546- ``dimm_mem_type`` - Memory Type attribute file
 547
 548	This attribute file will display what type of memory is currently
 549	on this csrow. Normally, either buffered or unbuffered memory.
 550	Examples:
 551
 552		- Registered-DDR
 553		- Unbuffered-DDR
 554
 555.. [#f5] On some systems, the memory controller doesn't have any logic
 556  to identify the memory module. On such systems, the directory is called ``rankX``.
 557  On modern Intel memory controllers, the memory controller identifies the
 558  memory modules directly. On such systems, the directory is called ``dimmX``.
 559
 560.. [#f6] There are also some ``power`` directories and ``subsystem``
 561  symlinks inside the sysfs mapping that are automatically created by
 562  the sysfs subsystem. Currently, they serve no purpose.
 563
 564
 565System Logging
 566--------------
 567
 568If logging for UEs and CEs is enabled, then system logs will contain
 569information indicating that errors have been detected::
 570
 571  EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac
 572  EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac
 573
 574
 575The structure of the message is:
 576
 577	+---------------------------------------+-------------+
 578	| Content                               | Example     |
 579	+=======================================+=============+
 580	| The memory controller                 | MC0         |
 581	+---------------------------------------+-------------+
 582	| Error type                            | CE          |
 583	+---------------------------------------+-------------+
 584	| Memory page                           | 0x283       |
 585	+---------------------------------------+-------------+
 586	| Offset in the page                    | 0xce0       |
 587	+---------------------------------------+-------------+
 588	| The byte granularity                  | grain 8     |
 589	| or resolution of the error            |             |
 590	+---------------------------------------+-------------+
 591	| The error syndrome                    | 0xb741      |
 592	+---------------------------------------+-------------+
 593	| Memory row                            | row 0       |
 594	+---------------------------------------+-------------+
 595	| Memory channel                        | channel 1   |
 596	+---------------------------------------+-------------+
 597	| DIMM label, if set prior              | DIMM B1     |
 598	+---------------------------------------+-------------+
 599	| And then an optional, driver-specific |             |
 600	| message that may have additional      |             |
 601	| information.                          |             |
 602	+---------------------------------------+-------------+
 603
 604Both UEs and CEs with no info will lack all but memory controller, error
 605type, a notice of "no info" and then an optional, driver-specific error
 606message.
 607
 608
 609PCI Bus Parity Detection
 610------------------------
 611
 612On Header Type 00 devices, the primary status is looked at for any
 613parity error regardless of whether parity is enabled on the device or
 614not. (The spec indicates parity is generated in some cases). On Header
 615Type 01 bridges, the secondary status register is also looked at to see
 616if parity occurred on the bus on the other side of the bridge.
 617
 618
 619Sysfs configuration
 620-------------------
 621
 622Under ``/sys/devices/system/edac/pci`` are control and attribute files as
 623follows:
 624
 625
 626- ``check_pci_parity`` - Enable/Disable PCI Parity checking control file
 627
 628	This control file enables or disables the PCI Bus Parity scanning
 629	operation. Writing a 1 to this file enables the scanning. Writing
 630	a 0 to this file disables the scanning.
 631
 632	Enable::
 633
 634		echo "1" >/sys/devices/system/edac/pci/check_pci_parity
 635
 636	Disable::
 637
 638		echo "0" >/sys/devices/system/edac/pci/check_pci_parity
 639
 640
 641- ``pci_parity_count`` - Parity Count
 642
 643	This attribute file will display the number of parity errors that
 644	have been detected.
 645
 646
 647Module parameters
 648-----------------
 649
 650- ``edac_mc_panic_on_ue`` - Panic on UE control file
 651
 652	An uncorrectable error will cause a machine panic.  This is usually
 653	desirable.  It is a bad idea to continue when an uncorrectable error
 654	occurs - it is indeterminate what was uncorrected and the operating
 655	system context might be so mangled that continuing will lead to further
 656	corruption. If the kernel has MCE configured, then EDAC will never
 657	notice the UE.
 658
 659	LOAD TIME::
 660
 661		module/kernel parameter: edac_mc_panic_on_ue=[0|1]
 662
 663	RUN TIME::
 664
 665		echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
 666
 667
 668- ``edac_mc_log_ue`` - Log UE control file
 669
 670
 671	Generate kernel messages describing uncorrectable errors.  These errors
 672	are reported through the system message log system.  UE statistics
 673	will be accumulated even when UE logging is disabled.
 674
 675	LOAD TIME::
 676
 677		module/kernel parameter: edac_mc_log_ue=[0|1]
 678
 679	RUN TIME::
 680
 681		echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
 682
 683
 684- ``edac_mc_log_ce`` - Log CE control file
 685
 686
 687	Generate kernel messages describing correctable errors.  These
 688	errors are reported through the system message log system.
 689	CE statistics will be accumulated even when CE logging is disabled.
 690
 691	LOAD TIME::
 692
 693		module/kernel parameter: edac_mc_log_ce=[0|1]
 694
 695	RUN TIME::
 696
 697		echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
 698
 699
 700- ``edac_mc_poll_msec`` - Polling period control file
 701
 702
 703	The time period, in milliseconds, for polling for error information.
 704	Too small a value wastes resources.  Too large a value might delay
 705	necessary handling of errors and might loose valuable information for
 706	locating the error.  1000 milliseconds (once each second) is the current
 707	default. Systems which require all the bandwidth they can get, may
 708	increase this.
 709
 710	LOAD TIME::
 711
 712		module/kernel parameter: edac_mc_poll_msec=[0|1]
 713
 714	RUN TIME::
 715
 716		echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
 717
 718
 719- ``panic_on_pci_parity`` - Panic on PCI PARITY Error
 720
 721
 722	This control file enables or disables panicking when a parity
 723	error has been detected.
 724
 725
 726	module/kernel parameter::
 727
 728			edac_panic_on_pci_pe=[0|1]
 729
 730	Enable::
 731
 732		echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
 733
 734	Disable::
 735
 736		echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
 737
 738
 739
 740EDAC device type
 741----------------
 742
 743In the header file, edac_pci.h, there is a series of edac_device structures
 744and APIs for the EDAC_DEVICE.
 745
 746User space access to an edac_device is through the sysfs interface.
 747
 748At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices
 749will appear.
 750
 751There is a three level tree beneath the above ``edac`` directory. For example,
 752the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net
 753website) installs itself as::
 754
 755	/sys/devices/system/edac/test-instance
 756
 757in this directory are various controls, a symlink and one or more ``instance``
 758directories.
 759
 760The standard default controls are:
 761
 762	==============	=======================================================
 763	log_ce		boolean to log CE events
 764	log_ue		boolean to log UE events
 765	panic_on_ue	boolean to ``panic`` the system if an UE is encountered
 766			(default off, can be set true via startup script)
 767	poll_msec	time period between POLL cycles for events
 768	==============	=======================================================
 769
 770The test_device_edac device adds at least one of its own custom control:
 771
 772	==============	==================================================
 773	test_bits	which in the current test driver does nothing but
 774			show how it is installed. A ported driver can
 775			add one or more such controls and/or attributes
 776			for specific uses.
 777			One out-of-tree driver uses controls here to allow
 778			for ERROR INJECTION operations to hardware
 779			injection registers
 780	==============	==================================================
 781
 782The symlink points to the 'struct dev' that is registered for this edac_device.
 783
 784Instances
 785---------
 786
 787One or more instance directories are present. For the ``test_device_edac``
 788case:
 789
 790	+----------------+
 791	| test-instance0 |
 792	+----------------+
 793
 794
 795In this directory there are two default counter attributes, which are totals of
 796counter in deeper subdirectories.
 797
 798	==============	====================================
 799	ce_count	total of CE events of subdirectories
 800	ue_count	total of UE events of subdirectories
 801	==============	====================================
 802
 803Blocks
 804------
 805
 806At the lowest directory level is the ``block`` directory. There can be 0, 1
 807or more blocks specified in each instance:
 808
 809	+-------------+
 810	| test-block0 |
 811	+-------------+
 812
 813In this directory the default attributes are:
 814
 815	==============	================================================
 816	ce_count	which is counter of CE events for this ``block``
 817			of hardware being monitored
 818	ue_count	which is counter of UE events for this ``block``
 819			of hardware being monitored
 820	==============	================================================
 821
 822
 823The ``test_device_edac`` device adds 4 attributes and 1 control:
 824
 825	================== ====================================================
 826	test-block-bits-0	for every POLL cycle this counter
 827				is incremented
 828	test-block-bits-1	every 10 cycles, this counter is bumped once,
 829				and test-block-bits-0 is set to 0
 830	test-block-bits-2	every 100 cycles, this counter is bumped once,
 831				and test-block-bits-1 is set to 0
 832	test-block-bits-3	every 1000 cycles, this counter is bumped once,
 833				and test-block-bits-2 is set to 0
 834	================== ====================================================
 835
 836
 837	================== ====================================================
 838	reset-counters		writing ANY thing to this control will
 839				reset all the above counters.
 840	================== ====================================================
 841
 842
 843Use of the ``test_device_edac`` driver should enable any others to create their own
 844unique drivers for their hardware systems.
 845
 846The ``test_device_edac`` sample driver is located at the
 847http://bluesmoke.sourceforge.net project site for EDAC.
 848
 849
 850Usage of EDAC APIs on Nehalem and newer Intel CPUs
 851--------------------------------------------------
 852
 853On older Intel architectures, the memory controller was part of the North
 854Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and
 855newer Intel architectures integrated an enhanced version of the memory
 856controller (MC) inside the CPUs.
 857
 858This chapter will cover the differences of the enhanced memory controllers
 859found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and
 860``sbx_edac`` drivers.
 861
 862.. note::
 863
 864   The Xeon E7 processor families use a separate chip for the memory
 865   controller, called Intel Scalable Memory Buffer. This section doesn't
 866   apply for such families.
 867
 8681) There is one Memory Controller per Quick Patch Interconnect
 869   (QPI). At the driver, the term "socket" means one QPI. This is
 870   associated with a physical CPU socket.
 871
 872   Each MC have 3 physical read channels, 3 physical write channels and
 873   3 logic channels. The driver currently sees it as just 3 channels.
 874   Each channel can have up to 3 DIMMs.
 875
 876   The minimum known unity is DIMMs. There are no information about csrows.
 877   As EDAC API maps the minimum unity is csrows, the driver sequentially
 878   maps channel/DIMM into different csrows.
 879
 880   For example, supposing the following layout::
 881
 882	Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
 883	  dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
 884	  dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
 885        Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
 886	  dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
 887	Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
 888	  dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
 889
 890   The driver will map it as::
 891
 892	csrow0: channel 0, dimm0
 893	csrow1: channel 0, dimm1
 894	csrow2: channel 1, dimm0
 895	csrow3: channel 2, dimm0
 896
 897   exports one DIMM per csrow.
 898
 899   Each QPI is exported as a different memory controller.
 900
 9012) The MC has the ability to inject errors to test drivers. The drivers
 902   implement this functionality via some error injection nodes:
 903
 904   For injecting a memory error, there are some sysfs nodes, under
 905   ``/sys/devices/system/edac/mc/mc?/``:
 906
 907   - ``inject_addrmatch/*``:
 908      Controls the error injection mask register. It is possible to specify
 909      several characteristics of the address to match an error code::
 910
 911         dimm = the affected dimm. Numbers are relative to a channel;
 912         rank = the memory rank;
 913         channel = the channel that will generate an error;
 914         bank = the affected bank;
 915         page = the page address;
 916         column (or col) = the address column.
 917
 918      each of the above values can be set to "any" to match any valid value.
 919
 920      At driver init, all values are set to any.
 921
 922      For example, to generate an error at rank 1 of dimm 2, for any channel,
 923      any bank, any page, any column::
 924
 925		echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
 926		echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
 927
 928	To return to the default behaviour of matching any, you can do::
 929
 930		echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
 931		echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
 932
 933   - ``inject_eccmask``:
 934          specifies what bits will have troubles,
 935
 936   - ``inject_section``:
 937       specifies what ECC cache section will get the error::
 938
 939		3 for both
 940		2 for the highest
 941		1 for the lowest
 942
 943   - ``inject_type``:
 944       specifies the type of error, being a combination of the following bits::
 945
 946		bit 0 - repeat
 947		bit 1 - ecc
 948		bit 2 - parity
 949
 950   - ``inject_enable``:
 951       starts the error generation when something different than 0 is written.
 952
 953   All inject vars can be read. root permission is needed for write.
 954
 955   Datasheet states that the error will only be generated after a write on an
 956   address that matches inject_addrmatch. It seems, however, that reading will
 957   also produce an error.
 958
 959   For example, the following code will generate an error for any write access
 960   at socket 0, on any DIMM/address on channel 2::
 961
 962	echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
 963	echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
 964	echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
 965	echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
 966	echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
 967	dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
 968
 969   For socket 1, it is needed to replace "mc0" by "mc1" at the above
 970   commands.
 971
 972   The generated error message will look like::
 973
 974	EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
 975
 9763) Corrected Error memory register counters
 977
 978   Those newer MCs have some registers to count memory errors. The driver
 979   uses those registers to report Corrected Errors on devices with Registered
 980   DIMMs.
 981
 982   However, those counters don't work with Unregistered DIMM. As the chipset
 983   offers some counters that also work with UDIMMs (but with a worse level of
 984   granularity than the default ones), the driver exposes those registers for
 985   UDIMM memories.
 986
 987   They can be read by looking at the contents of ``all_channel_counts/``::
 988
 989     $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
 990	/sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
 991	0
 992	/sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
 993	0
 994	/sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
 995	0
 996
 997   What happens here is that errors on different csrows, but at the same
 998   dimm number will increment the same counter.
 999   So, in this memory mapping::
1000
1001	csrow0: channel 0, dimm0
1002	csrow1: channel 0, dimm1
1003	csrow2: channel 1, dimm0
1004	csrow3: channel 2, dimm0
1005
1006   The hardware will increment udimm0 for an error at the first dimm at either
1007   csrow0, csrow2  or csrow3;
1008
1009   The hardware will increment udimm1 for an error at the second dimm at either
1010   csrow0, csrow2  or csrow3;
1011
1012   The hardware will increment udimm2 for an error at the third dimm at either
1013   csrow0, csrow2  or csrow3;
1014
10154) Standard error counters
1016
1017   The standard error counters are generated when an mcelog error is received
1018   by the driver. Since, with UDIMM, this is counted by software, it is
1019   possible that some errors could be lost. With RDIMM's, they display the
1020   contents of the registers
1021
1022Reference documents used on ``amd64_edac``
1023------------------------------------------
1024
1025``amd64_edac`` module is based on the following documents
1026(available from http://support.amd.com/en-us/search/tech-docs):
1027
10281. :Title:  BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
1029	   Opteron Processors
1030   :AMD publication #: 26094
1031   :Revision: 3.26
1032   :Link: http://support.amd.com/TechDocs/26094.PDF
1033
10342. :Title:  BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
1035	   Processors
1036   :AMD publication #: 32559
1037   :Revision: 3.00
1038   :Issue Date: May 2006
1039   :Link: http://support.amd.com/TechDocs/32559.pdf
1040
10413. :Title:  BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
1042	   Processors
1043   :AMD publication #: 31116
1044   :Revision: 3.00
1045   :Issue Date: September 07, 2007
1046   :Link: http://support.amd.com/TechDocs/31116.pdf
1047
10484. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1049	  Models 30h-3Fh Processors
1050   :AMD publication #: 49125
1051   :Revision: 3.06
1052   :Issue Date: 2/12/2015 (latest release)
1053   :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
1054
10555. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1056	  Models 60h-6Fh Processors
1057   :AMD publication #: 50742
1058   :Revision: 3.01
1059   :Issue Date: 7/23/2015 (latest release)
1060   :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
1061
10626. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
1063	  Models 00h-0Fh Processors
1064   :AMD publication #: 48751
1065   :Revision: 3.03
1066   :Issue Date: 2/23/2015 (latest release)
1067   :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
1068
1069Credits
1070=======
1071
1072* Written by Doug Thompson <dougthompson@xmission.com>
1073
1074  - 7 Dec 2005
1075  - 17 Jul 2007	Updated
1076
1077* |copy| Mauro Carvalho Chehab
1078
1079  - 05 Aug 2009	Nehalem interface
1080  - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section
1081
1082* EDAC authors/maintainers:
1083
1084  - Doug Thompson, Dave Jiang, Dave Peterson et al,
1085  - Mauro Carvalho Chehab
1086  - Borislav Petkov
1087  - original author: Thayne Harbaugh