.. _skiboot-6.2-rc1:

skiboot-6.2-rc1
===============

skiboot v6.2-rc1 was released on Monday November 19th 2018. It is the first
release candidate of skiboot 6.2, which will become the new stable release
of skiboot following the 6.1 release, first released July 11th 2018.

Skiboot 6.2 will mark the basis for op-build v2.2.

skiboot v6.2-rc1 contains all bug fixes as of :ref:`skiboot-6.0.13`,
and :ref:`skiboot-5.4.10` (the currently maintained
stable releases).

For how the skiboot stable releases work, see :ref:`stable-rules` for details.

This release has been a longer cycle than typical for a variety of reasons. It
also contains a lot of cleanup work and minor bug fixes (much like skiboot 6.1
did).

Over skiboot 6.1, we have the following changes:

General
-------

- cpu: Quieten OS endian switch messages

  Users see these when loading an OS from Petitboot: ::

     [  119.486794100,5] OPAL: Switch to big-endian OS
     [  120.022302604,5] OPAL: Switch to little-endian OS

  Which is expected and doesn't provide any information the user can act
  on. Switch them to PR_INFO so they still appear in the log, but not on
  the serial console.
- Recognise signed VERSION partition

  A few things need to change to support a signed VERSION partition:

  - A signed VERSION partition will be 4K + SECURE_BOOT_HEADERS_SIZE (4K).
  - The VERSION partition needs to be loaded after secure/trusted boot is
    set up, and therefore after nvram_init().
  - Added to the trustedboot resources array.

  This also moves the ipmi_dt_add_bmc_info() call to after
  flash_dt_add_fw_version() since it adds info to ibm,firmware-versions.
- Run pollers in time_wait() when not booting

  This only bit us hard with hiomap in one scenario.

  Our OPAL API has been OPAL_POLL_EVENTS may be needed to make forward
  progress on ongoing operations, and the internal to skiboot API has been
  that time_wait() of a suitable time will run pollers (on at least one
  CPU) to help ensure forward progress can be made.

  In a perfect world, interrupts are used but they may: a) be disabled, or
  b) the thing we're doing can't use interrupts because computers are
  generally terrible.

  Back in 3db397ea5892a (circa 2015), we changed skiboot so that we'd run
  pollers only on the boot CPU, and not if we held any locks. This was to
  reduce the chance of programming code that could deadlock, as well as to
  ensure that we didn't just thrash all the cachelines for running pollers
  all over a large system during boot, or hard spin on the same locks on
  all secondary CPUs.

  The problem arises if the OS we're booting makes an OPAL call early on,
  with interrupts disabled, that requires a poller to run to make forward
  progress. An example of this would be OPAL_WRITE_NVRAM early in Linux
  boot (where Linux sets up the partitions it wants) - something that
  occurs iff we've had to reformat NVRAM this boot (i.e. first boot or
  corrupted NVRAM).

  The hiomap implementation should arguably *not* rely on synchronous IPMI
  messages, but this is a future improvement (as was for mbox before it).
  The mbox-flash code solved this problem by spinning on check_timers().

  More generically though, the approach of running the pollers when no
  longer booting means we behave more in line with what the API is meant
  to be, rather than have this odd case of "time_wait() for a condition
  that could also be tripped by an interrupt works fine unless the OS is
  up and running but hasn't set interrupts up yet".
- ipmi: Reduce ipmi_queue_msg_sync() polling loop time to 10ms

  On a plain boot, this reduces the time spent in OPAL by ~170ms on
  p9dsu. This is due to hiomap (currently) using synchronous IPMI
  messages.

  It will also *significantly* reduce latency on runtime flash
  operations for hiomap, as we'll spend typically 10-20ms in OPAL
  rather than 100-200ms. It's not an ideal solution to that, but
  it's a quick and obvious win for jitter.
- core/device: NULL pointer dereference fix
- core/flash: NULL pointer dereference fixes
- core/cpu: Call memset with proper cpu_thread offset
- libflash: Add ipmi-hiomap, and prefer it for PNOR access

  ipmi-hiomap implements the PNOR access control protocol formerly known
  as "the mbox protocol" but uses IPMI instead of the AST LPC mailbox as a
  transport. As there is no-longer any mailbox involved in this alternate
  implementation the old protocol name is quite misleading, and so it has
  been renamed to "the hiomap protoocol" (Host I/O Mapping protocol). The
  same commands and events are used though this client-side implementation
  assumes v2 of the protocol is supported by the BMC.

  The code is a heavily-reworked copy of the mbox-flash source and is
  introduced this way to allow for the mbox implementation's eventual
  removal.

  mbox-flash should in theory be renamed to mbox-hiomap for consistency,
  but as it is on life-support effective immediately we may as well just
  remove it entirely when the time is right.
- opal/hmi: Handle early HMIs on thread0 when secondaries are still in OPAL.

  When primary thread receives a CORE level HMI for timer facility errors
  while secondaries are still in OPAL, thread 0 ends up in rendez-vous
  waiting for secondaries to get into hmi handling. This is because OPAL
  runs with MSR(EE=0) and hence HMIs are delayed on secondary threads until
  they are given to Linux OS. Fix this by adding a check for secondary
  state and force them in hmi handling by queuing job on secondary threads.

  I have tested this by injecting HDEC parity error very early during Linux
  kernel boot. Recovery works fine for non-TB errors. But if TB is bad at
  this very eary stage we already doomed.

  Without this patch we see: ::

    [  285.046347408,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c
    [  285.051160609,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c
    [  285.055359021,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
    [  285.055361439,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e14000) Timer Facility Error
    [  286.232183823,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc1)
    [  287.409002056,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc1)
    [  289.073820164,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc1)
    [  290.250638683,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc2)
    [  291.427456821,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc2)
    [  293.092274807,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc2)
    [  294.269092904,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc3)
    [  295.445910944,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc3)
    [  297.110728970,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc3)

  After this patch: ::

    [  259.401719351,7] OPAL: Start CPU 0x0841 (PIR 0x0841) -> 0x000000000000a83c
    [  259.406259572,7] OPAL: Start CPU 0x0842 (PIR 0x0842) -> 0x000000000000a83c
    [  259.410615534,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c
    [  259.415444519,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c
    [  259.419641401,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
    [  259.419644124,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e04000) Timer Facility Error
    [  259.419650678,7] HMI: Sending hmi job to thread 1
    [  259.419652744,7] HMI: Sending hmi job to thread 2
    [  259.419653051,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
    [  259.419654725,7] HMI: Sending hmi job to thread 3
    [  259.419654916,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
    [  259.419658025,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
    [  259.419658406,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:2: TFMR(2e12002870e04000) Timer Facility Error
    [  259.419663095,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:3: TFMR(2e12002870e04000) Timer Facility Error
    [  259.419655234,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:1: TFMR(2e12002870e04000) Timer Facility Error
    [  259.425109779,7] OPAL: Start CPU 0x0845 (PIR 0x0845) -> 0x000000000000a83c
    [  259.429870681,7] OPAL: Start CPU 0x0846 (PIR 0x0846) -> 0x000000000000a83c
    [  259.434549250,7] OPAL: Start CPU 0x0847 (PIR 0x0847) -> 0x000000000000a83c

- core/cpu: Fix memory allocation for job array

  fixes: 7a3f307e core/cpu: parallelise global CPU register setting jobs

  This bug would result in boot-hang on some configurations due to
  cpu_wait_job() endlessly waiting for the last bogus jobs[cpu->pir] pointer.
- i2c: Fix multiple-enqueue of the same request on NACK

  i2c_request_send() will retry the request if the error is a NAK,
  however it forgets to clear the "ud.done" flag. It will thus
  loop again and try to re-enqueue the same request causing internal
  request list corruption.
- i2c: Ensure ordering between i2c_request_send() and completion

  i2c_request_send loops waiting for a flag "uc.done" set by
  the completion routine, and then look for a result code
  also set by that same completion.

  There is no synchronization, the completion can happen on another
  processor, so we need to order the stores to uc and the reads
  from uc so that uc.done is stored last and tested first using
  memory barriers.
- pci: Clarify power down logic

  Currently pci_scan_bus() unconditionally calls pci_slot_set_power_state()
  when it's finished scanning a bus. This is one of those things that
  makes you go "WHAT?" when you first see it and frankly the skiboot PCI
  code could do with less of that.

Fast Reboot
^^^^^^^^^^^

- fast-reboot: parallel memory clearing

  Arbitrarily pick 16GB as the unit of parallelism, and
  split up clearing memory into jobs and schedule them
  node-local to the memory (or on node 0 if we can't
  work that out because it's the memory up to SKIBOOT_BASE)

  This seems to cut at least ~40% time from memory zeroing on
  fast-reboot on a 256GB Boston system.

  For many systems, scanning PCI takes about as much time as
  zeroing all of RAM, so we may as well do them at the same time
  and cut a few seconds off the total fast reboot time.
- fast-reboot: verify firmware "romem" checksum

  This takes a checksum of skiboot memory after boot that should be
  unchanged during OS operation, and verifies it before allowing a
  fast reboot.

  This is not read-only memory from skiboot's point of view, beause
  it includes things like the opal branch table that gets populated
  during boot.

  This helps to improve the integrity of firmware against host and
  runtime firmware memory scribble bugs.

- core/fast-reboot: print the fast reboot disable reason

  Once things start to go wrong, disable_fast_reboot can be called a
  number of times, so make the first reason sticky, and also print it
  to the console at disable time. This helps with making sense of
  fast reboot disables.
- Add fast-reboot property to /ibm,opal DT node

  this means that if it's permanently disabled on boot, the test suite can
  pick that up and not try a fast reboot test.

Utilities
---------

- pflash: Add --skip option for reading

  Add a --skip=N option to pflash to skip N number of bytes when reading.
  This would allow users to print the VERSION partition without the STB
  header by specifying the --skip=4096 argument, and it's a more generic
  solution rather than making pflash depend on secure/trusted boot code.
- xscom-utils: Rework getsram

  Allow specifying a file on the command line to read OCC SRAM data into.
  If no file is specified then we print it to stdout as text. This is a
  bit inconsistent, but it retains compatibility with the existing tool.
- xscom-utils/getsram: Make it work on P9

  The XSCOM base address of the OCC control registers changed slightly
  between P8 and P9. Fix this up and add a bit of PVR checking so we look
  in the right place.
- opal-prd: Fix opal-prd crash

  Presently callback function from HBRT uses r11 to point to target function
  pointer. r12 is garbage. This works fine when we compile with "-no-pie" option
  (as we don't use r12 to calculate TOC).

  As per ABIv2 : "r12 : Function entry address at global entry point"

  With "-pie" compilation option, we have to set r12 to point to global function
  entry point. So that we can calculate TOC properly.

  Crash log without this patch: ::

      opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1


Development and Debugging
-------------------------

- core/lock: Use try_lock_caller() in lock_caller() to capture owner

  Otherwise we can get reports of core/lock.c owning the lock, which is
  not helpful when tracking down ownership issues.
- core/flash: Emit a warning if Skiboot version doesn't match

  This means you'll get a warning that you've modified skiboot separately
  to the rest of the PNOR image, which can be useful in determining what
  firmware is actually running on a machine.
- gcov: link in ctors* as newer GCC doesn't group them all

  It seems that newer toolchains get us multiple ctors sections to link in
  rather than just one. If we discard them (as we were doing), then we
  don't have a working gcov build (and we get the "doesn't look sane"
  warning on boot).
- core/flash: Log return code when ffs_init() fails

  Knowing the return code is at least better than not knowing the return
  code.
- gcov: Fix building with GCC8
- travis/ci: rework Dockerfiles to produce build artifacts

  ubuntu-latest was also missing clang, as ubuntu-latest is closer to
  ubuntu 18.04 than 16.04
- cpu: add cpu_queue_job_on_node()

  Add a job scheduling API which will run the job on the requested
  chip_id (or return failure).
- opal-ci: Build old dtc version for fedora 28

  There are patches that will go into dtc to fix the issues we hit, but
  for the moment let's just build and use a slightly older version.
- mem_region: Merge similar allocations when dumping

  Currently we print one line for each allocation done at runtime when
  dumping the memory allocations. We do a few thousand allocations at
  boot so this can result in a huge amount of text being printed which
  is a) slow to print, and b) Can result in the log buffer overflowing
  which destroys otherwise useful information.

  This patch adds a de-duplication to this memory allocation dump by
  merging "similar" allocations (same location, same size) into one.

  Unfortunately, the algorithm used to do the de-duplication is quadratic,
  but considering we only dump the allocations in the event of a fatal
  error I think this is acceptable. I also did some benchmarking and found
  that on a ZZ it takes ~3ms to do a dump with 12k allocations. On a Zaius
  it's slightly longer at about ~10ms for 10k allocs. However, the
  difference there was due to the output being written to the UART.

  This patch also bumps the log level to PR_NOTICE. PR_INFO messages are
  suppressed at the default log level, which probably isn't something you
  want considering we only dump the allocations when we run out of skiboot
  heap space.
- core/lock: fix timeout warning causing a deadlock false positive

  If a lock waiter exceeds the warning timeout, it prints a message
  while still registered as requesting the lock. Printing the message
  can take locks, so if one is held when the owner of the original
  lock tries to print a message, it will get a false positive deadlock
  detection, which brings down the system.

  This can easily be hit when there is a lot of HMI activity from a
  KVM guest, where the timebase was not returned to host timebase
  before calling the HMI handler.
- hw/p8-i2c: Print the set error bits

  This is purely to save me from having to look it up every time someone
  gets an I2C error.
- init: Fix starting stripped kernel

  Currently if we try to run a raw/stripped binary kernel (ie. without
  the elf header) we crash with: ::

      [    0.008757768,5] INIT: Waiting for kernel...
      [    0.008762937,5] INIT: platform wait for kernel load failed
      [    0.008768171,5] INIT: Assuming kernel at 0x20000000
      [    0.008779241,3] INIT: ELF header not found. Assuming raw binary.
      [    0.017047348,5] INIT: Starting kernel at 0x0, fdt at 0x3044b230 14339 bytes
      [    0.017054251,0] FATAL: Kernel is zeros, can't execute!
      [    0.017059054,0] Assert fail: core/init.c:590:0
      [    0.017065371,0] Aborting!

  This is because we haven't set kernel_entry correctly in this path.
  This fixes it.
- cpu: Better output when waiting for a very long job

  Instead of printing at the end if the job took more than 1s,
  print in the loop every 30s along with a backtrace. This will
  give us some output if the job is deadlocked.
- lock: Fix interactions between lock dependency checker and stack checker

  The lock dependency checker does a few nasty things that can cause
  re-entrancy deadlocks in conjunction with the stack checker or
  in fact other debug tests.

  A lot of it revolves around taking a new lock (dl_lock) as part
  of the locking process.

  This tries to fix it by making sure we do not hit the stack
  checker while holding dl_lock.

  We achieve that in part by directly using the low-level __try_lock
  and manually unlocking on the dl_lock, and making some functions
  "nomcount".

  In addition, we mark the dl_lock as being in the console path to
  avoid deadlocks with the UART driver.

  We move the enabling of the deadlock checker to a separate config
  option from DEBUG_LOCKS as well, in case we chose to disable it
  by default later on.
- xscom-utils/adu_scoms.py: run 2to3 over it
- clang: -Wno-error=ignored-attributes

Mambo Platform
^^^^^^^^^^^^^^

- mambo: Merge PMEM_DISK and PMEM_VOLATILE code

  PMEM_VOLATILE and PMEM_DISK can't be used together and are basically
  copies of the same code.

  This merges the two and allows them used together.  Same API is kept.
- hw/chiptod: test QUIRK_NO_CHIPTOD in opal_resync_timebase

  This allows some test coverage of deep stop states in Linux with
  Mambo.
- core/mem_region: mambo reserve kernel payload areas

  Mambo image payloads get overwritten by the OS and by
  fast reboot memory clearing because they have no region
  defined. Add them, which allows fast reboot to work.

Qemu platform
^^^^^^^^^^^^^

- nx: Don't abort on missing NX when using a QEMU machine

  These don't have an NX node (and probably never will) as they
  don't provide any coprocessor. However, the DARN instruction
  works so this abort is unnecessary.

POWER8 Platforms
----------------
- SBE-p8: Do all sbe timer update with xscom lock held

  Without this, on some P8 platforms, we could (falsely) think the SBE timer
  had stalled getting the dreaded "timer stuck" message.

  The code was doing the mftb() to set the start of the timeout period while
  *not* holding the lock, so the 1ms timeout started sometime when somebody
  else had the xscom lock.

  The simple solution is to just do the whole routine holding the xscom lock,
  so do it that way.

Vesnin Platform
^^^^^^^^^^^^^^^
- platforms/astbmc/vesnin: Send list of PCI devices to BMC through IPMI

  Implements sending a list of installed PCI devices through IPMI protocol.
  Each PCI device description is sent as a standalone IPMI message.
  A list of devices can be gathered from separate messages using the
  session identifier. The session Id is an incremental counter that is
  updated at the start of synchronization session.


POWER9 Platforms
----------------

- STOP API: API conditionally supports 255 SCOM restore entries for each quad.
- hdata/i2c: Skip unknown device type

  Do not add unknown I2C devices to device tree.
- hdata/i2c: Add whitelisting for Host I2C devices

  Many of the devices that we get information about through HDAT are for
  use by firmware rather than the host operating system. This patch adds
  a boolean flag to hdat_i2c_info structure that indicates whether devices
  with a given purpose should be reserved for use inside of OPAL (or some
  other firmware component, such as the OCC).
- hdata/iohub: Fix Cumulus Hub ID number
- opal/hmi: Wakeup the cpu before reading core_fir

  When stop state 5 is enabled, reading the core_fir during an HMI can
  result in a xscom read error with xscom_read() returning an
  OPAL_XSCOM_PARTIAL_GOOD error code and core_fir value of all FFs. At
  present this return error code is not handled in decode_core_fir()
  hence the invalid core_fir value is sent to the kernel where it
  interprets it as a FATAL hmi causing a system check-stop.

  This can be prevented by forcing the core to wake-up using before
  reading the core_fir. Hence this patch wraps the call to
  read_core_fir() within calls to dctl_set_special_wakeup() and
  dctl_clear_special_wakeup().
- xive: Disable block tracker

  Due to some HW errata, the block tracking facility (performance optimisation
  for large systems) should be disabled on Nimbus chips. Disable it unconditionally
  for now.
- opal/hmi: Ignore debug trigger inject core FIR.

  Core FIR[60] is a side effect of the work around for the CI Vector Load
  issue in DD2.1. Usually this gets delivered as HMI with HMER[17] where
  Linux already ignores it. But it looks like in some cases we may happen
  to see CORE_FIR[60] while we are already in Malfunction Alert HMI
  (HMER[0]) due to other reasons e.g. CAPI recovery or NPU xstop. If that
  happens then just ignore it instead of crashing kernel as not recoverable.
- hdata: Make sure reserved node name starts with "ibm, "

  HDAT does not provide consistent label format for reserved memory label.
  Few starts with "ibm," while few other starts with component name.
- hdata: Fix dtc warnings

  Fix dtc warnings related to mcbist node. ::

    Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
    Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
    Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
    Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)

  Ideally we should add proper xscom range here... but we are not getting that
  information in HDAT today. Lets fix warning until we get proper data in HDAT.

PHB4
^^^^

- phb4: Generate checkstop on AIB ECC corr/uncorr for DD2.0 parts

  On DD2.0 parts, PCIe ECC protection is not warranted in the response
  data path. Thus, for these parts, we need to flag any ECC errors
  detected from the adjacent AIB RX Data path so the part can be
  replaced.

  This patch configures the FIRs so that we escalate these AIB ECC
  errors to a checkstop so the parts can be replaced.
- phb4: Reset pfir and nfir if new errors reported during ETU reset

  During fast-reboot new PEC errors can be latched even after ETU-Reset
  is asserted. This will result in values of variables nfir_cache and
  pfir_cache to be out of sync.

  During step-2 of CRESET nfir_cache and pfir_cache values are used to
  bring the PHB out of reset state. However if these variables are out
  as noted above of date the nfir/pfir registers are never reset
  completely and ETU still remains frozen.

  Hence this patch updates step-2 of phb4_creset to re-read the values of
  nfir/pfir registers to check if any new errors were reported after
  ETU-reset was asserted, report these new errors and reset the
  nfir/pfir registers. This should bring the ETU out of reset
  successfully.
- phb4: Disable nodal scoped DMA accesses when PB pump mode is enabled

  By default when a PCIe device issues a read request via the PHB it is first
  issued with nodal scope. When accessing GPU memory the NPU does not know at the
  time of response if the requested memory page is off node or not. Therefore
  every read of GPU memory by a PHB is retried with larger scope which introduces
  bandwidth and latency issues.

  On smaller boxes which have pump mode enabled nodal and group scoped reads are
  treated the same and both types of request are broadcast to one chip. Therefore
  we can avoid the retry by disabling nodal scope on the PHB for these boxes. On
  larger boxes nodal (single chip) and group (multiple chip) scoped reads are
  treated differently. Therefore we avoid disabling nodal scope on large boxes
  which have pump mode disabled to avoid all PHB requests being broadcast to
  multiple chips.
- phb4/capp: Only reset FIR bits that cause capp machine check

  During CAPP recovery do_capp_recovery_scoms() will reset the CAPP Fir
  register just after CAPP recovery is completed. This has an
  unintentional side effect of preventing PRD from analyzing and
  reporting this error. If PRD tries to read the CAPP FIR after opal has
  already reset it, then it logs a critical error complaining "No active
  error bits found".

  To prevent this from happening we update do_capp_recovery_scoms() to
  only reset fir bits that cause CAPP machine check (local xstop). This
  is done by reading the CAPP Fir Action0/1 & Mask registers and
  generating a mask which is then written on CAPP_FIR_CLEAR register.

- phb4: Check for RX errors after link training

  Some PHB4 PHYs can get stuck in a bad state where they are constantly
  retraining the link. This happens transparently to skiboot and Linux
  but will causes PCIe to be slow. Resetting the PHB4 clears the
  problem.

  We can detect this case by looking at the RX errors count where we
  check for link stability. This patch does this by modifying the link
  optimal code to check for RX errors. If errors are occurring we
  retrain the link irrespective of the chip rev or card.

  Normally when this problem occurs, the RX error count is maxed out at
  255. When there is no problem, the count is 0. We chose 8 as the max
  rx errors value to give us some margin for a few errors. There is also
  a knob that can be used to set the error threshold for when we should
  retrain the link. ie ::

      nvram -p ibm,skiboot --update-config phb-rx-err-max=8

- hw/phb4: Add a helper to dump the PELT-V

  The "Partitionable Endpoint Lookup Table (Vector)" is used by the PHB
  when processing EEH events. The PELT-V defines which PEs should be
  additionally frozen in the event of an error being flagged on a
  given PE. Knowing the state of the PELT-V is sometimes useful for
  debugging PHB issues so this patch adds a helper to dump it.

- hw/phb4: Print the PEs in the EEH dump in hex

  Linux always displays the PE number in hexidecimal while skiboot
  displays the PEST index (PE number) in decimal. This makes correlating
  errors between Skiboot and Linux more annoying than it should be so
  this patch makes Skiboot print the PEST number in hex.

- phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth

  We reallocate additional 16/8 DMA-Read engines allocated to stack0/1
  on PEC2 respectively. This is needed to improve bandwidth available to
  the Mellanox CX5 adapter when trying to read GPU memory (GPU-Direct).

  If kernel cxl driver indicates a request to allocate maximum possible
  DMA read engines when calling enable_capi_mode() and card is attached
  to PEC2/stack0 slot then we assume its a Mellanox CX5 adapter. We then
  allocate additional 16/8 extra DMA read engines to stack0 and stack1
  respectively on PEC2. This is done by populating the
  XPEC_PCI_PRDSTKOVR and XPEC_NEST_READ_STACK_OVERRIDE as suggested by
  the h/w team.
- phb4: Enable PHB MMIO-0/1 Bars only when mmio window exists

  Presently phb4_probe_stack() will always enable PHB MMIO0/1 windows
  even if they doesn't exist in phy_map. Hence we do some minor shuffling
  in the phb4_probe_stack() so that MMIO-0/1 Bars are only enabled if
  there corresponding MMIO window exists in the phy_map. In case phy_map
  for an mmio window is '0' we set the corresponding BAR register to
  '0'.
- hw/phb4: Use local_alloc for phb4 structures

  Struct phb4 is fairly heavyweight at 283664 bytes. On systems with
  6x PHBs per socket this results in using 3.2MB of heap space the PHB
  structures alone. This is a fairly large chunk of our 12MB heap and
  on systems with particularly large PCIe topologies, or additional
  PHBs we can fail to boot because we cannot allocate space for the
  FDT blob.

  This patch switches to using local_alloc() for the PHB structures
  so they don't consume too large a portion of our 12MB heap space.
- phb4: Fix typo in disable lane eq code

  In this commit ::

      commit 737c0ba3d72b8aab05a765a9fc111a48faac0f75
      Author: Michael Neuling <mikey@neuling.org>
      Date:   Thu Feb 22 10:52:18 2018 +1100
      phb4: Disable lane eq when retrying some nvidia GEN3 devices

  We made a typo and set PH2 twice. This fixes it.

  It worked previously as if only phase 2 (PH2) is set it, skips phase 2
  and phase 3 (PH3).
- phb4: Don't probe a PHB if its garded

  Presently phb4_probe_stack() causes an exception while trying to probe
  a PHB if its garded. This causes skiboot to go into a reboot loop with
  following exception log: ::

     ***********************************************
     Fatal MCE at 000000003006ecd4   .probe_phb4+0x570
     CFAR : 00000000300b98a0
     <snip>
     Aborting!
    CPU 0018 Backtrace:
     S: 0000000031cc37e0 R: 000000003001a51c   ._abort+0x4c
     S: 0000000031cc3860 R: 0000000030028170   .exception_entry+0x180
     S: 0000000031cc3a40 R: 0000000000001f10 *
     S: 0000000031cc3c20 R: 000000003006ecb0   .probe_phb4+0x54c
     S: 0000000031cc3e30 R: 0000000030014ca4   .main_cpu_entry+0x5b0
     S: 0000000031cc3f00 R: 0000000030002700   boot_entry+0x1b8

  This is caused as phb4_probe_stack() will ignore all xscom read/write
  errors to enable PHB Bars and then tries to perform an mmio to read
  PHB Version registers that cause the fatal MCE.

  We fix this by ignoring the PHB probe if the first xscom_write() to
  populate the PHB Bar register fails, which indicates that there is
  something wrong with the PHB.
- phb4: Workaround PHB errata with CFG write UR/CA errors

  If the PHB encounters a UR or CA status on a CFG write, it will
  incorrectly freeze the wrong PE. Instead of using the PE# specified
  in the CONFIG_ADDRESS register, it will use the PE# of whatever
  MMIO occurred last.

  Work around this disabling freeze on such errors
- phb4: Handle allocation errors in phb4_eeh_dump_regs()

  If the zalloc fails (and it can be a rather large allocation),
  we will overwite memory at 0 instead of failing.
- phb4: Don't try to access non-existent PEST entries

  In a POWER9 chip, some PHB4s have 256 PEs, some have 512.

  Currently, the diagnostics code retrieves 512 unconditionally,
  which is wrong and causes us to incorrectly report bogus values
  for the "high" PEs on the small PHBs.

  Use the actual number of implemented PEs instead

CAPI2
^^^^^

- phb4/capp: Use link width to allocate STQ engines to CAPP

  Update phb4_init_capp_regs() to allocates STQ Engines to CAPP/PEC2
  based on link width instead of always assuming it to x8.

  Also re-factor the function slightly to evaluate the link-width only
  once and cache it so that it can also be used to allocate DMA read
  engines.
- phb4/capp: Update DMA read engines set in APC_FSM_READ_MASK based on link-width

  Commit 47c09cdfe7a3("phb4/capp: Calculate STQ/DMA read engines based
  on link-width for PEC") update the CAPP init sequence by calculating
  the needed STQ/DMA-read engines based on link width and populating it
  in XPEC_NEST_CAPP_CNTL register. This however needs to be synchronized
  with the value set in CAPP APC FSM Read Machine Mask Register.

  Hence this patch update phb4_init_capp_regs() to calculate the link
  width of the stack on PEC2 and populate the same values as previously
  populated in PEC CAPP_CNTL register.
- capp: Fix the capp recovery timeout comparison

  The current capp recovery timeout control loop in
  do_capp_recovery_scoms() uses a wrong comparison for return value of
  tb_compare(). This may cause do_capp_recovery_scoms() to report an
  timeout earlier than the 168ms stipulated time.

  The patch fixes this by updating the loop timeout control branch in
  do_capp_recovery_scoms() to use the correct enum tb_cmpval.
- phb4: Disable 32-bit MSI in capi mode

  If a capi device does a DMA write targeting an address lower than 4GB,
  it does so through a 32-bit operation, per the PCI spec. In capi mode,
  the first TVE entry is configured in bypass mode, so the address is
  valid. But with any (bad) luck, the address could be 0xFFFFxxxx, thus
  looking like a 32-bit MSI.

  We currently enable both 32-bit and 64-bit MSIs, so the PHB will
  interpret the DMA write as a MSI, which very likely results in an EEH
  (MSI with a bad payload size).

  We can fix it by disabling 32-bit MSI when switching the PHB to capi
  mode. Capi devices are 64-bit.

NVLINK2
^^^^^^^
- npu2: Add support for relaxed-ordering mode

  Some device drivers support out of order access to GPU memory. This does
  not affect the CPU view of memory but it does affect the GPU view of
  memory. It should only be enabled if the GPU driver has requested it.

  Add OPAL APIs allowing the driver to query relaxed ordering state or
  request it to be set for a device. Current hardware only allows relaxed
  ordering to be enabled per PCIe root port. So the code here doesn't
  enable relaxed ordering until it has been explicitly requested for every
  device on the port.
- Add the other 7 ATSD registers to the device tree.
- npu2/hw-procedures: Don't open code NPU2_NTL_MISC_CFG2_BRICK_ENABLE

  Name this bit properly. There's a lot more cleanup like this to be done,
  but I'm catching this one now as part of some related changes.
- npu2/hw-procedures: Enable parity and credit overflow checks

  Enable these error checking features by setting the appropriate bits in
  our one-off initialization of each "NTL Misc Config 2" register.

  The exception is NDL RX parity checking, which should be disabled during
  the link training procedures.
- npu2: Use correct kill type for TCE invalidation

  kill_type is enum of OPAL_PCI_TCE_KILL_PAGES, OPAL_PCI_TCE_KILL_PE,
  OPAL_PCI_TCE_KILL_ALL and phb4_tce_kill() gets it right but
  npu2_tce_kill() uses OPAL_PCI_TCE_KILL which is an OPAL API token.

  This fixes an obvious mistype.

OpenCAPI
^^^^^^^^

- Support OpenCAPI on Witherspoon platform
- npu2-opencapi: Enable presence detection on ZZ

  Presence detection for opencapi adapters was broken for ZZ planars v3
  and below. All ZZ systems currently used in the lab have had their
  planar upgraded, so we can now remove the override we had to force
  presence and activate presence detection. Which should improve boot
  time.

  Considering the state of opal support on ZZ, this is really only for
  lab usage on BML. The opencapi enablement team has okay'd the
  change. In the unlikely case somebody tries opencapi on an old ZZ, the
  presence detection through i2c will show that no adapter is present
  and skiboot won't try to access or train the link.
- npu2-opencapi: Don't send commands to NPU when link is down

  Even if an opencapi link is down, we currently always try to issue a
  config read operation when probing for PCI devices, because of the
  default scan map used for an opencapi PHB. The config operation fails,
  as expected, but it can also raise a FIR bit and trigger an HMI.

  For opencapi, there's no root device like for a "normal" PCI PHB, so
  there's no reason to do the config operation. To fix it, we keep the
  scan map blank by default, and only add a device once the link is
  trained.
- opal/hmi: Catch NPU2 HMIs for opencapi

  HMIs for NPU2 are filtered with the 'compatible' string of the PHB, so
  add opencapi to the mix.
- occ: Wait if OCC GPU presence status not immediately available

  It takes a few seconds for the OCC to set everything up in order to read
  GPU presence. At present, we try to kick off OCC initialisation as early as
  possible to maximise the time it has to read GPU presence.

  Unfortunately sometimes that's not enough, so add a loop in
  occ_get_gpu_presence() so that on the first time we try to get GPU presence
  we keep trying for up to 2 seconds. Experimentally this seems to be
  adequate.
- hw/npu2-hw-procedures: Enable RX auto recal on OpenCAPI links

  The RX_RC_ENABLE_AUTO_RECAL flag is required on OpenCAPI but not NVLink.

  Traditionally, Hostboot sets this value according to the machine type.
  However, now that Witherspoon supports both NVLink and OpenCAPI, it can't
  tell whether or not a link is OpenCAPI.

  So instead, set it in skiboot, where it will only be triggered after we've
  done device detection and found an OpenCAPI device.
- hw/npu2-opencapi: Fix setting of supported OpenCAPI templates

  In opal_npu_tl_set(), we made a typo that means the OPAL_NPU_TL_SET call
  may not clear the enable bits for templates that were previously enabled
  but are now disabled.

  Fix the typo so we clear NPU2_OTL_CONFIG1_TX_TEMP2_EN as well as
  TEMP{1,3}_EN.

Barreleye G2 and Zaius platforms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- zaius: Add a slot table
- zaius: Add slots for the Barreleye G2 HDD rack

  The Barreleye G2 is distinct from the Zaius in that it features a 24
  Bay NVMe/SATA HDD rack. To provide meaningful slot names for each NVMe
  device we need to define a slot table for the NVMe capable HDD bays.

  Unfortunately this isn't straightforward because the PCIe path to the
  NVMe devices isn't fixed. The PCIe topology is something like:
  P9 -> HBA card -> 9797 switch -> 20x NVMe HDD slots

  The 9797 switch is partitioned into two (or four) virtual switches which
  allow multiple HBA cards to be used (e.g. one per socket). As a result
  the exact BDFN of the ports will vary depending on how the system is
  configured.

  That said, the virtual switch configuration of the 9797 does not change
  the device and function numbers of the switch downports. This means that
  we can define a single slot table that maps switch ports to the NVMe bay
  names.

  Unfortunately we still need to guess which bus to use this table on, so
  we assume that any switch downport we find with the PEX9797 VDID is part
  of the 9797 that supports the HDD rack.

FSP based platforms (firenze and ZZ)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- phb4/capp: Update the expected Eye-catcher for CAPP ucode lid

  Currently on a FSP based P9 system load_capp_code() expects CAPP ucode
  lid header to have eye-catcher magic of 'CAPPPSLL'. However skiboot
  currently supports CAPP ucode only lids that have a eye-catcher magic
  of 'CAPPLIDH'. This prevents skiboot from loading the ucode with this
  error message: ::

    CAPP: ucode header invalid

  We fix this issue by updating load_capp_ucode() to use the eye-catcher
  value of 'CAPPLIDH' instead of 'CAPPPSLL'.

- FSP: Improve Reset/Reload log message

  Below message is confusing. Lets make it clear.

  FSP sends "R/R complete notification" whenever there is a dump. We use `flag`
  to identify whether its its R/R completion -OR- just new dump notification. ::

    [  483.406351956,6] FSP: SP says Reset/Reload complete
    [  483.406354278,5] DUMP: FipS dump available. ID = 0x1a00001f [size: 6367640 bytes]
    [  483.406355968,7]   A Reset/Reload was NOT done

Witherspoon platform
^^^^^^^^^^^^^^^^^^^^

- platforms/astbmc/witherspoon: Implement OpenCAPI support

  OpenCAPI on Witherspoon is slightly more involved than on Zaius and ZZ, due
  to the OpenCAPI links using the SXM2 connectors that are used for NVLink
  GPUs.

  This patch adds the regular OpenCAPI platform information, and also a
  Witherspoon-specific presence detection callback that uses the previously
  added OCC GPU presence detection to figure out the device types plugged
  into each SXM2 socket.

  The SXM2 connectors are capable of carrying 2 OpenCAPI links, and future
  OpenCAPI devices are expected to make use of this. However, we don't yet
  support ganged links and the various implications that has for handling
  things like device reset, so for now, we only enable 1 brick per device.