| .. _skiboot-6.2-rc1: |
| |
| skiboot-6.2-rc1 |
| =============== |
| |
| skiboot v6.2-rc1 was released on Monday November 19th 2018. It is the first |
| release candidate of skiboot 6.2, which will become the new stable release |
| of skiboot following the 6.1 release, first released July 11th 2018. |
| |
| Skiboot 6.2 will mark the basis for op-build v2.2. |
| |
| skiboot v6.2-rc1 contains all bug fixes as of :ref:`skiboot-6.0.13`, |
| and :ref:`skiboot-5.4.10` (the currently maintained |
| stable releases). |
| |
| For how the skiboot stable releases work, see :ref:`stable-rules` for details. |
| |
| This release has been a longer cycle than typical for a variety of reasons. It |
| also contains a lot of cleanup work and minor bug fixes (much like skiboot 6.1 |
| did). |
| |
| Over skiboot 6.1, we have the following changes: |
| |
| General |
| ------- |
| |
| - cpu: Quieten OS endian switch messages |
| |
| Users see these when loading an OS from Petitboot: :: |
| |
| [ 119.486794100,5] OPAL: Switch to big-endian OS |
| [ 120.022302604,5] OPAL: Switch to little-endian OS |
| |
| Which is expected and doesn't provide any information the user can act |
| on. Switch them to PR_INFO so they still appear in the log, but not on |
| the serial console. |
| - Recognise signed VERSION partition |
| |
| A few things need to change to support a signed VERSION partition: |
| |
| - A signed VERSION partition will be 4K + SECURE_BOOT_HEADERS_SIZE (4K). |
| - The VERSION partition needs to be loaded after secure/trusted boot is |
| set up, and therefore after nvram_init(). |
| - Added to the trustedboot resources array. |
| |
| This also moves the ipmi_dt_add_bmc_info() call to after |
| flash_dt_add_fw_version() since it adds info to ibm,firmware-versions. |
| - Run pollers in time_wait() when not booting |
| |
| This only bit us hard with hiomap in one scenario. |
| |
| Our OPAL API has been OPAL_POLL_EVENTS may be needed to make forward |
| progress on ongoing operations, and the internal to skiboot API has been |
| that time_wait() of a suitable time will run pollers (on at least one |
| CPU) to help ensure forward progress can be made. |
| |
| In a perfect world, interrupts are used but they may: a) be disabled, or |
| b) the thing we're doing can't use interrupts because computers are |
| generally terrible. |
| |
| Back in 3db397ea5892a (circa 2015), we changed skiboot so that we'd run |
| pollers only on the boot CPU, and not if we held any locks. This was to |
| reduce the chance of programming code that could deadlock, as well as to |
| ensure that we didn't just thrash all the cachelines for running pollers |
| all over a large system during boot, or hard spin on the same locks on |
| all secondary CPUs. |
| |
| The problem arises if the OS we're booting makes an OPAL call early on, |
| with interrupts disabled, that requires a poller to run to make forward |
| progress. An example of this would be OPAL_WRITE_NVRAM early in Linux |
| boot (where Linux sets up the partitions it wants) - something that |
| occurs iff we've had to reformat NVRAM this boot (i.e. first boot or |
| corrupted NVRAM). |
| |
| The hiomap implementation should arguably *not* rely on synchronous IPMI |
| messages, but this is a future improvement (as was for mbox before it). |
| The mbox-flash code solved this problem by spinning on check_timers(). |
| |
| More generically though, the approach of running the pollers when no |
| longer booting means we behave more in line with what the API is meant |
| to be, rather than have this odd case of "time_wait() for a condition |
| that could also be tripped by an interrupt works fine unless the OS is |
| up and running but hasn't set interrupts up yet". |
| - ipmi: Reduce ipmi_queue_msg_sync() polling loop time to 10ms |
| |
| On a plain boot, this reduces the time spent in OPAL by ~170ms on |
| p9dsu. This is due to hiomap (currently) using synchronous IPMI |
| messages. |
| |
| It will also *significantly* reduce latency on runtime flash |
| operations for hiomap, as we'll spend typically 10-20ms in OPAL |
| rather than 100-200ms. It's not an ideal solution to that, but |
| it's a quick and obvious win for jitter. |
| - core/device: NULL pointer dereference fix |
| - core/flash: NULL pointer dereference fixes |
| - core/cpu: Call memset with proper cpu_thread offset |
| - libflash: Add ipmi-hiomap, and prefer it for PNOR access |
| |
| ipmi-hiomap implements the PNOR access control protocol formerly known |
| as "the mbox protocol" but uses IPMI instead of the AST LPC mailbox as a |
| transport. As there is no-longer any mailbox involved in this alternate |
| implementation the old protocol name is quite misleading, and so it has |
| been renamed to "the hiomap protoocol" (Host I/O Mapping protocol). The |
| same commands and events are used though this client-side implementation |
| assumes v2 of the protocol is supported by the BMC. |
| |
| The code is a heavily-reworked copy of the mbox-flash source and is |
| introduced this way to allow for the mbox implementation's eventual |
| removal. |
| |
| mbox-flash should in theory be renamed to mbox-hiomap for consistency, |
| but as it is on life-support effective immediately we may as well just |
| remove it entirely when the time is right. |
| - opal/hmi: Handle early HMIs on thread0 when secondaries are still in OPAL. |
| |
| When primary thread receives a CORE level HMI for timer facility errors |
| while secondaries are still in OPAL, thread 0 ends up in rendez-vous |
| waiting for secondaries to get into hmi handling. This is because OPAL |
| runs with MSR(EE=0) and hence HMIs are delayed on secondary threads until |
| they are given to Linux OS. Fix this by adding a check for secondary |
| state and force them in hmi handling by queuing job on secondary threads. |
| |
| I have tested this by injecting HDEC parity error very early during Linux |
| kernel boot. Recovery works fine for non-TB errors. But if TB is bad at |
| this very eary stage we already doomed. |
| |
| Without this patch we see: :: |
| |
| [ 285.046347408,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c |
| [ 285.051160609,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c |
| [ 285.055359021,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000 |
| [ 285.055361439,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e14000) Timer Facility Error |
| [ 286.232183823,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc1) |
| [ 287.409002056,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc1) |
| [ 289.073820164,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc1) |
| [ 290.250638683,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc2) |
| [ 291.427456821,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc2) |
| [ 293.092274807,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc2) |
| [ 294.269092904,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc3) |
| [ 295.445910944,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc3) |
| [ 297.110728970,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc3) |
| |
| After this patch: :: |
| |
| [ 259.401719351,7] OPAL: Start CPU 0x0841 (PIR 0x0841) -> 0x000000000000a83c |
| [ 259.406259572,7] OPAL: Start CPU 0x0842 (PIR 0x0842) -> 0x000000000000a83c |
| [ 259.410615534,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c |
| [ 259.415444519,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c |
| [ 259.419641401,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000 |
| [ 259.419644124,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e04000) Timer Facility Error |
| [ 259.419650678,7] HMI: Sending hmi job to thread 1 |
| [ 259.419652744,7] HMI: Sending hmi job to thread 2 |
| [ 259.419653051,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000 |
| [ 259.419654725,7] HMI: Sending hmi job to thread 3 |
| [ 259.419654916,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000 |
| [ 259.419658025,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000 |
| [ 259.419658406,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:2: TFMR(2e12002870e04000) Timer Facility Error |
| [ 259.419663095,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:3: TFMR(2e12002870e04000) Timer Facility Error |
| [ 259.419655234,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:1: TFMR(2e12002870e04000) Timer Facility Error |
| [ 259.425109779,7] OPAL: Start CPU 0x0845 (PIR 0x0845) -> 0x000000000000a83c |
| [ 259.429870681,7] OPAL: Start CPU 0x0846 (PIR 0x0846) -> 0x000000000000a83c |
| [ 259.434549250,7] OPAL: Start CPU 0x0847 (PIR 0x0847) -> 0x000000000000a83c |
| |
| - core/cpu: Fix memory allocation for job array |
| |
| fixes: 7a3f307e core/cpu: parallelise global CPU register setting jobs |
| |
| This bug would result in boot-hang on some configurations due to |
| cpu_wait_job() endlessly waiting for the last bogus jobs[cpu->pir] pointer. |
| - i2c: Fix multiple-enqueue of the same request on NACK |
| |
| i2c_request_send() will retry the request if the error is a NAK, |
| however it forgets to clear the "ud.done" flag. It will thus |
| loop again and try to re-enqueue the same request causing internal |
| request list corruption. |
| - i2c: Ensure ordering between i2c_request_send() and completion |
| |
| i2c_request_send loops waiting for a flag "uc.done" set by |
| the completion routine, and then look for a result code |
| also set by that same completion. |
| |
| There is no synchronization, the completion can happen on another |
| processor, so we need to order the stores to uc and the reads |
| from uc so that uc.done is stored last and tested first using |
| memory barriers. |
| - pci: Clarify power down logic |
| |
| Currently pci_scan_bus() unconditionally calls pci_slot_set_power_state() |
| when it's finished scanning a bus. This is one of those things that |
| makes you go "WHAT?" when you first see it and frankly the skiboot PCI |
| code could do with less of that. |
| |
| Fast Reboot |
| ^^^^^^^^^^^ |
| |
| - fast-reboot: parallel memory clearing |
| |
| Arbitrarily pick 16GB as the unit of parallelism, and |
| split up clearing memory into jobs and schedule them |
| node-local to the memory (or on node 0 if we can't |
| work that out because it's the memory up to SKIBOOT_BASE) |
| |
| This seems to cut at least ~40% time from memory zeroing on |
| fast-reboot on a 256GB Boston system. |
| |
| For many systems, scanning PCI takes about as much time as |
| zeroing all of RAM, so we may as well do them at the same time |
| and cut a few seconds off the total fast reboot time. |
| - fast-reboot: verify firmware "romem" checksum |
| |
| This takes a checksum of skiboot memory after boot that should be |
| unchanged during OS operation, and verifies it before allowing a |
| fast reboot. |
| |
| This is not read-only memory from skiboot's point of view, beause |
| it includes things like the opal branch table that gets populated |
| during boot. |
| |
| This helps to improve the integrity of firmware against host and |
| runtime firmware memory scribble bugs. |
| |
| - core/fast-reboot: print the fast reboot disable reason |
| |
| Once things start to go wrong, disable_fast_reboot can be called a |
| number of times, so make the first reason sticky, and also print it |
| to the console at disable time. This helps with making sense of |
| fast reboot disables. |
| - Add fast-reboot property to /ibm,opal DT node |
| |
| this means that if it's permanently disabled on boot, the test suite can |
| pick that up and not try a fast reboot test. |
| |
| Utilities |
| --------- |
| |
| - pflash: Add --skip option for reading |
| |
| Add a --skip=N option to pflash to skip N number of bytes when reading. |
| This would allow users to print the VERSION partition without the STB |
| header by specifying the --skip=4096 argument, and it's a more generic |
| solution rather than making pflash depend on secure/trusted boot code. |
| - xscom-utils: Rework getsram |
| |
| Allow specifying a file on the command line to read OCC SRAM data into. |
| If no file is specified then we print it to stdout as text. This is a |
| bit inconsistent, but it retains compatibility with the existing tool. |
| - xscom-utils/getsram: Make it work on P9 |
| |
| The XSCOM base address of the OCC control registers changed slightly |
| between P8 and P9. Fix this up and add a bit of PVR checking so we look |
| in the right place. |
| - opal-prd: Fix opal-prd crash |
| |
| Presently callback function from HBRT uses r11 to point to target function |
| pointer. r12 is garbage. This works fine when we compile with "-no-pie" option |
| (as we don't use r12 to calculate TOC). |
| |
| As per ABIv2 : "r12 : Function entry address at global entry point" |
| |
| With "-pie" compilation option, we have to set r12 to point to global function |
| entry point. So that we can calculate TOC properly. |
| |
| Crash log without this patch: :: |
| |
| opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1 |
| |
| |
| Development and Debugging |
| ------------------------- |
| |
| - core/lock: Use try_lock_caller() in lock_caller() to capture owner |
| |
| Otherwise we can get reports of core/lock.c owning the lock, which is |
| not helpful when tracking down ownership issues. |
| - core/flash: Emit a warning if Skiboot version doesn't match |
| |
| This means you'll get a warning that you've modified skiboot separately |
| to the rest of the PNOR image, which can be useful in determining what |
| firmware is actually running on a machine. |
| - gcov: link in ctors* as newer GCC doesn't group them all |
| |
| It seems that newer toolchains get us multiple ctors sections to link in |
| rather than just one. If we discard them (as we were doing), then we |
| don't have a working gcov build (and we get the "doesn't look sane" |
| warning on boot). |
| - core/flash: Log return code when ffs_init() fails |
| |
| Knowing the return code is at least better than not knowing the return |
| code. |
| - gcov: Fix building with GCC8 |
| - travis/ci: rework Dockerfiles to produce build artifacts |
| |
| ubuntu-latest was also missing clang, as ubuntu-latest is closer to |
| ubuntu 18.04 than 16.04 |
| - cpu: add cpu_queue_job_on_node() |
| |
| Add a job scheduling API which will run the job on the requested |
| chip_id (or return failure). |
| - opal-ci: Build old dtc version for fedora 28 |
| |
| There are patches that will go into dtc to fix the issues we hit, but |
| for the moment let's just build and use a slightly older version. |
| - mem_region: Merge similar allocations when dumping |
| |
| Currently we print one line for each allocation done at runtime when |
| dumping the memory allocations. We do a few thousand allocations at |
| boot so this can result in a huge amount of text being printed which |
| is a) slow to print, and b) Can result in the log buffer overflowing |
| which destroys otherwise useful information. |
| |
| This patch adds a de-duplication to this memory allocation dump by |
| merging "similar" allocations (same location, same size) into one. |
| |
| Unfortunately, the algorithm used to do the de-duplication is quadratic, |
| but considering we only dump the allocations in the event of a fatal |
| error I think this is acceptable. I also did some benchmarking and found |
| that on a ZZ it takes ~3ms to do a dump with 12k allocations. On a Zaius |
| it's slightly longer at about ~10ms for 10k allocs. However, the |
| difference there was due to the output being written to the UART. |
| |
| This patch also bumps the log level to PR_NOTICE. PR_INFO messages are |
| suppressed at the default log level, which probably isn't something you |
| want considering we only dump the allocations when we run out of skiboot |
| heap space. |
| - core/lock: fix timeout warning causing a deadlock false positive |
| |
| If a lock waiter exceeds the warning timeout, it prints a message |
| while still registered as requesting the lock. Printing the message |
| can take locks, so if one is held when the owner of the original |
| lock tries to print a message, it will get a false positive deadlock |
| detection, which brings down the system. |
| |
| This can easily be hit when there is a lot of HMI activity from a |
| KVM guest, where the timebase was not returned to host timebase |
| before calling the HMI handler. |
| - hw/p8-i2c: Print the set error bits |
| |
| This is purely to save me from having to look it up every time someone |
| gets an I2C error. |
| - init: Fix starting stripped kernel |
| |
| Currently if we try to run a raw/stripped binary kernel (ie. without |
| the elf header) we crash with: :: |
| |
| [ 0.008757768,5] INIT: Waiting for kernel... |
| [ 0.008762937,5] INIT: platform wait for kernel load failed |
| [ 0.008768171,5] INIT: Assuming kernel at 0x20000000 |
| [ 0.008779241,3] INIT: ELF header not found. Assuming raw binary. |
| [ 0.017047348,5] INIT: Starting kernel at 0x0, fdt at 0x3044b230 14339 bytes |
| [ 0.017054251,0] FATAL: Kernel is zeros, can't execute! |
| [ 0.017059054,0] Assert fail: core/init.c:590:0 |
| [ 0.017065371,0] Aborting! |
| |
| This is because we haven't set kernel_entry correctly in this path. |
| This fixes it. |
| - cpu: Better output when waiting for a very long job |
| |
| Instead of printing at the end if the job took more than 1s, |
| print in the loop every 30s along with a backtrace. This will |
| give us some output if the job is deadlocked. |
| - lock: Fix interactions between lock dependency checker and stack checker |
| |
| The lock dependency checker does a few nasty things that can cause |
| re-entrancy deadlocks in conjunction with the stack checker or |
| in fact other debug tests. |
| |
| A lot of it revolves around taking a new lock (dl_lock) as part |
| of the locking process. |
| |
| This tries to fix it by making sure we do not hit the stack |
| checker while holding dl_lock. |
| |
| We achieve that in part by directly using the low-level __try_lock |
| and manually unlocking on the dl_lock, and making some functions |
| "nomcount". |
| |
| In addition, we mark the dl_lock as being in the console path to |
| avoid deadlocks with the UART driver. |
| |
| We move the enabling of the deadlock checker to a separate config |
| option from DEBUG_LOCKS as well, in case we chose to disable it |
| by default later on. |
| - xscom-utils/adu_scoms.py: run 2to3 over it |
| - clang: -Wno-error=ignored-attributes |
| |
| Mambo Platform |
| ^^^^^^^^^^^^^^ |
| |
| - mambo: Merge PMEM_DISK and PMEM_VOLATILE code |
| |
| PMEM_VOLATILE and PMEM_DISK can't be used together and are basically |
| copies of the same code. |
| |
| This merges the two and allows them used together. Same API is kept. |
| - hw/chiptod: test QUIRK_NO_CHIPTOD in opal_resync_timebase |
| |
| This allows some test coverage of deep stop states in Linux with |
| Mambo. |
| - core/mem_region: mambo reserve kernel payload areas |
| |
| Mambo image payloads get overwritten by the OS and by |
| fast reboot memory clearing because they have no region |
| defined. Add them, which allows fast reboot to work. |
| |
| Qemu platform |
| ^^^^^^^^^^^^^ |
| |
| - nx: Don't abort on missing NX when using a QEMU machine |
| |
| These don't have an NX node (and probably never will) as they |
| don't provide any coprocessor. However, the DARN instruction |
| works so this abort is unnecessary. |
| |
| POWER8 Platforms |
| ---------------- |
| - SBE-p8: Do all sbe timer update with xscom lock held |
| |
| Without this, on some P8 platforms, we could (falsely) think the SBE timer |
| had stalled getting the dreaded "timer stuck" message. |
| |
| The code was doing the mftb() to set the start of the timeout period while |
| *not* holding the lock, so the 1ms timeout started sometime when somebody |
| else had the xscom lock. |
| |
| The simple solution is to just do the whole routine holding the xscom lock, |
| so do it that way. |
| |
| Vesnin Platform |
| ^^^^^^^^^^^^^^^ |
| - platforms/astbmc/vesnin: Send list of PCI devices to BMC through IPMI |
| |
| Implements sending a list of installed PCI devices through IPMI protocol. |
| Each PCI device description is sent as a standalone IPMI message. |
| A list of devices can be gathered from separate messages using the |
| session identifier. The session Id is an incremental counter that is |
| updated at the start of synchronization session. |
| |
| |
| POWER9 Platforms |
| ---------------- |
| |
| - STOP API: API conditionally supports 255 SCOM restore entries for each quad. |
| - hdata/i2c: Skip unknown device type |
| |
| Do not add unknown I2C devices to device tree. |
| - hdata/i2c: Add whitelisting for Host I2C devices |
| |
| Many of the devices that we get information about through HDAT are for |
| use by firmware rather than the host operating system. This patch adds |
| a boolean flag to hdat_i2c_info structure that indicates whether devices |
| with a given purpose should be reserved for use inside of OPAL (or some |
| other firmware component, such as the OCC). |
| - hdata/iohub: Fix Cumulus Hub ID number |
| - opal/hmi: Wakeup the cpu before reading core_fir |
| |
| When stop state 5 is enabled, reading the core_fir during an HMI can |
| result in a xscom read error with xscom_read() returning an |
| OPAL_XSCOM_PARTIAL_GOOD error code and core_fir value of all FFs. At |
| present this return error code is not handled in decode_core_fir() |
| hence the invalid core_fir value is sent to the kernel where it |
| interprets it as a FATAL hmi causing a system check-stop. |
| |
| This can be prevented by forcing the core to wake-up using before |
| reading the core_fir. Hence this patch wraps the call to |
| read_core_fir() within calls to dctl_set_special_wakeup() and |
| dctl_clear_special_wakeup(). |
| - xive: Disable block tracker |
| |
| Due to some HW errata, the block tracking facility (performance optimisation |
| for large systems) should be disabled on Nimbus chips. Disable it unconditionally |
| for now. |
| - opal/hmi: Ignore debug trigger inject core FIR. |
| |
| Core FIR[60] is a side effect of the work around for the CI Vector Load |
| issue in DD2.1. Usually this gets delivered as HMI with HMER[17] where |
| Linux already ignores it. But it looks like in some cases we may happen |
| to see CORE_FIR[60] while we are already in Malfunction Alert HMI |
| (HMER[0]) due to other reasons e.g. CAPI recovery or NPU xstop. If that |
| happens then just ignore it instead of crashing kernel as not recoverable. |
| - hdata: Make sure reserved node name starts with "ibm, " |
| |
| HDAT does not provide consistent label format for reserved memory label. |
| Few starts with "ibm," while few other starts with component name. |
| - hdata: Fix dtc warnings |
| |
| Fix dtc warnings related to mcbist node. :: |
| |
| Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1) |
| Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1) |
| Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1) |
| Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1) |
| |
| Ideally we should add proper xscom range here... but we are not getting that |
| information in HDAT today. Lets fix warning until we get proper data in HDAT. |
| |
| PHB4 |
| ^^^^ |
| |
| - phb4: Generate checkstop on AIB ECC corr/uncorr for DD2.0 parts |
| |
| On DD2.0 parts, PCIe ECC protection is not warranted in the response |
| data path. Thus, for these parts, we need to flag any ECC errors |
| detected from the adjacent AIB RX Data path so the part can be |
| replaced. |
| |
| This patch configures the FIRs so that we escalate these AIB ECC |
| errors to a checkstop so the parts can be replaced. |
| - phb4: Reset pfir and nfir if new errors reported during ETU reset |
| |
| During fast-reboot new PEC errors can be latched even after ETU-Reset |
| is asserted. This will result in values of variables nfir_cache and |
| pfir_cache to be out of sync. |
| |
| During step-2 of CRESET nfir_cache and pfir_cache values are used to |
| bring the PHB out of reset state. However if these variables are out |
| as noted above of date the nfir/pfir registers are never reset |
| completely and ETU still remains frozen. |
| |
| Hence this patch updates step-2 of phb4_creset to re-read the values of |
| nfir/pfir registers to check if any new errors were reported after |
| ETU-reset was asserted, report these new errors and reset the |
| nfir/pfir registers. This should bring the ETU out of reset |
| successfully. |
| - phb4: Disable nodal scoped DMA accesses when PB pump mode is enabled |
| |
| By default when a PCIe device issues a read request via the PHB it is first |
| issued with nodal scope. When accessing GPU memory the NPU does not know at the |
| time of response if the requested memory page is off node or not. Therefore |
| every read of GPU memory by a PHB is retried with larger scope which introduces |
| bandwidth and latency issues. |
| |
| On smaller boxes which have pump mode enabled nodal and group scoped reads are |
| treated the same and both types of request are broadcast to one chip. Therefore |
| we can avoid the retry by disabling nodal scope on the PHB for these boxes. On |
| larger boxes nodal (single chip) and group (multiple chip) scoped reads are |
| treated differently. Therefore we avoid disabling nodal scope on large boxes |
| which have pump mode disabled to avoid all PHB requests being broadcast to |
| multiple chips. |
| - phb4/capp: Only reset FIR bits that cause capp machine check |
| |
| During CAPP recovery do_capp_recovery_scoms() will reset the CAPP Fir |
| register just after CAPP recovery is completed. This has an |
| unintentional side effect of preventing PRD from analyzing and |
| reporting this error. If PRD tries to read the CAPP FIR after opal has |
| already reset it, then it logs a critical error complaining "No active |
| error bits found". |
| |
| To prevent this from happening we update do_capp_recovery_scoms() to |
| only reset fir bits that cause CAPP machine check (local xstop). This |
| is done by reading the CAPP Fir Action0/1 & Mask registers and |
| generating a mask which is then written on CAPP_FIR_CLEAR register. |
| |
| - phb4: Check for RX errors after link training |
| |
| Some PHB4 PHYs can get stuck in a bad state where they are constantly |
| retraining the link. This happens transparently to skiboot and Linux |
| but will causes PCIe to be slow. Resetting the PHB4 clears the |
| problem. |
| |
| We can detect this case by looking at the RX errors count where we |
| check for link stability. This patch does this by modifying the link |
| optimal code to check for RX errors. If errors are occurring we |
| retrain the link irrespective of the chip rev or card. |
| |
| Normally when this problem occurs, the RX error count is maxed out at |
| 255. When there is no problem, the count is 0. We chose 8 as the max |
| rx errors value to give us some margin for a few errors. There is also |
| a knob that can be used to set the error threshold for when we should |
| retrain the link. ie :: |
| |
| nvram -p ibm,skiboot --update-config phb-rx-err-max=8 |
| |
| - hw/phb4: Add a helper to dump the PELT-V |
| |
| The "Partitionable Endpoint Lookup Table (Vector)" is used by the PHB |
| when processing EEH events. The PELT-V defines which PEs should be |
| additionally frozen in the event of an error being flagged on a |
| given PE. Knowing the state of the PELT-V is sometimes useful for |
| debugging PHB issues so this patch adds a helper to dump it. |
| |
| - hw/phb4: Print the PEs in the EEH dump in hex |
| |
| Linux always displays the PE number in hexidecimal while skiboot |
| displays the PEST index (PE number) in decimal. This makes correlating |
| errors between Skiboot and Linux more annoying than it should be so |
| this patch makes Skiboot print the PEST number in hex. |
| |
| - phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth |
| |
| We reallocate additional 16/8 DMA-Read engines allocated to stack0/1 |
| on PEC2 respectively. This is needed to improve bandwidth available to |
| the Mellanox CX5 adapter when trying to read GPU memory (GPU-Direct). |
| |
| If kernel cxl driver indicates a request to allocate maximum possible |
| DMA read engines when calling enable_capi_mode() and card is attached |
| to PEC2/stack0 slot then we assume its a Mellanox CX5 adapter. We then |
| allocate additional 16/8 extra DMA read engines to stack0 and stack1 |
| respectively on PEC2. This is done by populating the |
| XPEC_PCI_PRDSTKOVR and XPEC_NEST_READ_STACK_OVERRIDE as suggested by |
| the h/w team. |
| - phb4: Enable PHB MMIO-0/1 Bars only when mmio window exists |
| |
| Presently phb4_probe_stack() will always enable PHB MMIO0/1 windows |
| even if they doesn't exist in phy_map. Hence we do some minor shuffling |
| in the phb4_probe_stack() so that MMIO-0/1 Bars are only enabled if |
| there corresponding MMIO window exists in the phy_map. In case phy_map |
| for an mmio window is '0' we set the corresponding BAR register to |
| '0'. |
| - hw/phb4: Use local_alloc for phb4 structures |
| |
| Struct phb4 is fairly heavyweight at 283664 bytes. On systems with |
| 6x PHBs per socket this results in using 3.2MB of heap space the PHB |
| structures alone. This is a fairly large chunk of our 12MB heap and |
| on systems with particularly large PCIe topologies, or additional |
| PHBs we can fail to boot because we cannot allocate space for the |
| FDT blob. |
| |
| This patch switches to using local_alloc() for the PHB structures |
| so they don't consume too large a portion of our 12MB heap space. |
| - phb4: Fix typo in disable lane eq code |
| |
| In this commit :: |
| |
| commit 737c0ba3d72b8aab05a765a9fc111a48faac0f75 |
| Author: Michael Neuling <mikey@neuling.org> |
| Date: Thu Feb 22 10:52:18 2018 +1100 |
| phb4: Disable lane eq when retrying some nvidia GEN3 devices |
| |
| We made a typo and set PH2 twice. This fixes it. |
| |
| It worked previously as if only phase 2 (PH2) is set it, skips phase 2 |
| and phase 3 (PH3). |
| - phb4: Don't probe a PHB if its garded |
| |
| Presently phb4_probe_stack() causes an exception while trying to probe |
| a PHB if its garded. This causes skiboot to go into a reboot loop with |
| following exception log: :: |
| |
| *********************************************** |
| Fatal MCE at 000000003006ecd4 .probe_phb4+0x570 |
| CFAR : 00000000300b98a0 |
| <snip> |
| Aborting! |
| CPU 0018 Backtrace: |
| S: 0000000031cc37e0 R: 000000003001a51c ._abort+0x4c |
| S: 0000000031cc3860 R: 0000000030028170 .exception_entry+0x180 |
| S: 0000000031cc3a40 R: 0000000000001f10 * |
| S: 0000000031cc3c20 R: 000000003006ecb0 .probe_phb4+0x54c |
| S: 0000000031cc3e30 R: 0000000030014ca4 .main_cpu_entry+0x5b0 |
| S: 0000000031cc3f00 R: 0000000030002700 boot_entry+0x1b8 |
| |
| This is caused as phb4_probe_stack() will ignore all xscom read/write |
| errors to enable PHB Bars and then tries to perform an mmio to read |
| PHB Version registers that cause the fatal MCE. |
| |
| We fix this by ignoring the PHB probe if the first xscom_write() to |
| populate the PHB Bar register fails, which indicates that there is |
| something wrong with the PHB. |
| - phb4: Workaround PHB errata with CFG write UR/CA errors |
| |
| If the PHB encounters a UR or CA status on a CFG write, it will |
| incorrectly freeze the wrong PE. Instead of using the PE# specified |
| in the CONFIG_ADDRESS register, it will use the PE# of whatever |
| MMIO occurred last. |
| |
| Work around this disabling freeze on such errors |
| - phb4: Handle allocation errors in phb4_eeh_dump_regs() |
| |
| If the zalloc fails (and it can be a rather large allocation), |
| we will overwite memory at 0 instead of failing. |
| - phb4: Don't try to access non-existent PEST entries |
| |
| In a POWER9 chip, some PHB4s have 256 PEs, some have 512. |
| |
| Currently, the diagnostics code retrieves 512 unconditionally, |
| which is wrong and causes us to incorrectly report bogus values |
| for the "high" PEs on the small PHBs. |
| |
| Use the actual number of implemented PEs instead |
| |
| CAPI2 |
| ^^^^^ |
| |
| - phb4/capp: Use link width to allocate STQ engines to CAPP |
| |
| Update phb4_init_capp_regs() to allocates STQ Engines to CAPP/PEC2 |
| based on link width instead of always assuming it to x8. |
| |
| Also re-factor the function slightly to evaluate the link-width only |
| once and cache it so that it can also be used to allocate DMA read |
| engines. |
| - phb4/capp: Update DMA read engines set in APC_FSM_READ_MASK based on link-width |
| |
| Commit 47c09cdfe7a3("phb4/capp: Calculate STQ/DMA read engines based |
| on link-width for PEC") update the CAPP init sequence by calculating |
| the needed STQ/DMA-read engines based on link width and populating it |
| in XPEC_NEST_CAPP_CNTL register. This however needs to be synchronized |
| with the value set in CAPP APC FSM Read Machine Mask Register. |
| |
| Hence this patch update phb4_init_capp_regs() to calculate the link |
| width of the stack on PEC2 and populate the same values as previously |
| populated in PEC CAPP_CNTL register. |
| - capp: Fix the capp recovery timeout comparison |
| |
| The current capp recovery timeout control loop in |
| do_capp_recovery_scoms() uses a wrong comparison for return value of |
| tb_compare(). This may cause do_capp_recovery_scoms() to report an |
| timeout earlier than the 168ms stipulated time. |
| |
| The patch fixes this by updating the loop timeout control branch in |
| do_capp_recovery_scoms() to use the correct enum tb_cmpval. |
| - phb4: Disable 32-bit MSI in capi mode |
| |
| If a capi device does a DMA write targeting an address lower than 4GB, |
| it does so through a 32-bit operation, per the PCI spec. In capi mode, |
| the first TVE entry is configured in bypass mode, so the address is |
| valid. But with any (bad) luck, the address could be 0xFFFFxxxx, thus |
| looking like a 32-bit MSI. |
| |
| We currently enable both 32-bit and 64-bit MSIs, so the PHB will |
| interpret the DMA write as a MSI, which very likely results in an EEH |
| (MSI with a bad payload size). |
| |
| We can fix it by disabling 32-bit MSI when switching the PHB to capi |
| mode. Capi devices are 64-bit. |
| |
| NVLINK2 |
| ^^^^^^^ |
| - npu2: Add support for relaxed-ordering mode |
| |
| Some device drivers support out of order access to GPU memory. This does |
| not affect the CPU view of memory but it does affect the GPU view of |
| memory. It should only be enabled if the GPU driver has requested it. |
| |
| Add OPAL APIs allowing the driver to query relaxed ordering state or |
| request it to be set for a device. Current hardware only allows relaxed |
| ordering to be enabled per PCIe root port. So the code here doesn't |
| enable relaxed ordering until it has been explicitly requested for every |
| device on the port. |
| - Add the other 7 ATSD registers to the device tree. |
| - npu2/hw-procedures: Don't open code NPU2_NTL_MISC_CFG2_BRICK_ENABLE |
| |
| Name this bit properly. There's a lot more cleanup like this to be done, |
| but I'm catching this one now as part of some related changes. |
| - npu2/hw-procedures: Enable parity and credit overflow checks |
| |
| Enable these error checking features by setting the appropriate bits in |
| our one-off initialization of each "NTL Misc Config 2" register. |
| |
| The exception is NDL RX parity checking, which should be disabled during |
| the link training procedures. |
| - npu2: Use correct kill type for TCE invalidation |
| |
| kill_type is enum of OPAL_PCI_TCE_KILL_PAGES, OPAL_PCI_TCE_KILL_PE, |
| OPAL_PCI_TCE_KILL_ALL and phb4_tce_kill() gets it right but |
| npu2_tce_kill() uses OPAL_PCI_TCE_KILL which is an OPAL API token. |
| |
| This fixes an obvious mistype. |
| |
| OpenCAPI |
| ^^^^^^^^ |
| |
| - Support OpenCAPI on Witherspoon platform |
| - npu2-opencapi: Enable presence detection on ZZ |
| |
| Presence detection for opencapi adapters was broken for ZZ planars v3 |
| and below. All ZZ systems currently used in the lab have had their |
| planar upgraded, so we can now remove the override we had to force |
| presence and activate presence detection. Which should improve boot |
| time. |
| |
| Considering the state of opal support on ZZ, this is really only for |
| lab usage on BML. The opencapi enablement team has okay'd the |
| change. In the unlikely case somebody tries opencapi on an old ZZ, the |
| presence detection through i2c will show that no adapter is present |
| and skiboot won't try to access or train the link. |
| - npu2-opencapi: Don't send commands to NPU when link is down |
| |
| Even if an opencapi link is down, we currently always try to issue a |
| config read operation when probing for PCI devices, because of the |
| default scan map used for an opencapi PHB. The config operation fails, |
| as expected, but it can also raise a FIR bit and trigger an HMI. |
| |
| For opencapi, there's no root device like for a "normal" PCI PHB, so |
| there's no reason to do the config operation. To fix it, we keep the |
| scan map blank by default, and only add a device once the link is |
| trained. |
| - opal/hmi: Catch NPU2 HMIs for opencapi |
| |
| HMIs for NPU2 are filtered with the 'compatible' string of the PHB, so |
| add opencapi to the mix. |
| - occ: Wait if OCC GPU presence status not immediately available |
| |
| It takes a few seconds for the OCC to set everything up in order to read |
| GPU presence. At present, we try to kick off OCC initialisation as early as |
| possible to maximise the time it has to read GPU presence. |
| |
| Unfortunately sometimes that's not enough, so add a loop in |
| occ_get_gpu_presence() so that on the first time we try to get GPU presence |
| we keep trying for up to 2 seconds. Experimentally this seems to be |
| adequate. |
| - hw/npu2-hw-procedures: Enable RX auto recal on OpenCAPI links |
| |
| The RX_RC_ENABLE_AUTO_RECAL flag is required on OpenCAPI but not NVLink. |
| |
| Traditionally, Hostboot sets this value according to the machine type. |
| However, now that Witherspoon supports both NVLink and OpenCAPI, it can't |
| tell whether or not a link is OpenCAPI. |
| |
| So instead, set it in skiboot, where it will only be triggered after we've |
| done device detection and found an OpenCAPI device. |
| - hw/npu2-opencapi: Fix setting of supported OpenCAPI templates |
| |
| In opal_npu_tl_set(), we made a typo that means the OPAL_NPU_TL_SET call |
| may not clear the enable bits for templates that were previously enabled |
| but are now disabled. |
| |
| Fix the typo so we clear NPU2_OTL_CONFIG1_TX_TEMP2_EN as well as |
| TEMP{1,3}_EN. |
| |
| Barreleye G2 and Zaius platforms |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| - zaius: Add a slot table |
| - zaius: Add slots for the Barreleye G2 HDD rack |
| |
| The Barreleye G2 is distinct from the Zaius in that it features a 24 |
| Bay NVMe/SATA HDD rack. To provide meaningful slot names for each NVMe |
| device we need to define a slot table for the NVMe capable HDD bays. |
| |
| Unfortunately this isn't straightforward because the PCIe path to the |
| NVMe devices isn't fixed. The PCIe topology is something like: |
| P9 -> HBA card -> 9797 switch -> 20x NVMe HDD slots |
| |
| The 9797 switch is partitioned into two (or four) virtual switches which |
| allow multiple HBA cards to be used (e.g. one per socket). As a result |
| the exact BDFN of the ports will vary depending on how the system is |
| configured. |
| |
| That said, the virtual switch configuration of the 9797 does not change |
| the device and function numbers of the switch downports. This means that |
| we can define a single slot table that maps switch ports to the NVMe bay |
| names. |
| |
| Unfortunately we still need to guess which bus to use this table on, so |
| we assume that any switch downport we find with the PEX9797 VDID is part |
| of the 9797 that supports the HDD rack. |
| |
| FSP based platforms (firenze and ZZ) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| - phb4/capp: Update the expected Eye-catcher for CAPP ucode lid |
| |
| Currently on a FSP based P9 system load_capp_code() expects CAPP ucode |
| lid header to have eye-catcher magic of 'CAPPPSLL'. However skiboot |
| currently supports CAPP ucode only lids that have a eye-catcher magic |
| of 'CAPPLIDH'. This prevents skiboot from loading the ucode with this |
| error message: :: |
| |
| CAPP: ucode header invalid |
| |
| We fix this issue by updating load_capp_ucode() to use the eye-catcher |
| value of 'CAPPLIDH' instead of 'CAPPPSLL'. |
| |
| - FSP: Improve Reset/Reload log message |
| |
| Below message is confusing. Lets make it clear. |
| |
| FSP sends "R/R complete notification" whenever there is a dump. We use `flag` |
| to identify whether its its R/R completion -OR- just new dump notification. :: |
| |
| [ 483.406351956,6] FSP: SP says Reset/Reload complete |
| [ 483.406354278,5] DUMP: FipS dump available. ID = 0x1a00001f [size: 6367640 bytes] |
| [ 483.406355968,7] A Reset/Reload was NOT done |
| |
| Witherspoon platform |
| ^^^^^^^^^^^^^^^^^^^^ |
| |
| - platforms/astbmc/witherspoon: Implement OpenCAPI support |
| |
| OpenCAPI on Witherspoon is slightly more involved than on Zaius and ZZ, due |
| to the OpenCAPI links using the SXM2 connectors that are used for NVLink |
| GPUs. |
| |
| This patch adds the regular OpenCAPI platform information, and also a |
| Witherspoon-specific presence detection callback that uses the previously |
| added OCC GPU presence detection to figure out the device types plugged |
| into each SXM2 socket. |
| |
| The SXM2 connectors are capable of carrying 2 OpenCAPI links, and future |
| OpenCAPI devices are expected to make use of this. However, we don't yet |
| support ganged links and the various implications that has for handling |
| things like device reset, so for now, we only enable 1 brick per device. |