| Rocker Network Switch Register Programming Guide |
| Copyright (c) Scott Feldman <sfeldma@gmail.com> |
| Copyright (c) Neil Horman <nhorman@tuxdriver.com> |
| Version 0.11, 12/29/2014 |
| |
| LICENSE |
| ======= |
| |
| This program is free software; you can redistribute it and/or modify |
| it under the terms of the GNU General Public License as published by |
| the Free Software Foundation; either version 2 of the License, or |
| (at your option) any later version. |
| |
| This program is distributed in the hope that it will be useful, |
| but WITHOUT ANY WARRANTY; without even the implied warranty of |
| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
| GNU General Public License for more details. |
| |
| SECTION 1: Introduction |
| ======================= |
| |
| Overview |
| -------- |
| |
| This document describes the hardware/software interface for the Rocker switch |
| device. The intended audience is authors of OS drivers and device emulation |
| software. |
| |
| Notations and Conventions |
| ------------------------- |
| |
| o In register descriptions, [n:m] indicates a range from bit n to bit m, |
| inclusive. |
| o Use of leading 0x indicates a hexadecimal number. |
| o Use of leading 0b indicates a binary number. |
| o The use of RSVD or Reserved indicates that a bit or field is reserved for |
| future use. |
| o Field width is in bytes, unless otherwise noted. |
| o Register are (R) read-only, (R/W) read/write, (W) write-only, or (COR) clear |
| on read |
| o TLV values in network-byte-order are designated with (N). |
| |
| |
| SECTION 2: PCI Configuration Registers |
| ====================================== |
| |
| PCI Configuration Space |
| ----------------------- |
| |
| Each switch instance registers as a PCI device with PCI configuration space: |
| |
| offset width description value |
| --------------------------------------------- |
| 0x0 2 Vendor ID 0x1b36 |
| 0x2 2 Device ID 0x0006 |
| 0x4 4 Command/Status |
| 0x8 1 Revision ID 0x01 |
| 0x9 3 Class code 0x2800 |
| 0xC 1 Cache line size |
| 0xD 1 Latency timer |
| 0xE 1 Header type |
| 0xF 1 Built-in self test |
| 0x10 4 Base address low |
| 0x14 4 Base address high |
| 0x18-28 Reserved |
| 0x2C 2 Subsystem vendor ID * |
| 0x2E 2 Subsystem ID * |
| 0x30-38 Reserved |
| 0x3C 1 Interrupt line |
| 0x3D 1 Interrupt pin 0x00 |
| 0x3E 1 Min grant 0x00 |
| 0x3D 1 Max latency 0x00 |
| 0x40 1 TRDY timeout |
| 0x41 1 Retry count |
| 0x42 2 Reserved |
| |
| |
| * Assigned by sub-system implementation |
| |
| SECTION 3: Memory-Mapped Register Space |
| ======================================= |
| |
| There are two memory-mapped BARs. BAR0 maps device register space and is |
| 0x2000 in size. BAR1 maps MSI-X vector and PBA tables and is also 0x2000 in |
| size, allowing for 256 MSI-X vectors. |
| |
| All registers are 4 or 8 bytes long. It is assumed host software will access 4 |
| byte registers with one 4-byte access, and 8 byte registers with either two |
| 4-byte accesses or a single 8-byte access. In the case of two 4-byte accesses, |
| access must be lower and then upper 4-bytes, in that order. |
| |
| BAR0 device register space is organized as follows: |
| |
| offset description |
| ------------------------------------------------------ |
| 0x0000-0x000f Bogus registers to catch misbehaving |
| drivers. Writes do nothing. Reads |
| back as 0xDEADBABE. |
| 0x0010-0x00ff Test registers |
| 0x0300-0x03ff General purpose registers |
| 0x1000-0x1fff Descriptor control |
| |
| Holes in register space are reserved. Writes to reserved registers do nothing. |
| Reads to reserved registers read back as 0. |
| |
| No fancy stuff like write-combining is enabled on any of the registers. |
| |
| BAR1 MSI-X register space is organized as follows: |
| |
| offset description |
| ------------------------------------------------------ |
| 0x0000-0x0fff MSI-X vector table (256 vectors total) |
| 0x1000-0x1fff MSI-X PBA table |
| |
| |
| SECTION 4: Interrupts, DMA, and Endianness |
| ========================================== |
| |
| PCI Interrupts |
| -------------- |
| |
| The device supports only MSI-X interrupts. BAR1 memory-mapped region contains |
| the MSI-X vector and PBA tables, with support for up to 256 MSI-X vectors. |
| |
| The vector assignment is: |
| |
| vector description |
| ----------------------------------------------------- |
| 0 Command descriptor ring completion |
| 1 Event descriptor ring completion |
| 2 Test operation completion |
| 3 RSVD |
| 4-255 Tx and Rx descriptor ring completion |
| Tx vector is even |
| Rx vector is odd |
| |
| A MSI-X vector table entry is 16 bytes: |
| |
| field offset width description |
| ------------------------------------------------------------- |
| lower_addr 0x0 4 [31:2] message address[31:2] |
| [1:0] Rsvd (4 byte alignment |
| required) |
| upper_addr 0x4 4 [31:19] Rsvd |
| [14:0] message address[46:32] |
| data 0x8 4 message data[31:0] |
| control 0xc 4 [31:1] Rsvd |
| [0] mask (0 = enable, |
| 1 = masked) |
| |
| Software should install the Interrupt Service Routine (ISR) before any ports |
| are enabled or any commands are issued on the command ring. |
| |
| DMA Operations |
| -------------- |
| |
| DMA operations are used for packet DMA to/from the CPU, command and event |
| processing. Command processing includes statistical counters and table dumps, |
| table insertion/deletion, and more. Event processing provides an async |
| notification method for device-originating events. Each DMA operation has a |
| set of control registers to manage a descriptor ring. The descriptor rings are |
| allocated from contiguous host DMA-able memory and registers specify the rings |
| base address, size and current head and tail indices. Software always writes |
| the head, and hardware always writes the tail. |
| |
| The higher-order bit of DMA_DESC_COMP_ERR is used to mark hardware completion |
| of a descriptor. Software will clear this bit when posting a descriptor to the |
| ring, and hardware will set this bit when the descriptor is complete. |
| |
| Descriptor ring sizes must be a power of 2 and range from 2 to 64K entries. |
| Descriptor rings' base address must be 8-byte aligned. Descriptors must be |
| packed within ring. Each descriptor in each ring must also be aligned on an 8 |
| byte boundary. Each descriptor ring will have these registers: |
| |
| DMA_DESC_xxx_BASE_ADDR, offset 0x1000 + (x * 32), 64-bit, (R/W) |
| DMA_DESC_xxx_SIZE, offset 0x1008 + (x * 32), 32-bit, (R/W) |
| DMA_DESC_xxx_HEAD, offset 0x100c + (x * 32), 32-bit, (R/W) |
| DMA_DESC_xxx_TAIL, offset 0x1010 + (x * 32), 32-bit, (R) |
| DMA_DESC_xxx_CTRL, offset 0x1014 + (x * 32), 32-bit, (W) |
| DMA_DESC_xxx_CREDITS, offset 0x1018 + (x * 32), 32-bit, (R/W) |
| DMA_DESC_xxx_RSVD1, offset 0x101c + (x * 32), 32-bit, (R/W) |
| |
| Where x is descriptor ring index: |
| |
| index ring |
| -------------------- |
| 0 CMD |
| 1 EVENT |
| 2 TX (port 0) |
| 3 RX (port 0) |
| 4 TX (port 1) |
| 5 RX (port 1) |
| . |
| . |
| . |
| 124 TX (port 61) |
| 125 RX (port 61) |
| 126 Resv |
| 127 Resv |
| |
| Writing BASE_ADDR or SIZE will reset HEAD and TAIL to zero. HEAD cannot be |
| written past TAIL. To do so would wrap the ring. An empty ring is when HEAD |
| == TAIL. A full ring is when HEAD is one position behind TAIL. Both HEAD and |
| TAIL increment and modulo wrap at the ring size. |
| |
| CTRL register bits: |
| |
| bit name description |
| ------------------------------------------------------------------------ |
| [0] CTRL_RESET Reset the descriptor ring |
| [1:31] Reserved |
| |
| All descriptor types share some common fields: |
| |
| field width description |
| ------------------------------------------------------------------- |
| DMA_DESC_BUF_ADDR 8 Phys addr of desc payload, 8-byte |
| aligned |
| DMA_DESC_COOKIE 8 Desc cookie for completion matching, |
| upper-most bit is reserved |
| DMA_DESC_BUF_SIZE 2 Desc payload size in bytes |
| DMA_DESC_TLV_SIZE 2 Desc payload total size in bytes |
| used for TLVs. Must be <= |
| DMA_DESC_BUF_SIZE. |
| DMA_DESC_COMP_ERR 2 Completion status of associated |
| desc payload. High order bit is |
| clear on new descs, toggled by |
| hw for completed items. |
| |
| To support forward- and backward-compatibility, descriptor and completion |
| payloads are specified in TLV format. Fields are packed with Type=field name, |
| Length=field length, and Value=field value. Software will ignore unknown fields |
| filled in by the switch. Likewise, the switch will ignore unknown fields |
| filled in by software. |
| |
| Descriptor payload buffer is 8-byte aligned and TLVs are 8-byte aligned. The |
| value within a TLV is also 8-byte aligned. The (packed, 8 byte) TLV header is: |
| |
| field width description |
| ----------------------------- |
| type 4 TLV type |
| len 2 TLV value length |
| pad 2 Reserved |
| |
| The alignment requirements for descriptors and TLVs are to avoid unaligned |
| access exceptions in software. Note that the payload for each TLV is also |
| 8 byte aligned. |
| |
| Figure 1 shows an example descriptor buffer with two TLVs. |
| |
| <------- 8 bytes -------> |
| |
| 8-byte +––––+ +–––––––––––+–––––+–––––+ +–+ |
| align | type | len | pad | TLV#1 hdr | |
| +–––––––––––+–––––+–––––+ (len=22) | |
| | | | |
| | value | TVL#1 value | |
| | | (padded to 8-byte | |
| | +–––––+ alignment) | |
| | |/////| | |
| 8-byte +––––+ +–––––––––––+–––––––––––+ | |
| align | type | len | pad | TLV#2 hdr DESC_BUF_SIZE |
| +–––––+–––––+–––––+–––––+ (len=2) | |
| |value|/////////////////| TLV#2 value | |
| +–––––+/////////////////| | |
| |///////////////////////| | |
| |///////////////////////| | |
| |///////////////////////| | |
| |////////unused/////////| | |
| |////////space//////////| | |
| |///////////////////////| | |
| |///////////////////////| | |
| |///////////////////////| | |
| +–––––––––––––––––––––––+ +–+ |
| |
| fig. 1 |
| |
| TLVs can be nested within the NEST TLV type. |
| |
| Interrupt credits |
| ^^^^^^^^^^^^^^^^^ |
| |
| MSI-X vectors used for descriptor ring completions use a credit mechanism for |
| efficient device, PCIe bus, OS and driver operations. Each descriptor ring has |
| a credit count which represents the number of outstanding descriptors to be |
| processed by the driver. As the device marks descriptors complete, the credit |
| count is incremented. As the driver processes those outstanding descriptors, |
| it returns credits back to the device. This way, the device knows the driver's |
| progress and can make decisions about when to fire the next interrupt or not. |
| When the credit count is zero, and the first descriptors are posted for the |
| driver, a single interrupt is fired. Once the interrupt is fired, the |
| interrupt is disabled (auto-masked*). In response to the interrupt, the driver |
| will process descriptors and PIO write a returned credit value for that |
| descriptor ring. If the driver returns all credits (the driver caught up with |
| the device and there is no outstanding work), then the interrupt is unmasked, |
| but not fired. If only partial credits are returned, the interrupt remains |
| masked but the device generates an interrupt, signaling the driver that more |
| outstanding work is available. |
| |
| (* this masking is unrelated to the MSI-X interrupt mask register) |
| |
| Endianness |
| ---------- |
| |
| Device registers are hard-coded to little-endian (LE). The driver should |
| convert to/from host endianess to LE for device register accesses. |
| |
| Descriptors are LE. Descriptor buffer TLVs will have LE type and length |
| fields, but the value field can either be LE or network-byte-order, depending |
| on context. TLV values containing network packet data will be in network-byte |
| order. A TLV value containing a field or mask used to compare against network |
| packet data is network-byte order. For example, flow match fields (and masks) |
| are network-byte-order since they're matched directly, byte-by-byte, against |
| network packet data. All non-network-packet TLV multi-byte values will be LE. |
| |
| TLV values in network-byte-order are designated with (N). |
| |
| |
| SECTION 5: Test Registers |
| ========================= |
| |
| Rocker has several test registers to support troubleshooting register access, |
| interrupt generation, and DMA operations: |
| |
| TEST_REG, offset 0x0010, 32-bit (R/W) |
| TEST_REG64, offset 0x0018, 64-bit (R/W) |
| TEST_IRQ, offset 0x0020, 32-bit (R/W) |
| TEST_DMA_ADDR, offset 0x0028, 64-bit (R/W) |
| TEST_DMA_SIZE, offset 0x0030, 32-bit (R/W) |
| TEST_DMA_CTRL, offset 0x0034, 32-bit (R/W) |
| |
| Reads to TEST_REG and TEST_REG64 will read a value equal to twice the last |
| value written to the register. The 32-bit and 64-bit versions are for testing |
| 32-bit and 64-bit host accesses. |
| |
| A vector can be written to TEST_IRQ and the device will generate an interrupt |
| for that vector. |
| |
| To test basic DMA operations, allocate a DMA-able host buffer and put the |
| buffer address into TEST_DMA_ADDR and size into TEST_DMA_SIZE. Then, write to |
| TEST_DMA_CTRL to manipulate the buffer contents. TEST_DMA_CTRL operations are: |
| |
| operation value description |
| ----------------------------------------------------------- |
| TEST_DMA_CTRL_CLEAR 1 clear buffer |
| TEST_DMA_CTRL_FILL 2 fill buffer bytes with 0x96 |
| TEST_DMA_CTRL_INVERT 4 invert bytes in buffer |
| |
| Various buffer address and sizes should be tested to verify no address boundary |
| issue exists. In particular, buffers that start on odd-8-byte boundary and/or |
| span multiple PAGE sizes should be tested. |
| |
| |
| SECTION 6: Ports |
| ================ |
| |
| Physical and Logical Ports |
| ------------------------------------ |
| |
| The switch supports up to 62 physical (front-panel) ports. Register |
| PORT_PHYS_COUNT returns the actual number of physical ports available: |
| |
| PORT_PHYS_COUNT, offset 0x0304, 32-bit, (R) |
| |
| In addition to front-panel ports, the switch supports logical ports for |
| tunnels. |
| |
| Front-panel ports and logical tunnel ports are mapped into a single 32-bit port |
| space. A special CPU port is assigned port 0. The front-panel ports are |
| mapped to ports 1-62. A special loopback port is assigned port 63. Logical |
| tunnel ports are assigned ports 0x0001000-0x0001ffff. |
| To summarize the port assignments: |
| |
| port mapping |
| ------------------------------------------------------- |
| 0 CPU port (for packets to/from host CPU) |
| 1-62 front-panel physical ports |
| 63 loopback port |
| 64-0x0000ffff RSVD |
| 0x00010000-0x0001ffff logical tunnel ports |
| 0x00020000-0xffffffff RSVD |
| |
| Physical Port Mode |
| ------------------ |
| |
| Switch front-panel ports operate in a mode. Currently, the only mode is |
| OF-DPA. OF-DPA[1] mode is based on OpenFlow Data Plane Abstraction (OF-DPA) |
| Abstract Switch Specification, Version 1.0, from Broadcom Corporation. To |
| set/get the mode for front-panel ports, see port settings, below. |
| |
| Port Settings |
| ------------- |
| |
| Link status for all front-panel ports is available via PORT_PHYS_LINK_STATUS: |
| |
| PORT_PHYS_LINK_STATUS, offset 0x0310, 64-bit, (R) |
| |
| Value is port bitmap. Bits 0 and 63 always read 0. Bits 1-62 |
| read 1 for link UP and 0 for link DOWN for respective front-panel ports. |
| |
| Other properties for front-panel ports are available via DMA CMD descriptors: |
| |
| Get PORT_SETTINGS descriptor: |
| |
| field width description |
| ---------------------------------------------- |
| PORT_SETTINGS 2 CMD_GET |
| PPORT 4 Physical port # |
| |
| Get PORT_SETTINGS completion: |
| |
| field width description |
| ---------------------------------------------- |
| PPORT 4 Physical port # |
| SPEED 4 Current port interface speed, in Mbps |
| DUPLEX 1 1 = Full, 0 = Half |
| AUTONEG 1 1 = enabled, 0 = disabled |
| MACADDR 6 Port MAC address |
| MODE 1 0 = OF-DPA |
| LEARNING 1 MAC address learning on port |
| 1 = enabled |
| 0 = disabled |
| PHYS_NAME <var> Physical port name (string) |
| |
| Set PORT_SETTINGS descriptor: |
| |
| field width description |
| ---------------------------------------------- |
| PORT_SETTINGS 2 CMD_SET |
| PPORT 4 Physical port # |
| SPEED 4 Port interface speed, in Mbps |
| DUPLEX 1 1 = Full, 0 = Half |
| AUTONEG 1 1 = enabled, 0 = disabled |
| MACADDR 6 Port MAC address |
| MODE 1 0 = OF-DPA |
| |
| Port Enable |
| ----------- |
| |
| Front-panel ports are initially disabled, which means port ingress and egress |
| packets will be dropped. To enable or disable a port, use PORT_PHYS_ENABLE: |
| |
| PORT_PHYS_ENABLE: offset 0x0318, 64-bit, (R/W) |
| |
| Value is bitmap of first 64 ports. Bits 0 and 63 are ignored |
| and always read as 0. Write 1 to enable port; write 0 to disable it. |
| Default is 0. |
| |
| |
| SECTION 7: Switch Control |
| ========================= |
| |
| This section covers switch-wide register settings. |
| |
| Control |
| ------- |
| |
| This register is used for low level control of the switch. |
| |
| CONTROL: offset 0x0300, 32-bit, (W) |
| |
| bit name description |
| ------------------------------------------------------------------------ |
| [0] CONTROL_RESET If set, device will perform reset |
| [1:31] Reserved |
| |
| Switch ID |
| --------- |
| |
| The switch has a SWITCH_ID to be used by software to uniquely identify the |
| switch: |
| |
| SWITCH_ID: offset 0x0320, 64-bit, (R) |
| |
| Value is opaque to switch software and no special encoding is implied. |
| |
| |
| SECTION 8: Events |
| ================= |
| |
| Non-I/O asynchronous events from the device are notified to the host using the |
| event ring. The TLV structure for events is: |
| |
| field width description |
| --------------------------------------------------- |
| TYPE 4 Event type, one of: |
| 1: LINK_CHANGED |
| 2: MAC_VLAN_SEEN |
| INFO <nest> Event info (details below) |
| |
| Link Changed Event |
| ------------------ |
| |
| When link status changes on a physical port, this event is generated. |
| |
| field width description |
| --------------------------------------------------- |
| INFO <nest> |
| PPORT 4 Physical port |
| LINKUP 1 Link status: |
| 0: down |
| 1: up |
| |
| MAC VLAN Seen Event |
| ------------------- |
| |
| When a packet ingresses on a port and the source MAC/VLAN isn't known to the |
| device, the device will generate this event. In response to the event, the |
| driver should install to the device the MAC/VLAN on the port into the bridge |
| table. Once installed, the MAC/VLAN is known on the port and this event will |
| no longer be generated. |
| |
| field width description |
| --------------------------------------------------- |
| INFO <nest> |
| PPORT 4 Physical port |
| MAC 6 MAC address |
| VLAN 2 VLAN ID |
| |
| |
| SECTION 9: CPU Packet Processing |
| ================================ |
| |
| Ingress packets directed to the host CPU for further processing are delivered |
| in the DMA RX ring. Likewise, host CPU originating packets destined to egress |
| on switch ports are scheduled by software using the DMA TX ring. |
| |
| Tx Packet Processing |
| -------------------- |
| |
| Software schedules packets for egress on switch ports using the DMA TX ring. A |
| TX descriptor buffer describes the packet location and size in host DMA-able |
| memory, the destination port, and any hardware-offload functions (such as L3 |
| payload checksum offload). Software then bumps the descriptor head to signal |
| hardware of new Tx work. In response, hardware will DMA read Tx descriptors up |
| to head, DMA read descriptor buffer and packet data, perform offloading |
| functions, and finally frame packet on wire (network). Once packet processing |
| is complete, hardware will writeback status to descriptor(s) to signal to |
| software that Tx is complete and software resources (e.g. skb) backing packet |
| can be released. |
| |
| Figure 2 shows an example 3-fragment packet queued with one Tx descriptor. A |
| TLV is used for each packet fragment. |
| |
| pkt frag 1 |
| +–––––––+ +–+ |
| +–––+ | | |
| desc buf | | | | |
| +––––––––+ | | | | |
| Tx ring +–––+ +–––––+ | | | |
| +–––––––––+ | | TLVs | +–––––––+ | |
| | +–––+ +––––––––+ pkt frag 2 | |
| | desc 0 | | +–––––+ +–––––––+ | |
| +–––––––––+ | TLVs | +–––+ | | |
| head+–+ | +––––––––+ | | | |
| | desc 1 | | +–––––+ +–––––––+ |pkt |
| +–––––––––+ | TLVs | | | |
| | | +––––––––+ | pkt frag 3 | |
| | | | +–––––––+ | |
| +–––––––––+ +–––+ | | |
| | | | | | |
| | | | | | |
| +–––––––––+ | | | |
| | | | | | |
| | | | | | |
| +–––––––––+ | | | |
| | | +–––––––+ +–+ |
| | | |
| +–––––––––+ |
| |
| fig 2. |
| |
| The TLVs for Tx descriptor buffer are: |
| |
| field width description |
| --------------------------------------------------------------------- |
| PPORT 4 Destination physical port # |
| TX_OFFLOAD 1 Hardware offload modes: |
| 0: no offload |
| 1: insert IP csum (ipv4 only) |
| 2: insert TCP/UDP csum |
| 3: L3 csum calc and insert |
| into csum offset (TX_L3_CSUM_OFF) |
| 16-bit 1's complement csum value. |
| IPv4 pseudo-header and IP |
| already calculated by OS |
| and inserted. |
| 4: TSO (TCP Segmentation Offload) |
| TX_L3_CSUM_OFF 2 For L3 csum offload mode, the offset, |
| from the beginning of the packet, |
| of the csum field in the L3 header |
| TX_TSO_MSS 2 For TSO offload mode, the |
| Maximum Segment Size in bytes |
| TX_TSO_HDR_LEN 2 For TSO offload mode, the |
| length of ethernet, IP, and |
| TCP/UDP headers, including IP |
| and TCP options. |
| TX_FRAGS <array> Packet fragments |
| TX_FRAG <nest> Packet fragment |
| TX_FRAG_ADDR 8 DMA address of packet fragment |
| TX_FRAG_LEN 2 Packet fragment length |
| |
| Possible status return codes in descriptor on completion are: |
| |
| DESC_COMP_ERR reason |
| -------------------------------------------------------------------- |
| 0 OK |
| -ROCKER_ENXIO address or data read err on desc buf or packet |
| fragment |
| -ROCKER_EINVAL bad pport or TSO or csum offloading error |
| -ROCKER_ENOMEM no memory for internal staging tx fragment |
| |
| Rx Packet Processing |
| -------------------- |
| |
| For packets ingressing on switch ports that are not forwarded by the switch but |
| rather directed to the host CPU for further processing are delivered in the DMA |
| RX ring. Rx descriptor buffers are allocated by software and placed on the |
| ring. Hardware will fill Rx descriptor buffers with packet data, write the |
| completion, and signal to software that a new packet is ready. Since Rx packet |
| size is not known a-priori, the Rx descriptor buffer must be allocated for |
| worst-case packet size. A single Rx descriptor will contain the entire Rx |
| packet data in one RX_FRAG. Other Rx TLVs describe and hardware offloads |
| performed on the packet, such as checksum validation. |
| |
| The TLVs for Rx descriptor buffer are: |
| |
| field width description |
| --------------------------------------------------- |
| PPORT 4 Source physical port # |
| RX_FLAGS 2 Packet parsing flags: |
| (1 << 0): IPv4 packet |
| (1 << 1): IPv6 packet |
| (1 << 2): csum calculated |
| (1 << 3): IPv4 csum good |
| (1 << 4): IP fragment |
| (1 << 5): TCP packet |
| (1 << 6): UDP packet |
| (1 << 7): TCP/UDP csum good |
| (1 << 8): Offload forward |
| RX_CSUM 2 IP calculated checksum: |
| IPv4: IP payload csum |
| IPv6: header and payload csum |
| (Only valid is RX_FLAGS:csum calc is set) |
| RX_FRAG_ADDR 8 DMA address of packet fragment |
| RX_FRAG_MAX_LEN 2 Packet maximum fragment length |
| RX_FRAG_LEN 2 Actual packet fragment length after receive |
| |
| Offload forward RX_FLAG indicates the device has already forwarded the packet |
| so the host CPU should not also forward the packet. |
| |
| Possible status return codes in descriptor on completion are: |
| |
| DESC_COMP_ERR reason |
| -------------------------------------------------------------------- |
| 0 OK |
| -ROCKER_ENXIO address or data read err on desc buf |
| -ROCKER_ENOMEM no memory for internal staging desc buf |
| -ROCKER_EMSGSIZE Rx descriptor buffer wasn't big enough to contain |
| packet data TLV and other TLVs. |
| |
| |
| SECTION 10: OF-DPA Mode |
| ====================== |
| |
| OF-DPA mode allows the switch to offload flow packet processing functions to |
| hardware. An OpenFlow controller would communicate with an OpenFlow agent |
| installed on the switch. The OpenFlow agent would (directly or indirectly) |
| communicate with the Rocker switch driver, which in turn would program switch |
| hardware with flow functionality, as defined in OF-DPA. The block diagram is: |
| |
| +–––––––––––––––----–––+ |
| | OF | |
| | Remote Controller | |
| +––––––––+––----–––––––+ |
| | |
| | |
| +––––––––+–––––––––+ |
| | OF | |
| | Local Agent | |
| +––––––––––––––––––+ |
| | | |
| | Rocker Driver | |
| +––––––––––––––––––+ |
| <this spec> |
| +––––––––––––––––––+ |
| | | |
| | Rocker Switch | |
| +––––––––––––––––––+ |
| |
| To participate in flow functions, ports must be configure for OF-DPA mode |
| during switch initialization. |
| |
| OF-DPA Flow Table Interface |
| --------------------------- |
| |
| There are commands to add, modify, delete, and get stats of flow table entries. |
| The commands are issued using the DMA CMD descriptor ring. The following |
| commands are defined: |
| |
| CMD_ADD: add an entry to flow table |
| CMD_MOD: modify an entry in flow table |
| CMD_DEL: delete an entry from flow table |
| CMD_GET_STATS: get stats for flow entry |
| |
| TLVs for add and modify commands are: |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_CMD 2 CMD_[ADD|MOD] |
| OF_DPA_TBL 2 Flow table ID |
| 0: ingress port |
| 10: vlan |
| 20: termination mac |
| 30: unicast routing |
| 40: multicast routing |
| 50: bridging |
| 60: ACL policy |
| OF_DPA_PRIORITY 4 Flow priority |
| OF_DPA_HARDTIME 4 Hard timeout for flow |
| OF_DPA_IDLETIME 4 Idle timeout for flow |
| OF_DPA_COOKIE 8 Cookie |
| |
| Additional TLVs based on flow table ID: |
| |
| Table ID 0: ingress port |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_IN_PPORT 4 ingress physical port number |
| OF_DPA_GOTO_TBL 2 goto table ID; zero to drop |
| |
| Table ID 10: vlan |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_IN_PPORT 4 ingress physical port number |
| OF_DPA_VLAN_ID 2 (N) vlan ID |
| OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask |
| OF_DPA_GOTO_TBL 2 goto table ID; zero to drop |
| OF_DPA_NEW_VLAN_ID 2 (N) new vlan ID |
| |
| Table ID 20: termination mac |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_IN_PPORT 4 ingress physical port number |
| OF_DPA_IN_PPORT_MASK 4 ingress physical port number mask |
| OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd |
| OF_DPA_DST_MAC 6 (N) destination MAC |
| OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask |
| OF_DPA_VLAN_ID 2 (N) vlan ID |
| OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask |
| OF_DPA_GOTO_TBL 2 only acceptable values are |
| unicast or multicast routing |
| table IDs |
| OF_DPA_OUT_PPORT 2 if specified, must be |
| controller, set zero otherwise |
| |
| Table ID 30: unicast routing |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd |
| OF_DPA_DST_IP 4 (N) destination IPv4 address. |
| Must be unicast address |
| OF_DPA_DST_IP_MASK 4 (N) IP mask. Must be prefix mask |
| OF_DPA_DST_IPV6 16 (N) destination IPv6 address. |
| Must be unicast address |
| OF_DPA_DST_IPV6_MASK 16 (N) IPv6 mask. Must be prefix mask |
| OF_DPA_GOTO_TBL 2 goto table ID; zero to drop |
| OF_DPA_GROUP_ID 4 data for GROUP action must |
| be an L3 Unicast group entry |
| |
| Table ID 40: multicast routing |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_ETHERTYPE 2 (N) must be either 0x0800 or 0x86dd |
| OF_DPA_VLAN_ID 2 (N) vlan ID |
| OF_DPA_SRC_IP 4 (N) source IPv4. Optional, |
| can contain IPv4 address, |
| must be completely masked |
| if not used |
| OF_DPA_SRC_IP_MASK 4 (N) IP Mask |
| OF_DPA_DST_IP 4 (N) destination IPv4 address. |
| Must be multicast address |
| OF_DPA_SRC_IPV6 16 (N) source IPv6 Address. Optional. |
| Can contain IPv6 address, |
| must be completely masked |
| if not used |
| OF_DPA_SRC_IPV6_MASK 16 (N) IPv6 mask. |
| OF_DPA_DST_IPV6 16 (N) destination IPv6 Address. Must |
| be multicast address |
| Must be multicast address |
| OF_DPA_GOTO_TBL 2 goto table ID; zero to drop |
| OF_DPA_GROUP_ID 4 data for GROUP action must |
| be an L3 multicast group entry |
| |
| Table ID 50: bridging |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_VLAN_ID 2 (N) vlan ID |
| OF_DPA_TUNNEL_ID 4 tunnel ID |
| OF_DPA_DST_MAC 6 (N) destination MAC |
| OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask |
| OF_DPA_GOTO_TBL 2 goto table ID; zero to drop |
| OF_DPA_GROUP_ID 4 data for GROUP action must |
| be a L2 Interface, L2 |
| Multicast, L2 Flood, |
| or L2 Overlay group entry |
| as appropriate |
| OF_DPA_TUNNEL_LPORT 4 unicast Tenant Bridging |
| flows specify a tunnel |
| logical port ID |
| OF_DPA_OUT_PPORT 2 data for OUTPUT action, |
| restricted to CONTROLLER, |
| set to 0 otherwise |
| |
| Table ID 60: acl policy |
| |
| field width description |
| ---------------------------------------------------- |
| OF_DPA_IN_PPORT 4 ingress physical port number |
| OF_DPA_IN_PPORT_MASK 4 ingress physical port number mask |
| OF_DPA_ETHERTYPE 2 (N) ethertype |
| OF_DPA_VLAN_ID 2 (N) vlan ID |
| OF_DPA_VLAN_ID_MASK 2 (N) vlan ID mask |
| OF_DPA_VLAN_PCP 2 (N) vlan Priority Code Point |
| OF_DPA_VLAN_PCP_MASK 2 (N) vlan Priority Code Point mask |
| OF_DPA_SRC_MAC 6 (N) source MAC |
| OF_DPA_SRC_MAC_MASK 6 (N) source MAC mask |
| OF_DPA_DST_MAC 6 (N) destination MAC |
| OF_DPA_DST_MAC_MASK 6 (N) destination MAC mask |
| OF_DPA_TUNNEL_ID 4 tunnel ID |
| OF_DPA_SRC_IP 4 (N) source IPv4. Optional, |
| can contain IPv4 address, |
| must be completely masked |
| if not used |
| OF_DPA_SRC_IP_MASK 4 (N) IP Mask |
| OF_DPA_DST_IP 4 (N) destination IPv4 address. |
| Must be multicast address |
| OF_DPA_DST_IP_MASK 4 (N) IP Mask |
| OF_DPA_SRC_IPV6 16 (N) source IPv6 Address. Optional. |
| Can contain IPv6 address, |
| must be completely masked |
| if not used |
| OF_DPA_SRC_IPV6_MASK 16 (N) IPv6 mask |
| OF_DPA_DST_IPV6 16 (N) destination IPv6 Address. Must |
| be multicast address. |
| OF_DPA_DST_IPV6_MASK 16 (N) IPv6 mask |
| OF_DPA_SRC_ARP_IP 4 (N) source IPv4 address in the ARP |
| payload. Only used if ethertype |
| == 0x0806. |
| OF_DPA_SRC_ARP_IP_MASK 4 (N) IP Mask |
| OF_DPA_IP_PROTO 1 IP protocol |
| OF_DPA_IP_PROTO_MASK 1 IP protocol mask |
| OF_DPA_IP_DSCP 1 DSCP |
| OF_DPA_IP_DSCP_MASK 1 DSCP mask |
| OF_DPA_IP_ECN 1 ECN |
| OF_DPA_IP_ECN_MASK 1 ECN mask |
| OF_DPA_L4_SRC_PORT 2 (N) L4 source port, only for |
| TCP, UDP, or SCTP |
| OF_DPA_L4_SRC_PORT_MASK 2 (N) L4 source port mask |
| OF_DPA_L4_DST_PORT 2 (N) L4 source port, only for |
| TCP, UDP, or SCTP |
| OF_DPA_L4_DST_PORT_MASK 2 (N) L4 source port mask |
| OF_DPA_ICMP_TYPE 1 ICMP type, only if IP |
| protocol is 1 |
| OF_DPA_ICMP_TYPE_MASK 1 ICMP type mask |
| OF_DPA_ICMP_CODE 1 ICMP code |
| OF_DPA_ICMP_CODE_MASK 1 ICMP code mask |
| OF_DPA_IPV6_LABEL 4 (N) IPv6 flow label |
| OF_DPA_IPV6_LABEL_MASK 4 (N) IPv6 flow label mask |
| OF_DPA_GROUP_ID 4 data for GROUP action |
| OF_DPA_QUEUE_ID_ACTION 1 write the queue ID |
| OF_DPA_NEW_QUEUE_ID 1 queue ID |
| OF_DPA_VLAN_PCP_ACTION 1 write the VLAN priority |
| OF_DPA_NEW_VLAN_PCP 1 VLAN priority |
| OF_DPA_IP_DSCP_ACTION 1 write the DSCP |
| OF_DPA_NEW_IP_DSCP 1 new DSCP |
| OF_DPA_TUNNEL_LPORT 4 restrct to valid tunnel |
| logical port, set to 0 |
| otherwise. |
| OF_DPA_OUT_PPORT 2 data for OUTPUT action, |
| restricted to CONTROLLER, |
| set to 0 otherwise |
| OF_DPA_CLEAR_ACTIONS 4 if 1 packets matching flow are |
| dropped (all other instructions |
| ignored) |
| |
| TLVs for flow delete and get stats command are: |
| |
| field width description |
| --------------------------------------------------- |
| OF_DPA_CMD 2 CMD_[DEL|GET_STATS] |
| OF_DPA_COOKIE 8 Cookie |
| |
| On completion of get stats command, the descriptor buffer is written back with |
| the following TLVs: |
| |
| field width description |
| --------------------------------------------------- |
| OF_DPA_STAT_DURATION 4 Flow duration |
| OF_DPA_STAT_RX_PKTS 8 Received packets |
| OF_DPA_STAT_TX_PKTS 8 Transmit packets |
| |
| Possible status return codes in descriptor on completion are: |
| |
| DESC_COMP_ERR command reason |
| -------------------------------------------------------------------- |
| 0 all OK |
| -ROCKER_EFAULT all head or tail index outside |
| of ring |
| -ROCKER_ENXIO all address or data read err on |
| desc buf |
| -ROCKER_EMSGSIZE GET_STATS cmd descriptor buffer wasn't |
| big enough to contain write-back |
| TLVs |
| -ROCKER_EINVAL all invalid parameters passed in |
| -ROCKER_EEXIST ADD entry already exists |
| -ROCKER_ENOSPC ADD no space left in flow table |
| -ROCKER_ENOENT MOD|DEL|GET_STATS cookie invalid |
| |
| Group Table Interface |
| --------------------- |
| |
| There are commands to add, modify, delete, and get stats of group table |
| entries. The commands are issued using the DMA CMD descriptor ring. The |
| following commands are defined: |
| |
| CMD_ADD: add an entry to group table |
| CMD_MOD: modify an entry in group table |
| CMD_DEL: delete an entry from group table |
| CMD_GET_STATS: get stats for group entry |
| |
| TLVs for add and modify commands are: |
| |
| field width description |
| ----------------------------------------------------------- |
| FLOW_GROUP_CMD 2 CMD_[ADD|MOD] |
| FLOW_GROUP_ID 2 Flow group ID |
| FLOW_GROUP_TYPE 1 Group type: |
| 0: L2 interface |
| 1: L2 rewrite |
| 2: L3 unicast |
| 3: L2 multicast |
| 4: L2 flood |
| 5: L3 interface |
| 6: L3 multicast |
| 7: L3 ECMP |
| 8: L2 overlay |
| FLOW_VLAN_ID 2 Vlan ID (types 0, 3, 4, 6) |
| FLOW_L2_PORT 2 Port (types 0) |
| FLOW_INDEX 4 Index (all types but 0) |
| FLOW_OVERLAY_TYPE 1 Overlay sub-type (type 8): |
| 0: Flood unicast tunnel |
| 1: Flood multicast tunnel |
| 2: Multicast unicast tunnel |
| 3: Multicast multicast tunnel |
| FLOW_GROUP_ACTION nest |
| FLOW_GROUP_ID 2 next group ID in chain (all |
| types except 0) |
| FLOW_OUT_PORT 4 egress port (types 0, 8) |
| FLOW_POP_VLAN_TAG 1 strip outer VLAN tag (type 1 |
| only) |
| FLOW_VLAN_ID 2 (types 1, 5) |
| FLOW_SRC_MAC 6 (types 1, 2, 5) |
| FLOW_DST_MAC 6 (types 1, 2) |
| |
| TLVs for flow delete and get stats command are: |
| |
| field width description |
| ----------------------------------------------------------- |
| FLOW_GROUP_CMD 2 CMD_[DEL|GET_STATS] |
| FLOW_GROUP_ID 2 Flow group ID |
| |
| On completion of get stats command, the descriptor buffer is written back with |
| the following TLVs: |
| |
| field width description |
| --------------------------------------------------- |
| FLOW_GROUP_ID 2 Flow group ID |
| FLOW_STAT_DURATION 4 Flow duration |
| FLOW_STAT_REF_COUNT 4 Flow reference count |
| FLOW_STAT_BUCKET_COUNT 4 Flow bucket count |
| |
| Possible status return codes in descriptor on completion are: |
| |
| DESC_COMP_ERR command reason |
| -------------------------------------------------------------------- |
| 0 all OK |
| -ROCKER_EFAULT all head or tail index outside |
| of ring |
| -ROCKER_ENXIO all address or data read err on |
| desc buf |
| -ROCKER_ENOSPC GET_STATS cmd descriptor buffer wasn't |
| big enough to contain write-back |
| TLVs |
| -ROCKER_EINVAL ADD|MOD invalid parameters passed in |
| -ROCKER_EEXIST ADD entry already exists |
| -ROCKER_ENOSPC ADD no space left in flow table |
| -ROCKER_ENOENT MOD|DEL|GET_STATS group ID invalid |
| -ROCKER_EBUSY DEL group reference count non-zero |
| -ROCKER_ENODEV ADD next group ID doesn't exist |
| |
| |
| |
| References |
| ========== |
| |
| [1] OpenFlow Data Plane Abstraction (OF-DPA) Abstract Switch Specification, |
| Version 1.0, from Broadcom Corporation, February 21, 2014. |
