Data Link Layer in PCI Express

PCIe Architectural Deep Dive: Data Link Layer & PIPE Interface

Table of Contents

• Part 1: Architectural Foundations and Packet Framing
• Part 2: The ACK/NAK Protocol and Replay Mechanism
• Part 3: Credit-Based Flow Control and the DLL State Machine
• Part 4: The PIPE Interface Standard and Cross-Layer Interfacing

Part 1: Architectural Foundations and Packet Framing

In a PCI Express (PCIe) system, the Data Link Layer (DLL) serves as the critical intermediary between the high-level transaction logic and the physical wires. Its primary objective is simple yet absolute: guarantee reliable, in-order, and error-free point-to-point packet delivery over a single physical link.

Core Responsibilities of the Data Link Layer

The DLL handles four foundational operations to insulate upper-layer software from hardware-level noise and buffer overflows:

• Data Integrity & Framing: Prepending sequence markers and computing checksum signatures for transit.
• Error Detection & Retransmission: Operating the hardware-level ACK/NAK confirmation engine.
• Flow Control Management: Governing a credit-based gatekeeping architecture to prevent buffer overruns.
• Link Power Management: Tracking active states and orchestrating low-power scaling targets.

The Transaction Layer Boundary: The Transaction Layer focuses entirely on what the data means (reads, writes, completions). It relies on the Data Link Layer to guarantee that these transmissions actually arrive intact without requiring system software interventions.

How the DLL Modifies Packets (TLP Formatting)

When a Transaction Layer Packet (TLP) is pushed down, the DLL appends specific tracking headers and footers to ensure verification capability at the receiving node:

+-------------------+-----------------------------------+-------------------+ | Sequence Number | Original TLP Data Payload | 32-bit LCRC | | (12 Bits) | (Header + Optional Data) | (Check Signature) | +-------------------+-----------------------------------+-------------------+ ^-- Added by DLL ^-- Added by DLL

• Sequence Number Prepending: A 12-bit sequential index is attached to the front of the TLP. This enables the receiver to identify if an individual packet has dropped completely out of line.
• LCRC Appending: The DLL computes a 32-bit Link Cyclic Redundancy Check (LCRC) over the packet data and the sequence number. The receiving device recalculates this value; if a single bit flipped during transit, the anomaly is flagged immediately.

Part 2: The ACK/NAK Protocol and Replay Mechanism

If high-speed electrical noise degrades a packet, PCIe does not crash. The Data Link Layer resolves the data corruption dynamically using a hardware-managed tracking sequence called the ACK/NAK Protocol.

The Core Elements of Retransmission

This automated verification feedback loop utilizes two primary physical resources:

• The Replay Buffer: Located on the transmitter side, this dedicated hardware buffer stores an exact copy of every single transmitted TLP. A packet cannot be deleted from this buffer until the receiver explicitly confirms its successful delivery.
• Data Link Layer Packets (DLLPs): These are highly compact, 6-byte control packets generated directly inside the DLL. They bypass transaction systems entirely to pass immediate operational updates (like confirmations or errors) across the link.

The Communication Flow

[Transmitter Side DLL] [Receiver Side DLL] | | |---- 1. Transmit TLP (Seq #102 + LCRC) --------->| | (Store duplicate copy in Replay Buffer) | Check LCRC... | | Match Successful! |<--- 2. Respond with ACK DLLP (Seq #102) --------| | | [Purge Seq #102] |

Handling System Errors (The NAK Sequence)

When a problem occurs, the link seamlessly recovers via the following state logic:

• Detection: The receiver detects an incorrect LCRC or discovers a gap in expected sequence scaling (e.g., receiving packet #104 right after #102).
• Signaling NAK: The receiver discards the corrupted frame and fires back a NAK DLLP indicating the last known good sequence marker.
• Replay Execution: Upon receiving the NAK, the transmitter pauses fresh data lines, flushes out unconfirmed data pipelines, rewinds to the targeted index inside its Replay Buffer, and re-transmits the sequence.

If a specific packet fails repeatedly (typically up to 4 attempts), the DLL stops, drops the line status, and instructs the Physical Layer to retrain the electrical interface link completely.

Part 3: Credit-Based Flow Control and the DLL State Machine

To completely avoid dropping packets due to memory saturation, PCIe uses a strict, hardware-level Credit-Based Flow Control system managed by the Data Link Layer.

The Math of Credit Allocations

Before a transmitter can release a packet, it must ensure the receiver has enough advertised space. Memory allocation is split into two pools:

• Header Credits (H): 1 Credit = Space for exactly 1 TLP Header.
• Data Credits (D): 1 Credit = Space for exactly 4 DWORDs (16 Bytes) of payload.

The transmission requirement is computed using integer ceiling rules:

Transaction Type	Payload Size	Header Credits Required	Data Credits Required
Memory Read Request	0 Bytes (Command Only)	1 Credit	0 Credits
Memory Write	64 Bytes	1 Credit	4 Credits (64 / 16)
Memory Write	100 Bytes	1 Credit	7 Credits (100 / 16 = 6.25, rounded up)

The DLL State Machine Lifecycle

The operational readiness of the link transitions through three core states:

• DL_Inactive: The default power-on/reset state. Physical wires are electrically unaligned. The DLL is disabled.
• DL_Init: Triggered once the Physical Layer signals a LinkUp status. TLPs are blocked while the endpoints exchange InitFC1 and InitFC2 DLLPs to advertise and establish their starting credit budgets.
• DL_Active: Credit handshakes match successfully. The gates open for structural application data (TLPs), continuous UpdateFC modifications, and power tracking loops.

Part 4: The PIPE Interface Standard and Cross-Layer Interfacing

How does the logical Data Link Layer communicate with the high-speed analog physical components? It relies on the industry-standard PIPE (PHY Interface for PCI Express) specification.

The PIPE interface acts as a wide parallel digital bus that cleanly splits a chip design down the middle, decoupling the protocol controller logic from the challenging layout constraints of the high-frequency analog SerDes (Serializer/Deserializer).

Primary PIPE Signal Classifications

A standard PIPE implementation groups its control lines into four operational domains:

• Data Path Signals (TxData / RxData): Wide parallel data pipelines (8, 16, 32, or 64-bits wide per physical lane) used to move framed information blocks across the silicon fabric.
• Control Primitives (TxDataK / RxDataK): Direct flag bits indicating whether the data bus contains typical data payloads or special framing tokens (such as STP or END).
• Power Management Controls (PowerDown[2:0]): Commands issued by the MAC controller to transition the analog PHY through electrical efficiency states (P0 active down to deep-sleep P2).
• Status Indicators (RxValid / RxElecIdle): Dynamic feedback from the analog receivers telling the controller that inbound data signals are stable and ready to process.

Architectural Shift: Modern SerDes-Direct Architecture

The boundary responsibilities of the PIPE interface have evolved significantly to meet Gen5 and Gen6 speed requirements:

Traditional Design (PIPE 4.0) Modern SerDes Design (PIPE 5.x+) +--------------------------------+ +--------------------------------+ | MAC Controller | | MAC Controller | | (TLP/DLLP Management Engine) | | (Includes PCS: 130b Encoding) | +--------------------------------+ +--------------------------------+ || || [Standard Parallel Bus] [Streamlined SerDes Bus] || || +--------------------------------+ +--------------------------------+ | PHY Layer Chip | | Pure Analog PMA Block | | (Includes PCS: 8b/10b Encoder) | | (Raw Serializer / Drivers) | +--------------------------------+ +--------------------------------+

Summary

By pairing the Data Link Layer's robust error checking and credit flow management with the clean hardware abstraction provided by the PIPE interface standard, PCI Express achieves its signature combination of low latency and rock-solid system stability.

Transaction Layer in PCI Express

Transaction Layer Packets, or TLPs which contain a header, data payload, and optionally an end-to-end CRC, ECRC. The ECRC, if used, is generated by the user logic at the transmitter and checked by the user logic at the receiver.

There are many types of packets: memory read, memory write, I/O read, I/O write, message, completion, etc... Our job in the transaction layer is to accept packets and issue packets. The packets are presented to us in a specific format called "transaction layer packets" (TLPs), and each 32-bits data arriving on the bus is called a "double word" (or DW in short).

So a packet a TLP) is a bunch of DWs.
How a TLP looks

TLPs are pretty simple to interpret. Here's a general view of their structure.



The header contains 3 or 4 DWs but the most important fields are part of the first DW.



The "Fmt" field tells how long is the header, and if a data payload is present.



Then together with "Type", it describes the TLP operation. The remainder of the TLP header content depends of the TLP operation.

For example, here's a 32-bits memory write TLP header, where you can see that the write address is at the end of the header (and the write data is not shown as it is in the payload after the header).



The "Fmt" field is "10" which means "3 DW, with data". That makes sense, a memory write needs data to write, so upon getting the data payload after the header, we write that data to some memory (or use it somehow) and we are done with it. The Field "Length" tells how many DWs are in the payload (from 0 to 1023). Commonly, it's a single DW to write, in which case length is equal to 1 and the total TLP length is 4 DWs (3 for the header and 1 of the payload).

Now what about a memory read? Somehow we have to return data.
Completion with data

If the TLP is a memory read instead of a write, we have to execute the read and then respond. There is a special TLP for that response, it is called CplD (completion with data) and its payload contains the data that we want to return.

Let's take a closer look at the 32-bit memory read TLP header - it is very similar to the 32-bit memory write we had previously.



One difference is the Fmt=00, which means "no data". Makes sense, we don't need data to do a read, just an address. But we now have to respond with data. And as important, the response will need to be routed back to whoever asked for the read... Do you see the problem?

Ok, we received a read request. Did it come from the CPU? Or from the interrupt controller? Or from a graphic card? After all, many devices are capable of issuing such request. The answer is given in the "Requester ID" - it shows who requested the read. So when we create the CplD TLP, we have to recopy the "Requester ID" in it. This way, it'll be routed where it belongs by the PCI Express bridge(s). BTW, we also have to recopy the "Tag" (which is useful in case of multiple reads pending).


TLP size

A typical 32-bit address/data memory read TLP is made of 3 DWs in the header and no payload (so 96 bits total), while a similar memory write is made of 4 DWs (3 for the header and 1 for the payload). That's not very efficient in term of bandwidth because of the TLP header overhead, so it is better to use bigger TLP payloads when possible. A TLP payload can theoretically be as big as 1023 DWs, pretty handy for burst reads and writes, although PCs can limit the maximum size to a lower value (32 DWs is typical).

Table of Contents