Overview
When system architects design their next platform, they first consider the CPU, accelerators, memory, flash, network interface cards, PCI Express controllers, and switch vendor roadmaps. In systems that mix compute and memory types or share accelerators and memory across clusters, switches are becoming increasingly important.
Mismatch in Roadmap Cadence
The problem is that roadmaps are not always aligned. Most CPU and GPU vendors aim for major compute-engine upgrades roughly every two years, with interim architectural and process refinements so they have new products each year. Network switch and interface chipset vendors in the Ethernet and InfiniBand markets have historically followed a roughly two-year cadence as well, and when Intel Xeon CPUs dominated, many aligned their product releases with Intel's cycle. That alignment has been disrupted by Intel's frequently revised roadmaps, AMD's resurgence as a CPU supplier, and the emergence of other Arm CPU vendors including several hyperscalers and cloud builders.
PCI Express has been ubiquitous for two decades. Although the PCIe specification releases have become more predictable recently, PCIe controllers have followed the PCIe roadmap closely while PCIe switch products from vendors such as Microchip and Broadcom have lagged behind in some releases.
Need for Better Coordination
All these roadmaps need better coordination. Specifically, the PCI-SIG organization, which defines PCIe and works with the IT industry, should accelerate its cadence so that specifications and products better align with CPU, GPU, and network timelines. A two-year cadence for major PCIe steps would serve the industry better than the longer, uneven cadence seen over the past two decades. Advancing to PCIe 7.0 sooner would help match I/O bandwidth and lane counts with high-throughput compute engines. We expect increasing use of PCIe for handling CXL-based hierarchical and shared host memory.
Specification and Product Lag
We adjusted a PCI-SIG bandwidth chart that incorrectly indicated the PCIe 6.0 specification was released in 2022; it was actually released in 2021.
To be clear, the PCI-SIG has enabled cooperation across data center and edge companies. Since the late release of the PCIe 4.0 specification, the PCIe roadmap has been more predictable. However, signal and material challenges kept data centers on PCIe 3.0 for seven years. The dominant CPU supplier at the time was somewhat slow to advance I/O, in part because of chipset reliability issues affecting components released later than expected. Since then, there has been ongoing discussion about the difficulty of integrating PCIe 4.0 and 5.0 controllers into processors.
Typically, about a year after a PCIe specification is finalized, controllers begin appearing embedded in compute engines and network interface chips. For example, when PCIe 4.0 arrived in 2017, the first systems using it appeared in 2018, notably IBM Power Systems based on Power9, and then AMD's "Rome" Epyc 7002 in August 2019. Intel integrated PCIe 4.0 into Xeon SP processors with the Ice Lake generation in April 2021.
Although the PCIe 5.0 specification was finalized only two years after 4.0, it did not appear in products immediately. PCIe 5.0 became available in products when IBM shipped Power10 in its Power E1080 systems in 2021. AMD integrated PCIe 5.0 into server chips with Genoa Epyc in November 2022. Intel integrated PCIe 5.0 into Xeon SP with Sapphire Rapids in January 2023.
Therefore, as expressed by controller integration on CPUs, the effective cadence between products using PCIe 4.0 and 5.0 is about three years, even if the specifications advanced in two-year steps.
Why Faster Cadence Matters
Specs and products need to follow a tighter two-year cycle so compute engines and interconnects can align. This includes PCIe switch ASICs, which have traditionally lagged behind widely used PCIe 3.0, 4.0, and 5.0 generations.
Any generation-level lag between PCIe ports and PCIe switches forces system architects to choose between composability—ideally using PCIe switches at the pod level—or raw bandwidth provided by direct server slots. System and cluster designs require both composability and bandwidth, plus high radix where appropriate.
Current PCIe Switch Market
Today, there are only a few PCIe switch vendors: Broadcom (which acquired PLX Technologies) and Microchip. In February 2021, Microchip announced Switchtec ASICs at the PCIe 5.0 level scaling channels from 28 to 100 and ports from 16 to 52, but to our knowledge those had not shipped in volume at that time. Broadcom announced a PCIe 5.0 product family in February 2022, including the ExpressFabric PEX 89100 switch with 24 to 144 lanes and 24 to 72 ports. At the time of writing, shipment status for these products is being confirmed.
PCIe switches must be available broadly so compute servers, memory servers, and storage servers can be built using chips that support the relevant PCIe generation from day one. Systems should be able to embed switches without losing bandwidth, ports, or radix. A healthy supplier ecosystem is needed in case a single vendor fails to deliver.
PCIe 7.0 and Timing
These considerations are central as PCI-SIG publishes the PCIe 7.0 specification draft. When PCIe 6.0 was completed and PCIe 7.0 approached, the expectation was that the PCIe 7.0 specification would be approved in 2025, implying systems would see it in late 2026 or early 2027. That timeline is long. PCIe 7.0 is needed to provide bandwidth for accelerators processing the large data volumes required for inference or training of AI models, and to match the full complexity of CXL 4.0 for shared and pooled memory.
Accelerating PCIe 7.0 controller and switch availability is difficult, and all related product types must accelerate. Compute engine and peripheral vendors may be hesitant to move prematurely while maximizing investment in PCIe 6.0 cycles. Nevertheless, as PCIe 6.0 products enter the market and undergo rigorous testing—particularly due to PAM-4 signaling and FLIT low-latency encoding—the industry should start accelerating efforts to align PCIe with CPU and GPU roadmaps on a two-year cadence.
Conclusion
Components should remain balanced and advance together. Coordinated, predictable cadence across CPU, GPU, network, and PCIe specifications and products will improve system-level design choices for composability and bandwidth.
