Performance Analysis of Various Modulation Techniques Over 448Gbps Channels in Data Centers

To handle the growing level of AI data processing,   the data center requires scaling AI clusters as well as the ability to transmit more data within each cluster.   As clusters expand to hundreds or thousands of nodes, the interconnect fabric becomes the major bandwidth-limiting element in these communication channels. Copper interconnects remain a viable choice for scaling out AI clusters, but the use of copper at 448G per lane requires a deeper understanding of signal integrity (SI) as it relates to connector design and construction.  

This paper presents results of a study by Molex and a leading global semiconductor vendor, examining the right ways to design the data center and supporting components for optimal performance given the intense demand and high data rates. Specifically, the paper explores one of the most impactful design considerations: the feasibility of using copper interconnects with three modulation options: PAM-4, PAM-6 and PAM-8. Industry standards groups are currently debating these modulation options, and the signal integrity challenges involved in connector construction and interconnect design will depend on which modulation option is ultimately implemented for 448G.

Rising data rates in serial communication protocols are reshaping system architectures across data centers. In decades past, data rates and channel bandwidth requirements were small enough that losses recognized in the package as well as the PCB could effectively be ignored. This meant connectors required for building server-to-server interconnects could be placed anywhere with respect to an application-specific integrated circuit (ASIC) or processor.

As data rates increased, the cabled connectors used to build scale-up/scale-out architectures moved progressively closer to the ASIC or processor to reduce PCB losses. Systems using today’s fastest serial interfaces bypass the PCB entirely by implementing co-packaged optics or co-packaged copper, a technology in which copper cables (and, in many cases, optical fibers) are integrated directly into the chip package to provide ultra-high-speed connectivity within a device or between chips. With co-packaged copper, those scale-up/scale-out connectors sit on the ASIC package itself, and data is routed from them to the external I/O or backplane connectors through twinax cable, as shown in Figure 1.

Connectors for co-packaged copper interconnects are surface-mount components, making them vulnerable to signal integrity problems found in other high-density, high-pin-count surface-mount device (SMD) connectors. Any of the following factors can contribute to bandwidth limiting in connectors for co-packaged copper:

• Stubs in the connector mating interfaces and SMD mounting 
• Isolation between I/O pins to suppress crosstalk 
• Small holes that enable electrical connection between layers (vias) in the package routing into an SMD pin on the connector

These challenges would normally be exacerbated when connectors are placed on the PCB, particularly due to vias in the PCB and the insertion loss in long routes where standard PCB materials are used. The extent to which these design factors inhibit the use of specific modulation options in 448G is an open question. Therefore, there is a distinct need for a study to determine the feasibility of the various modulation options with co-packaged copper for 448G channels.

The study conducted by Molex and a leading global semiconductor vendor focuses on design factors influencing signal integrity in surface mount technology (SMT) connectors as well as co-packaged copper connectors for 448G channels. It specifically examines how connector design influences insertion loss bandwidth, as well as how noise contributes to signal-to-noise ratios at high frequencies. In essence, the study evaluates the feasibility of using SMT connectors or co-packaged copper to support 448G channels for three modulation options: PAM-4, PAM-6 and PAM-8.

The analysis methodology involved using idealized channel models for each element in the design, such as ball grid array (BGA) and connector attach, to determine which modulation formats can be best supported by existing technology.

Insertion Loss and Channel Reach

To first examine the feasibility of PAM-4, PAM-6 or PAM-8 in 448G channels, it is worth comparing the case of PCB routing against co-packaged copper in terms of insertion loss and channel bandwidth. As expected, initial findings showed that losses in the PCB channel are much higher than those in co-packaged copper. This can be seen quite clearly in Figure 2, which compares routing to an I/O connector on a PCB versus routing between co-packaged copper and an I/O connector over twinax cable.

While both PCB routing and co-packaged copper with twinax appear to provide sufficient bandwidth beyond the 56 GHz Nyquist frequency for 224Gbps-PAM-4, routing over long distances on the PCB incurs significant loss as shown in the left graph of Figure 2. Co-packaged copper channels allow for longer reach compared to PCB channels for the same insertion loss or, alternatively, co-packaged copper channels have lower insertion loss for the same reach in PCB channels.

Ultimately, nonlinear insertion loss arises near 80 GHz, where steep roll-off effectively terminates the channel bandwidth. Using co-packaged copper with twinax cable between the processor and I/O connector significantly increases the channel reach beyond the 56 GHz Nyquist frequency, which enables 224Gbps-PAM-4 channels. However, neither option provides enough bandwidth to enable the use of PAM-4 in 448G channels.

Significant variation in the channel bandwidth limits can be seen due to channel impairments, as shown in Figure 2. Figures 3 and 4 show which connector design factors can create the variations seen in Figure 2. The results in Figures 3 and 4 show how the insertion loss spectra are affected by changes in the connector mating interface stub length (Figure 3) and the J-lead PCB attachment-related stub length (Figure 4).

Figure 3 shows that a baseline design, qualified for use in a 224Gbps-PAM-4 channel, could fail to support 448G at PAM-6 or PAM-8, as these require minimum channel bandwidths of 90 or 75 GHz, respectively. Strong insertion losses arise due to the mating contact tip length and the size of the mating pad stub. Reducing the sizes of both elements in a standard connector design can extend the bandwidth to the point where a copper channel can support 448Gbps-PAM-6 or 448Gbps-PAM-8.

Figure 4 shows the effect of modifying the stub size on the connector’s SMD pad. A reduction of the SMD pad’s stub length also pushes the channel bandwidth limit up to higher frequencies. The increase in channel bandwidth up to higher frequencies is sufficient to enable the use of 448Gbps-PAM-6 and 448Gbps-PAM-8. In all the results presented so far, PAM-4 continues to remain useful for 224G in PCBs and co-packaged copper channels, but not for 448G.

Crosstalk and Noise Injection

High-density connectors with high pin counts cannot always isolate neighboring differential pairs, making it vital to understand the impact of differential crosstalk between neighboring channels.

To better understand the effects of crosstalk on total noise budgets in these channels, Figures 5 and 6 show the power-sum crosstalk (PSNEXT and PSFEXT) penalty in channels with different levels of insertion loss and different insertion loss roll-off frequencies. The baud rate associated with various modulations is also compared.

These graphs illustrate the noise penalty due to crosstalk as a function of signal-to-noise ratio and signal bandwidth. They demonstrate that, for a given signal level, the noise penalty diminishes alongside a reduction in crosstalk levels. Alternatively, for a given crosstalk level, the noise penalty and signal loss increase together. Co-packaged copper channels have lower signal loss and perform better than PCB-based channels, and PAM-6 modulation has a higher noise penalty than PAM-8 modulation. This is due to the fact that crosstalk levels are significantly higher at 90 GHz for PAM-6 compared to 75 GHz for PAM-8.

Finally, we see how signal losses and crosstalk contribute to the total noise for each channel: PAM-6 outperforms PAM-8 modulation when channel bandwidth is sufficient for PAM-6 operation. To improve channel reach with either modulation, crosstalk reduction requires a new shielding methodology and new mating interface technologies that can scale co-packaged copper to support 448G.

Co-packaged copper eliminates one of the major bandwidth-limiting elements: the interface between semiconductor device packages and the BGA footprint in the PCB. Signals crossing this interface must pass through PCB vias and lossy PCB materials (dielectrics that attenuate high-frequency signals because of their inherent dielectric losses), both of which limit signal propagation. Co-packaged copper bypasses this interface and keeps routing inside the package; connector construction then becomes the main bandwidth-limiting factor. 

By selecting co-packaged copper to support 448G data rates, new challenges emerge related to the design and construction of connectors:

• The separable mating interface in connectors may contain stubs which create strong insertion losses. 
• If J-lead attachments on connectors are used, via and SMT pad stub minimization will be required. 
• When twinax cable is used, the connector transition to the twinax must have broad bandwidth.

This in-depth study found that copper channels can provide sufficient bandwidth to support PAM-6 and PAM-8 modulation in 448G channels, with PAM-8 being better suited for channels with the lowest bandwidth. Outstanding signal integrity questions remain surrounding crosstalk and equalization:

• Can connector construction improve insertion loss resonance and crosstalk in 448G channels? 
• Will new equalization schemes be able to reduce the impact of nonlinear insertion loss and higher crosstalk in 448G channels?

These challenges still need to be addressed to ensure high-fidelity data transmission at 448G and beyond.