In this series:
- Nutaq’s Aurora BSDK/MBDK cores – Part 1: Introduction
- Nutaq’s Aurora BSDK/MBDK cores – Part 2: Aurora speed and throughput
- Nutaq’s Aurora BSDK/MBDK cores – Part 4: Latency benchmark
As described in the first and second parts of this series, Aurora is a good solution for low latency, high data rate transfers in systems with a large number of channels (for example, those used for massive multiple-input/multiple-output (MIMO), medical imaging, and particle accelerators). However, due to the high frequency of the multi-gigabit transceivers (MGT) used by the Aurora core, a stable and reliable link can be difficult to achieve when two nodes are far apart from each other.
The Perseus 611X, a double-width advanced mezzanine card (AMC) and accompanying rear-transition module (RTM), provides seven miniSAS connectors that are connected directly to the MGTs of its Virtex-6 FPGA. Each miniSAS connector is connected to a GTX group of four TX and four RX. With miniSAS cable, it’s possible to connect multiple Perseus 611X carrier boards together in mesh architecture (like in the TitanMIMO-4 solution).
In this blog post, we explain how to tune the GTX parameters in order to achieve a reliable 5-Gbps Aurora link between two Perseus 611X cards.
Figure 1: Perseus611X connected together using miniSAS cables
The best way to tune the GTX parameters in order to find the ideal configuration for your hardware is to use the Xilinx IBERT core1. IBERT is freely available with Xilinx CORE Generator. In this example, I generated an IBERT project to simultaneously test the seven GTX group at 5 Gbps. Once the compilation is done, the generated bitstream can be programmed into the FPGA with ChipScope Analyzer.
Once programmed, the IBERT window can be opened, as shown in Figure 2.
Figure 2: IBERT parameters and status
The IBERT console can be used to dynamically modify the GTX parameters while running test pattern generation and analysis. The dynamic parameter reconfiguration in IBERT is a huge advantage when tuning (instead of adjusting the parameters at build time). In IBERT console, all the GTX parameters and port can be modified in the DRP Settings and Port Settings tabs. The main parameters are shown in the MGT/BERT Settings tab, as shown in Figure 2.
When a loopback connector is connected to a GTX pair or when a GTX pair is connected to another one, the Loopback Mode must be set to None. This enables testing of the physical channel for the GTX pair. When using backplane connections, Xilinx recommends using the MGTAVTT voltage as the RX Termination Voltage and to disable the internal RX AC Coupling2.
When connecting a relatively long cable to link two GTX together, the cable attenuates the amplitude of the signal at the receiver input and also acts as a low-pass filter (LPF). When the higher frequencies are attenuated, the transmitted bit edges are less sharp and the bit power can leak into the adjacent channels. This unwanted behavior is called intersymbol interference (ISI). This interference should be avoided since it distorts the desired signal and can cause an unreliable data link.
The TX Diff Output Swing parameter is used to compensate for the amplitude loss of the channel while TX Pre-Emphasis and TX Post-Emphasis are used to compensate for the LPF behavior of the channel and to decrease ISI. Also, the RX Equalization parameter can be modified to boost the higher frequencies over the lower frequencies. The transfer functions of the different equalization values are shown in Figure 3. Be careful, however, by favoring the higher frequencies, the overall signal power will decrease.
Figure 3: RX Equalization transfer functions3
These parameters can be tuned manually in the MGT/BERT Settings tab to achieve an error-free link. The RX Margin Analysis tab, shown in Figure 4, can be used to scan a desired range of values and provide eye diagrams of the link.
Figure 4: RX Margin Analysis tab
Once the optimal parameters values are identified, a long-run test can be performed to catch low-probability errors and to monitor the link stability over time. The BERT Settings section in the MGT/BERT Settings tab (Figure 2) lets you modify the pattern generator in order to monitor the number of received bits and detected errors.
The TX and RX data pattern must be the same and the pattern should be chosen based on the signal encoding that will be used in your final project. For example, Xilinx recommends the PRBS-7 pattern when 8B/10B encoding is used (like in Nutaq’s Aurora implementation) and PRBS-31 for 64B/66B and 10-Gigabit Ethernet. A pattern with a higher PRBS number (PRBS-31) will produce longer sequences of zeros and ones than a lower PRBS number (PRBS-7). Due to the line capacitance, the long run-length of the same value will create data-dependent jitter (DDJ) that causes ISI.
Nutaq’s Aurora core uses 8B/10B encoding to make it easier to achieve a reliable link over long cable. However, 8B/10B has more overhead (25%) than 64B/66B encoding (3%).
Figure 2 shows a reliable link established between two Perseus 611X when their RTM backplanes are connected together using a one meter long miniSAS cable. The IBERT console shows no error in 9.7 x 1012 transmitted bits using a PRBS-31 pattern at 5 Gbps. Using the same GTX parameters for the Aurora core implementation should provide a reliable link between the two Perseus boards.
Apply parameter values to Aurora core
To modify the GTX parameters inside the Aurora core, the GTX instantiation should be located in the source files of the Aurora core. In the ADP 6.4, it is located in the aurora_8b10b_v5_3_gtx.vhd source file inside the lyt_aurora_v1_00_a core folder. Table 1 shows the mapping between the IBERT parameters and the generic parameters and ports of the GTXE1 instance.
|IBERT Parameter||GTXE1 Generic and Port|
|RX Termination Voltage: MGTAVTT||RCV_TERM_VTTRX => (TRUE)
RCV_TERM_GND => (FALSE)
|RX Equalization: 1||RXEQMIX(2 downto 0) => “001”|
|TX DIFF Output Swing: 1070 mV||TXDIFFCTRL => “1111”|
|TX Post-Emphasis 1.32 dB||TXPOSTEMPHASIS => “01110”|
|TX Pre-Emphasis 2.11 dB||TXPREEMPHASIS => “1100”|
Table 1: GTX parameters
Once the GTX parameters of the Aurora core are modified, the default project of Nutaq Aurora core can be recompiled to test that transmitted data from one Perseus board is available without error at the other Perseus board.
I hope that this brief overview of the GTX parameters will help you to achieve a reliable link between your FPGA boards and that you will be able to tune them properly for your hardware configuration.
2. http://www.xilinx.com/support/documentation/user_guides/ug366.pdf, See “RX Termination Use Mode 2 Configuration”
3. http://www.xilinx.com/support/documentation/user_guides/ug366.pdf, See “RX Equalizer”