In a previous blog, Data Communication in Nutaq MicroTCA Systems, I showed the performance levels that can be achieved by the Real-Time Data Exchange (RTDEx) module of Nutaq’s software tools when using the Gigabit Ethernet (GigE) link to transfer data between processing units. In this blog, I show the performance levels that can be achieved when the PCIe link for RTDEx transactions is used instead. As expected, the PCIe link achieved a much higher throughput. However, it’s important to note that, unlike GigE, the PCIe link is only supported for Linux hosts, not Windows 7.
The PCIe link benchmark results were achieved using the following hardware:
• µTCA Chassis: Nutaq Pico box
• Host Computer: HP Compaq Elite 8300 desktop computer, Core i7 3770 3.4 GHz, running 64-bit Fedora 17 operating system
• Perseus: Perseus AMC with an LX240 FPGA
The PCIe core in the Nutaq software tools is 4x Gen 1, which has a raw maximum throughput of 2.5 Gigabits/s per lane (10 Gigabits/s total). Due to 8b/10b encoding, the maximum effective throughput of the PCIe link is 8 Gigabits/s or 1 Gigabytes/s, but this number doesn’t take into account any overhead or flow control mechanics.
The following three scenarios were benchmarked using the PCIe tools that will be available as part of the ADP Software Tools, release 6.4(1). We get a data rate of about 665 MB/s when transferring data from the CPU to the Perseus FPGA. In figure 2 (Perseus to CPU transfers), we measured a peak transfer rate of about 815 MB/s, but get about 750 MB/s for sustain transfers. The difference directly comes from the time spent by the CPU in the interrupt routine to program the burst transfers. The peak transfer rate of 815 MB/s can be assumed for Perseus to Perseus transfers since no processor interrupts are required in this case (figure 3).
Figure 1 PC to Perseus PCIe transfer speed
Figure 2 Perseus to PC PCIe transfer speed
Figure 3 Perseus to Perseus PCIe transfer speed(2)
The PCIe link enables a much larger amount of data to be transferred between the different parts of a system than does the GigE link. The increase in data transfer speed makes the Perseus even more suitable than before when designing FPGA-based data acquisition systems that transfer large amounts of data. For example, a system like the one presented in this blog post. And, just like the GigE link, the configuration and control of the PCIe link is fully integrated in Nutaq’s MBDK environment, helping developers accelerate the prototyping of new applications.
(1) A version with lower PCIe performance is already available in ADP Software Tools release 6.3. The results shown here will be achievable with release 6.4 when it becomes available.
(2) In this test, a remote or embedded PC must act as a PCIe root-complex to perform the PCIe enumeration of the two Perseus AMCs.