Device Lending With a 2-Node Cluster
Dolphin eXpressWare SmartIO software offers a flexible way to enable PCIe IO devices (NVMes, FPGAs, GPUs etc) to be accessed across systemswithin a PCIe Network. Devices can be borrowed over the PCIe network without any software overhead or performance impact of PCIe .
PCIe Device Lending - Composable Infrastructure Made Easy
Device Lending is a simple way to reconfigure systems and reallocate resources. GPUs, NVMe drives and FPGAs can be added or removed to any system on the network, without having to be physically installed in that particular system. The result is a flexible simple method of creating a pool of devices that maximizes resource utilization across the network.
Since Device Lending uses standard PCIe, it doesn’t add any software overhead to the communication path. Standard PCIe transactions are used between the systems. Dolphin's eXpressWare software manages the connection and is responsible for setting up the PCIe Non-Transparent Bridge (NTB) mappings.
Dolphin eXpressWare SmartIO
SmartIO is a Dolphin technology that enables direct access to PCIe devices located anywhere on a PCIe network. Users can dynamically add PCIe devices to the SmartIO pool, enabling other cluster nodes on the network to utilize the devices. The PCIe-based NTB network enables DMA, register accesses, and interrupts to be forwarded in hardware between devices and Cluster Nodes without any additional overhead from protocol translation or software middleware. Not only does the direct communication allow for optimal performance, it also means that the same protocols and driver-models can be used for both local and remote devices.
2-Node Device Lending Benchmarks
We have run a series of benchmarks to demonstrate the performance of Dolphin's Device Lending as a CDI solution. Our benchmark setup involves a 2-node cluster with identical hardware, including two NVIDIA RTX PRO 4500 GPUs in each node. The two nodes are networked via a Dolphin MXH530 NTB PCIe 5.0 adapter card in each node, connected via a x16 PCIe cable. This allows us to measure and compare the performance of the GPUs when they are locally installed in the benchmarking node, when they are borrowed from the other node, or a mix of the two. The benchmarks include Blender, NCCL and nvBandwidth. See the results of each individual benchmark below, as well as a table and diagram of the benchmark topology and hardware configuration.
| Benchmark node hardware | |
|---|---|
| CPU | AMD EPYC 9015 8-Core |
| Motherboard | Supermicro H13SSL-N |
| Memory | 12x 16 DDR5 DIMMs at 6.4 GHz |
| Operating system | Ubuntu 24.04 |
| Kernel | Linux 6.8 |
| Host adapter cards | MXH530 |
| Cables | PCIe 5.0 / SFF-8614 Cables |
| Driver version | 5.24 |
| NVIDIA GPUs | 2x NVIDIA RTX PRO 4500 |
Local vs. Remote GPUs with Dolphin Device Lending
Blender Benchmark
Blender is an open-source 3D computer graphics software, often used as a benchmarking platformm to measure GPU rendering performance using real-world 3D rendering workloads based on Blender and the NVIDIA OptiX rendering engine.
We ran a 100 frame render workload using the Nvidia OptiX engine on our 2-node benchmarking cluster, with Dolphin's MXH530 installed in each node, connected via a direct x16 link, configured with 128GB BAR2 size.
As seen in the graph, remote GPUs delivered rendering performance closely matching that of locally installed GPUs, with as little as a 0,6% performancedifference between single remote GPU vs. a single local GPU on the fishy_cat render.
NCCL Benchmark
NCCL stands for NVIDIA Collective Communication Library, and is a popular framework for executing efficient collective operations on GPUs. This benchmark evaluates multi-GPU communication performance using NVIDIA NCCL, including scenarios like ours where GPUs are distributed across multiple systems.
While the data path distance between the GPUs are the same in the 2 remote GPUs and 2 local GPUs configurations, the workload is still dependant on the host CPU for initiation.
The performance differences were minimal across the different configurations, with less than 10 µs seperating 2 remote GPUs from 2 local GPUs.
| Topology | Element size | sendrecv running time |
|---|---|---|
| 2 remote GPUs | 16 MB | 380.5 µs |
| 1 remote GPU, 1 local GPU | 16 MB | 377.7 µs |
| 2 local GPUs | 16 MB | 370.2 µs |
nvBandwidth Benchmark
nvBandwidth is a benchmarking tool that measures memory transfer bandwidth between GPUs and system memory to evaluate data movement performance in GPU-accelerated systems.
We ran the test using 2 different configurations:
- 2 local GPUs (Local -> Local)
- 1 local GPU and 1 remote GPU, borrowed using Device Lending
(Local -> Remote, Remote -> Local)
The benchmark showed near-identical peer-to-peer bandwidth for local and remote GPUs, with remote devices achieving more than 97% of the bandwidth of locally installed GPUs.
More information about Dolphin Device Lending and CDI over PCIe
Contact us to book a demo, request a quote or learn more about Dolphin PCIe Device Lending and SmartIO.
