[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Arlin Davis]

Chuck Hartley wrote:
> We are doing performance measurements on an application that is using 
> uDAPL RDMA reads for some large transfers and the BW is less than we 
> expected.  The transfers are 4MB and we are seeing BW of 930MiB/sec 
> (DDR).  When we do the same transfer size using ib_read_bw we get 1475 
> MB/sec.  On a pair of machines with SDR interfaces, we get 697MiB/sec 
> and 918MB/sec respectively.
> 
> We expected some overhead from uDAPL, but this much seems excessive. Or 
> is this typical?  I looked at the code for   dat_ep_post_rdma_read() and 
> compared it to the ib_read_bw() code. The difference is that in uDAPL 
> there is some cookie management and some WR struct setup before it gets 
> around to calling the IB verbs function ibv_post_send().  It doesn't 
> look like something that would take much time - about 1.5msec given the 
> numbers above.
> 

You performance should be very close to the verbs rdma
bandwidth tests, especially at 4MB.

Can you try dapltest bw test suite to compare?

server:
dapltest

client:
dapltest -T P -m p -d -s cst-50-ib0 -i 100 RR 4096000

Here are my numbers (Intel x86_64 and DDR):

     Total Time           : 0.31 sec
     Total Data Exchanged : 390.62 MB
     CPU Utilization      : 25.80
     Operation Throughput : 319.65 ops/sec
     Bandwidth            : 1248.66 MB/sec

-arlin