[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Chuck Hartley]

We are doing performance measurements on an application that is using uDAPL
RDMA reads for some large transfers and the BW is less than we expected.
The transfers are 4MB and we are seeing BW of 930MiB/sec (DDR).  When we do
the same transfer size using ib_read_bw we get 1475 MB/sec.  On a pair of
machines with SDR interfaces, we get 697MiB/sec and 918MB/sec respectively.

We expected some overhead from uDAPL, but this much seems excessive. Or is
this typical?  I looked at the code for   dat_ep_post_rdma_read() and
compared it to the ib_read_bw() code. The difference is that in uDAPL there
is some cookie management and some WR struct setup before it gets around to
calling the IB verbs function ibv_post_send().  It doesn't look like
something that would take much time - about 1.5msec given the numbers above.

Are the numbers we are seeing similar to what other users are getting?  Do
we maybe have a problem with how the uDAPL libraries were built?  The DDR
machines are X86_64 (uDAPL 1.2) and the SDR machines are PPC64 (uDAPL 2.0.6
).

Thanks,
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080214/7790407e/attachment.html>