[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Arlin Davis]

Chuck Hartley wrote:

> Here are the results with the CM:
> 
> ib_rdma_bw -c -n100000 172.16.0.64 <http://172.16.0.64>
> 16855: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | 
> iters=100000 | duplex=0 | cma=1 |
> 16855: Local address:  LID 0000, QPN 000000, PSN 0x8e7362 RKey 
> 0xea042f00 VAddr 0x002aaaaaae9000
> 16855: Remote address: LID 0000, QPN 000000, PSN 0xb45bba, RKey 
> 0xc2042f00 VAddr 0x002aaaab10b000
> 
> 16855: Bandwidth peak (#0 to #27125): 938.241 MB/sec
> 16855: Bandwidth average: 938.238 MB/sec
> 16855: Service Demand peak (#0 to #27125): 2428 cycles/KB
> 16855: Service Demand Avg  : 2428 cycles/KB
> 
> And the straight verbs results:
> 
> ib_rdma_bw -n100000 172.16.0.64 <http://172.16.0.64>
> 16877: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | 
> iters=100000 | duplex=0 | cma=0 |
> 16877: Local address:  LID 0x05, QPN 0x20409, PSN 0xe329ee RKey 
> 0x72043200 VAddr 0x002aaaaaae8000
> 16877: Remote address: LID 0x07, QPN 0x170409, PSN 0x1dc0f2, RKey 
> 0x4a043200 VAddr 0x002aaaab10b000
> 
> 
> 16877: Bandwidth peak (#0 to #49797): 1493.66 MB/sec
> 16877: Bandwidth average: 1493.66 MB/sec
> 16877: Service Demand peak (#0 to #49797): 1525 cycles/KB
> 16877: Service Demand Avg  : 1525 cycles/KB
> 

ok, the performance issue is not uDAPL but rather the QP
optimization differences between rdma_cm and straight verbs.
Since uDAPL uses rdma_cm, this will be the base line for 
performance/optimization. When using rdma_cm to connect,
a path record is obtained and the results are used to setup
QP attributes. For straight verbs, the attributes are hard
coded and QP info exchanged over sockets.

I would concentrate on the path record information returned
from the SA and compare against the straight verbs test
configuration.

Hal/Sean, is there an easy way to see path record information
from the query?

-arlin