[openib-general] Re: rdma_lat vs. perf_main

[Michael S. Tsirkin]

Quoting yipee <yipeeyipeeyipeeyipee at yahoo.com>:
> Subject: rdma_lat vs. perf_main
> 
> Hi, 
>  
> I'm trying to measure rdma write latencies. 
> The platforms I'm using are a pair of 3GHz 64bit Xeons, 2GB ram, 2.6.13
> vanilla 
> kernel machines. 
> The two machines are connected back-to-back through PCIe memfree DDR
> Mellanox 
> HCA's (OpenSM handles the initialization of the "fabric"). 
>  
> When using the rdma_lat utility
> (gen2/trunk/src/userspace/perftest/rdma_lat/) 
> to measure 4K rdma write latencies I get 22.8 usec (one way). 
> Notice that I removed the IBV_SEND_INLINE flag from rdma_lat.c:376 in
> order to 
> be able to send large non-inlined data. 
> Also ".max_inline_data=size" was changed to ".max_inline_data=128" (line
> 342) 
> otherwise the qp creation fails. 

Hi!
1. Try setting max_inline_data to 0. You really dont need it
   since you've removed IBV_SEND_INLINE.

2. Try changing the MTU: rdma_lat sets it to IBV_MTU_256.
   Try IBV_MTU_2048 or IBV_MTU_4096.

Let me know how does it go.

As a side note, please note that rdma_lat measures memory-to-memory latency.
Some people mean other things by latency, e.g. TCP guys
often measure the time until an ack is sent, ignoring
the time it takes to pass the data from TCP stack to the
application buffer. Ack latency would be a lower number
than what rdma_lat reports.

-- 
MST