[openib-general] rdma_lat vs. perf_main

[yipee]

Hi, 
 
I'm trying to measure rdma write latencies. 
The platforms I'm using are a pair of 3GHz 64bit Xeons, 2GB ram, 2.6.13 vanilla 
kernel machines. 
The two machines are connected back-to-back through PCIe memfree DDR Mellanox 
HCA's (OpenSM handles the initialization of the "fabric"). 
 
When using the rdma_lat utility (gen2/trunk/src/userspace/perftest/rdma_lat/) 
to measure 4K rdma write latencies I get 22.8 usec (one way). 
Notice that I removed the IBV_SEND_INLINE flag from rdma_lat.c:376 in order to 
be able to send large non-inlined data. 
Also ".max_inline_data=size" was changed to ".max_inline_data=128" (line 342) 
otherwise the qp creation fails. 
 
When trying the perf_main utility from Mellanox 4.1 stack 
(vapi-linux-4_1_0.tgz) I get latencies of 12.8 usec. 
 
Is there a reason for the major latency differences? Do the two programs do 
rdma write latency tests in a fundamentally different way? 
Am I doing something wrong? 
 
 
The commands I used for rdma_lat are 
10.100.1.130] ./rdma_lat 
 
10.100.1.129] ./rdma_lat -s 4096 10.100.1.130 
 
 
The commands for perf_main are: 
10.100.1.129] perf_main --send --test=lat --rdma=write --size=4096 --iter=1000 
-a10.100.1.130 
10.100.1.130] perf_main --server 
 
 
 
Thanks, 
y