[openib-general] QP with large starting sequence adds latency to RDMA READ???

[Sean Hefty]

Hal Rosenstock wrote:
> On Tue, 2005-09-20 at 19:22, Arlin Davis wrote:
> 
>>I just noticed some RDMA read performance issues that seem to be related 
>>to the QP starting sequence number. If I set the starting sequence to 1 
>>then all is fine but if I set it to 0x10000 then it seems to add ~40us 
>>to my 32KB RDMA read operation (polling for completions). Has anyone 
>>seen anything like this?
>>
>>I am running a PCI-e HCA adapter in a EM64T server.
>>
>>RDMA writes and message sends seem to be unaffected.
> 
> 
> What application/ULP are you running ?

He's running a modified version of dtest over DAPL that times RDMA reads from 
the post time to poll CQ completion.

The test setup is actually a little more complex.  The latencies for the runs 
with the various PSNs were:

1 - 60us
0x10000 - 60us (first run)
0x10000 - 100us (subsequent runs)
0x20000 - 100us (first run)
0x20000 - 140us (subsequent runs)
0x40000 - 140us (first run)
0x40000 - 220us (subsequent runs)

The application halts and is re-started between each run, and the QP is 
different each time.  The only change from run to run is the PSN.

- Sean