[openib-general] QP with large starting sequence adds latencyto RDMA READ???

[Fab Tillier]

> From: Arlin Davis [mailto:ardavis at ichips.intel.com]
> Sent: Thursday, October 13, 2005 9:42 AM
> 
> Sean Hefty wrote:
> 
> > Arlin Davis wrote:
> >
> >> I just noticed some RDMA read performance issues that seem to be
> >> related to the QP starting sequence number. If I set the starting
> >> sequence to 1 then all is fine but if I set it to 0x10000 then it
> >> seems to add ~40us to my 32KB RDMA read operation (polling for
> >> completions). Has anyone seen anything like this?
> >
> >
> > Has anyone else noticed this issue?  You could try to reproduce this
> > by using the rdma_bw test and changing the PSN.
> >
> > - Sean
> >
> 
> I added a starting PSN and RDMA READ option to the rdma_bw test and was
> able to reproduce on a PCI-E adapter with 4.6.2 firmware. I retried on a
> system with 4.7.0 and it looks like the problem is fixed. However,  I
> see nothing about this problem in the "bug fix" list in the release
> notes. Can someone at Mellanox confirm this problem with RDMA reads and
> add to release notes as a fix so it is documented somewhere?
> 
> http://www.mellanox.com/products/fw_images/fw-25208-4_7_0-release_notes.pdf

Note that I have seen similar behavior (drop in bandwidth) correlated to
starting PSN using Winsock Direct under Windows, so this doesn't seem to be a
uDAPL or Linux issue.  As for Arlin, the issue disappeared in firmware 4.7.0,
and I too would like to see some confirmation that there was an issue and that
it was fixed.

Thanks,

- Fab