[openib-general] RDMA_READ SGE

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Quoting r. Tom Tucker <tom at opengridcomputing.com>:
>> Subject: RDMA_READ SGE
>> 
>> Roland:
>> 
>> iWARP RNIC's have a different SGE limit for RDMA_READ response then
>> they do for other SGE. To support iWARP, we need to add a
>> max_read_sge attribute to the ib_device structure. We had originally
>> mapped this to max_sge_rd, but someone pointed out that this is not
>> for RDMA_READ in general, but for RDMA_READ_ATOMIC in particular.
>> 
>> In any event, we need to add this value to the ib_verbs.h structure
>> so that ULP know how to limit sge for the RDMA_READ sink.
>> 
>> Do you want me to submit a patch on top of Steve's, or do you want me
>> to wait until you've got the git tree built?
>> 
>> Thanks,
>> Tom
> 
> Actually, Mellanox devices also have different SGE limits
> depending on the operation kinds and QP types.
> So far the approach there has been that the device should
> report one value for the best case result.
> 
> Practically I don't see reporting the exact values as a
> priority - I think applications really can figure this out
> easier by attempting operating with relevant parameters and
> fallback to smaller values on failure.
> 
> But assuming that applications really need this information -
> it seems we really should generalize this - maybe make the
> device provide a function mapping QP attributes and operation
> kinds to the max set of values allowed?

That was the consensus in RNIC-PI, report as many distinct values
as required so that the chances that a device would end up under-
reporting any capability would be minimal.

For iWARP, the RDMA Read SGE length is fairly critical, though.
It is almost always exactly one. Having to report that you only
support a single SGE when in fact *only* RDMA Reads have that
restriction would be misleading the application a lot.

The other distinctions don't tend to be a severe.