[ofa-general] Re: [PATCH draft, untested] ehca srq emulation (for IPoIB CM)

[Pradeep Satyanarayana]

Michael S. Tsirkin wrote:
>> Quoting Roland Dreier <rdreier at cisco.com>:
>> Subject: Re: [ofa-general] Re: [PATCH draft,?untested] ehca srq emulation (for IPoIB CM)
>>
>>  > > Note this is not a full emulation, just close enough to make IPoIB CM work.
>>
>>  > If the emulation is only enough for IPoIB, then I think it belongs in
>>  > IPoIB, and not in every HCA driver.
> 
> "every HCA driver" is an exagerration:
> 1. ehca is the only one that does not support SRQ in hardware
> 2. emulation (and ipoib nosrq patches, too) work by assuming only a
>    small number of connections and a huge amount of memory.
>    This is true for systems where ehca is used but not in the general case
> 

Pushing the changes into the driver is a potential maintenance
nightmare. How does one keep changes across layers in sync?

That was the reason I strived to use common code in the NOSRQ case; at
least  as much as possible and all of it in IPoIB.

In the emulation approach by apportioning off WRs across QPs, we will be
sacrificing performance by dropping packets or returning an RNR on a
really busy QP. As I see it, the alternative is to allocate a really big
SRQ, even when there are very few QPs and wasting a lot of the unused WRs.

Thus even with a small number of heavily used connections and huge
amounts of memory we will not be able to derive the performance
benefits that connected mode can potentially offer.

>> I was thinking the same thing.  Otherwise you're just setting a booby
>> trap for someone who tries to use SRQ for something else.
> 
> The emulation is quite close IMO - most likely it will just work,
> but if not, we can just document the limitations.
> 
> In case a ULP wants to avoid using the emulation, we could have a "SRQ is
> emulated bit" to distinguish between these.
> 
>> However it may be a good approach to put an abstraction layer in IPoIB
>> so that the CM code can use an SRQ-like interface to both HCAs that
>> support SRQ and HCAs that don't.
> 
> 2 issues with this:
> 
> 1. I think other ULPs can benefit from this emulation too.
> 2. The emulation does need help from hardware (e.g. I use a qp token
>    in CQE for QP lookups and SRQ detection).
>    Implementing it on top of exiting verbs can be done
>    only if verbs interface is extended.


Pradeep