[ofa-general] Re: Re: [PATCH RFC] sharing userspace IB objects

[Dror Goldenberg]

Michael S. Tsirkin wrote:
>> Quoting Hal Rosenstock <halr at voltaire.com>:
>> Subject: Re: [ofa-general] Re: Re: [PATCH RFC] sharing userspace IB objects
>>
>> On Mon, 2007-07-02 at 08:58, Dror Goldenberg wrote:
>>     
>>> Hal Rosenstock wrote:
>>>       
>>>> On Sun, 2007-07-01 at 15:05, Gleb Natapov wrote:
>>>>   
>>>>         
>>>>> On Sun, Jul 01, 2007 at 07:27:24PM +0300, Dror Goldenberg wrote:
>>>>>     
>>>>>           
>>>>>>> SSQ is needed for scalability, no need to explain this (by 
>>>>>>> the way RD is needed for the same reason too. What's Mellanox 
>>>>>>> plan to support it?
>>>>>>>         
>>>>>>>               
>>>>>> RD is not supported in hardware today. Implementing RD is extremely 
>>>>>> complicated. To solve the scalability issues on MPI like applications
>>>>>> we believe that SRC and SSQ are the right solutions. It is much simpler
>>>>>> for implementation by both software and hardware. By MPI-like I refer
>>>>>> to applications that have some level of trust between two processes of
>>>>>> the
>>>>>> same application. RD also has some performance issues as it only 
>>>>>> supports one message in the air. Those performance issues are solved
>>>>>> by design in SRC/SSQ.
>>>>>>
>>>>>>       
>>>>>>             
>>>>> Didn't know about RD limitation. Is this shortcomings of IB spec or
>>>>> general limitation of reliable datagram? RD looks much nice to me then SRC/SSQ.
>>>>>     
>>>>>           
>>>> I think Dror is referring to number of messages in flight per EEC and
>>>> number of messages in flight per QP being limited to 1 per IBA spec.
>>>> Number of messages enqueued per EEC/QP is implementation dependent.
>>>>
>>>> -- Hal
>>>>   
>>>>         
>>> Correct. The number of messages in flight per EEC is 1 per IB spec.
>>> The fact that IB requires SQ WQEs to complete in order, even if their 
>>> destination is different EECs,
>>>       
>> Where's this requirement in the spec (and could this be relaxed as it
>> seems like it is overly "specified") ? Just wondering...
>>     
>
> For example:
> 	10.8.5 RETURNING COMPLETED WORK REQUESTS
>
> 	...
>
> 	Except for RD Receive Work Queues and Receive Work Queues associ-
> 	ated with an SRQ, Work Completions are always returned in the order
> 	submitted to a given Work Queue with respect to other Work Requests on 
> 	that Work Queue.
>
>   
I referred to:

o10-52: If the CI supports RD Service, Work Requests submitted to the
same RD Send Queue shall complete in the same order in which they
were submitted.

And I agree with Roland that it doesn't worth breaking it. And even if 
you do want to break it, it is still a mess to actually implement it in 
hardware and that is the main reason you see no RD implementations out 
there.