[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Steve Wise]

Tom Tucker wrote:
> On Tue, 2008-05-27 at 12:39 -0400, Talpey, Thomas wrote:
>   
>> At 11:33 AM 5/27/2008, Tom Tucker wrote:
>>     
>>> So I think from an NFSRDMA coding perspective it's a wash...
>>>       
>> Just to be clear, you're talking about the NFS/RDMA server. However, it's
>> pretty much a wash on the client, for different reasons.
>>
>>     
> Tom:
>
> What client side memory registration strategy do you recommend if the
> default on the server side is fastreg?
>
> On the performance side we are limited by the min size of the
> read/write-chunk element. If the client still gives the server a 4k
> chunk, the performance benefit (fewer PDU on the wire) goes away.
>
> Tom
>
>   

I would hope that dma_mr usage will be replaced with fast_reg on both 
the client and the server. 

>>> When posting the WR, We check the fastreg capabilities bit + transport 
>>> type bit:
>>> If fastreg is true -->
>>>       Post FastReg
>>>       If iWARP (or with a cap bit read-with-inv-flag)
>>>               post rdma read w/ invalidate
>>>       
>>> ... For iWARP's case, this means rdma-read-w-inv,
>>> plus rdma-send-w-inv, etc... 
>>>       
>> Maybe I'm confused, but I don't understand this. iWARP RDMA Read requests
>> don't support remote invalidate. At least, the table in RFC5040 (p.22) doesn't:
>>
>>
>>
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    RDMA   | Message   | Tagged| STag | Queue | Invalidate| Message
>>    Message| Type      | Flag  | and  | Number| STag      | Length
>>    OpCode |           |       | TO   |       |           | Communicated
>>           |           |       |      |       |           | between DDP
>>           |           |       |      |       |           | and RDMAP
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0000b  | RDMA Write| 1     | Valid| N/A   | N/A       | Yes
>>           |           |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0001b  | RDMA Read | 0     | N/A  | 1     | N/A       | Yes
>>           | Request   |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0010b  | RDMA Read | 1     | Valid| N/A   | N/A       | Yes
>>           | Response  |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0011b  | Send      | 0     | N/A  | 0     | N/A       | Yes
>>           |           |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0100b  | Send with | 0     | N/A  | 0     | Valid     | Yes
>>           | Invalidate|       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0101b  | Send with | 0     | N/A  | 0     | N/A       | Yes
>>           | SE        |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0110b  | Send with | 0     | N/A  | 0     | Valid     | Yes
>>           | SE and    |       |      |       |           |
>>           | Invalidate|       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    0111b  | Terminate | 0     | N/A  | 2     | N/A       | Yes
>>           |           |       |      |       |           |
>>    -------+-----------+-------+------+-------+-----------+--------------
>>    1000b  |           |
>>    to     | Reserved  |               Not Specified
>>    1111b  |           |
>>    -------+-----------+-------------------------------------------------
>>
>>
>>
>> I want to take this opportunity to also mention that the RPC/RDMA client-server
>> exchange does not support remote-invalidate currently. Because of the multiple
>> stags supported by the rpcrdma chunking header, and because the client needs
>> to verify that the stags were in fact invalidated, there is significant overhead,
>> and the jury is out on that benefit. In fact, I suspect it's a lose at the client.
>>
>> Tom (Talpey).  
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>     
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>