[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Tom Tucker]

On Tue, 2008-05-27 at 12:39 -0400, Talpey, Thomas wrote:
> At 11:33 AM 5/27/2008, Tom Tucker wrote:
> >So I think from an NFSRDMA coding perspective it's a wash...
> 
> Just to be clear, you're talking about the NFS/RDMA server. However, it's
> pretty much a wash on the client, for different reasons.
> 
Tom:

What client side memory registration strategy do you recommend if the
default on the server side is fastreg?

On the performance side we are limited by the min size of the
read/write-chunk element. If the client still gives the server a 4k
chunk, the performance benefit (fewer PDU on the wire) goes away.

Tom

> >When posting the WR, We check the fastreg capabilities bit + transport 
> >type bit:
> >If fastreg is true -->
> >       Post FastReg
> >       If iWARP (or with a cap bit read-with-inv-flag)
> >               post rdma read w/ invalidate
> 
> >... For iWARP's case, this means rdma-read-w-inv,
> >plus rdma-send-w-inv, etc... 
> 
> 
> Maybe I'm confused, but I don't understand this. iWARP RDMA Read requests
> don't support remote invalidate. At least, the table in RFC5040 (p.22) doesn't:
> 
> 
> 
>    -------+-----------+-------+------+-------+-----------+--------------
>    RDMA   | Message   | Tagged| STag | Queue | Invalidate| Message
>    Message| Type      | Flag  | and  | Number| STag      | Length
>    OpCode |           |       | TO   |       |           | Communicated
>           |           |       |      |       |           | between DDP
>           |           |       |      |       |           | and RDMAP
>    -------+-----------+-------+------+-------+-----------+--------------
>    0000b  | RDMA Write| 1     | Valid| N/A   | N/A       | Yes
>           |           |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0001b  | RDMA Read | 0     | N/A  | 1     | N/A       | Yes
>           | Request   |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0010b  | RDMA Read | 1     | Valid| N/A   | N/A       | Yes
>           | Response  |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0011b  | Send      | 0     | N/A  | 0     | N/A       | Yes
>           |           |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0100b  | Send with | 0     | N/A  | 0     | Valid     | Yes
>           | Invalidate|       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0101b  | Send with | 0     | N/A  | 0     | N/A       | Yes
>           | SE        |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0110b  | Send with | 0     | N/A  | 0     | Valid     | Yes
>           | SE and    |       |      |       |           |
>           | Invalidate|       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    0111b  | Terminate | 0     | N/A  | 2     | N/A       | Yes
>           |           |       |      |       |           |
>    -------+-----------+-------+------+-------+-----------+--------------
>    1000b  |           |
>    to     | Reserved  |               Not Specified
>    1111b  |           |
>    -------+-----------+-------------------------------------------------
> 
> 
> 
> I want to take this opportunity to also mention that the RPC/RDMA client-server
> exchange does not support remote-invalidate currently. Because of the multiple
> stags supported by the rpcrdma chunking header, and because the client needs
> to verify that the stags were in fact invalidated, there is significant overhead,
> and the jury is out on that benefit. In fact, I suspect it's a lose at the client.
> 
> Tom (Talpey).  
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general