[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Talpey, Thomas]

At 11:33 AM 5/27/2008, Tom Tucker wrote:
>So I think from an NFSRDMA coding perspective it's a wash...

Just to be clear, you're talking about the NFS/RDMA server. However, it's
pretty much a wash on the client, for different reasons.

>When posting the WR, We check the fastreg capabilities bit + transport 
>type bit:
>If fastreg is true -->
>       Post FastReg
>       If iWARP (or with a cap bit read-with-inv-flag)
>               post rdma read w/ invalidate

>... For iWARP's case, this means rdma-read-w-inv,
>plus rdma-send-w-inv, etc... 


Maybe I'm confused, but I don't understand this. iWARP RDMA Read requests
don't support remote invalidate. At least, the table in RFC5040 (p.22) doesn't:



   -------+-----------+-------+------+-------+-----------+--------------
   RDMA   | Message   | Tagged| STag | Queue | Invalidate| Message
   Message| Type      | Flag  | and  | Number| STag      | Length
   OpCode |           |       | TO   |       |           | Communicated
          |           |       |      |       |           | between DDP
          |           |       |      |       |           | and RDMAP
   -------+-----------+-------+------+-------+-----------+--------------
   0000b  | RDMA Write| 1     | Valid| N/A   | N/A       | Yes
          |           |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0001b  | RDMA Read | 0     | N/A  | 1     | N/A       | Yes
          | Request   |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0010b  | RDMA Read | 1     | Valid| N/A   | N/A       | Yes
          | Response  |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0011b  | Send      | 0     | N/A  | 0     | N/A       | Yes
          |           |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0100b  | Send with | 0     | N/A  | 0     | Valid     | Yes
          | Invalidate|       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0101b  | Send with | 0     | N/A  | 0     | N/A       | Yes
          | SE        |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0110b  | Send with | 0     | N/A  | 0     | Valid     | Yes
          | SE and    |       |      |       |           |
          | Invalidate|       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   0111b  | Terminate | 0     | N/A  | 2     | N/A       | Yes
          |           |       |      |       |           |
   -------+-----------+-------+------+-------+-----------+--------------
   1000b  |           |
   to     | Reserved  |               Not Specified
   1111b  |           |
   -------+-----------+-------------------------------------------------



I want to take this opportunity to also mention that the RPC/RDMA client-server
exchange does not support remote-invalidate currently. Because of the multiple
stags supported by the rpcrdma chunking header, and because the client needs
to verify that the stags were in fact invalidated, there is significant overhead,
and the jury is out on that benefit. In fact, I suspect it's a lose at the client.

Tom (Talpey).