[rds-devel] [ofa-general] [PATCH RFC] rds: add iwarp support

[Or Gerlitz]

Olaf Kirch wrote:
> One theoretical problem is what happens if the application wants to register a memory range and send the rkey to two peers at the same time? We would have to stall the second send explicitly as long as we haven't seen the completion for the first send.
Indeed.

Basically I think that the fastreg designers envisioned a p2p (i.e 
client - server) transactional protocol where the client flow is:

c1. get bunch of pages from upper layer
c2. apply dma mapping
c3. get MR from a preallocated pool, compute new rkey (8/32 bits 
portion) for this MR
c4. issue fastreg WR on the pages to make the new rkey valid for this MR
c5. issue send "request for rdma transaction" which contains the new 
rkey/stag

second the server does

s1: process request, get rkey, get local buffer from pool
s2: issue rdma WR
s3: write the data to backend storage or deliver it to upper layers 
upper layers
s4: send response + remote invalidate WR on the rdma completion

third the client does

c6: recv response
c7: return MR to pool
c8: apply dma unmapping
c9: hand the response to upper layers

note that

- stage s1 can be further enhanced to use fastreg also at the server side
- there might be some ack to get from storage / upper layers between s3 
to s4
- you probably want to negotiate the remote invalidate capability 
between the client and server upon connection set up

Do you have an applicative usage for registering the same --memory-- 
with the same --rkey-- for two peers?

Or.