[ofa-general] IB verbs client/server programming...

[Dotan Barak]

On Tue, Jul 15, 2008 at 4:07 PM, Peter W. Morreale <pmorreale at novell.com> wrote:
> On Tue, 2008-07-15 at 09:39 +0300, Dotan Barak wrote:
>
>> > In other words, what interfaces do I have to look at to implement the
>> > "blocking" portion equivalent of a recv(2)?
>> Do you want to block until the data will be received?
>
> Yes.  Right now, all the examples I see involve a tight polling loop
> waiting for, and acknowledging the events.
>
You need to poll for completion on the receive queue
(IF you use opcode that will consume WR in the recieve queue at the receiver)



>
>> (if so: what is the meaning of this when the sender uses RDMA?)
>
> For my purposes, RDMA is merely another transport.
RC/UC/UD are the transports, the operations (opcode) that you use
define the behaviour (if the receiver is aware for incoming data or
not).

>
>
>>
>> What exactly to you want to do?
>>
>> >
>
> I'm creating a distributed memory system that will maintain data
> consistent chunks of memory on connected nodes.  The system has control
> messages as well as the memory hunks to transport across the wire.
>
> What I want to  wind up with is a one-to-one correspondence between
> memory hunks (that are variably sized) and "transport handles".  Along
> with another handle for control msgs.   At least, I think I want
> that.  :-)
>
> Right now I'm trying to shoe-horn in both control msgs and data "msgs"
> into the same transport abstraction.  This is proving difficult with the
> dearth of documentation for RDMA.
>
> I'm using RDMA-CM as well as (obviously) the IB verbs.
>
> I suppose I could move back to a socket interface for the control
> messages and reserve RDMA for the big data movement....
You can do it over the verbs, you don't really need to use socket for this.

>
>
> On a related note...  Is it possible to reuse a cma_id for different
> memory regions by appropriately calling ibv_reg_mr() and
> ibv_dereg_mr()?
Sorry, but i don't really know the answer for this.
You should contact Sean Hefty for questions about the cma ...

>
> Thanks so much.
>
> -PWM

Dotan