[ofa-general] IB verbs client/server programming...

[Peter W. Morreale]

On Tue, 2008-07-15 at 17:07 +0300, Dotan Barak wrote:
> On Tue, Jul 15, 2008 at 4:07 PM, Peter W. Morreale <pmorreale at novell.com> wrote:
> > On Tue, 2008-07-15 at 09:39 +0300, Dotan Barak wrote:
> >
> >> > In other words, what interfaces do I have to look at to implement the
> >> > "blocking" portion equivalent of a recv(2)?
> >> Do you want to block until the data will be received?
> >
> > Yes.  Right now, all the examples I see involve a tight polling loop
> > waiting for, and acknowledging the events.
> >
> You need to poll for completion on the receive queue
> (IF you use opcode that will consume WR in the recieve queue at the receiver)
> 

I'm sorry, I don't understand this.  What do you mean by "If you use
opcode that will consume the work request in the receive queue..."  

What opcodes are you referring to?  


Are you saying that the receiver does not have to acknowledge the
receipt of the data transfer?  (That would be ideal for the data
transfers.)  If so, how do I set that up?   

The examples I have seen have the receiver ack'ing events.   Can these
be deferred (or ignored even?)  

Thanks,
-PWM


> 
> 
> >
> >> (if so: what is the meaning of this when the sender uses RDMA?)
> >
> > For my purposes, RDMA is merely another transport.
> RC/UC/UD are the transports, the operations (opcode) that you use
> define the behaviour (if the receiver is aware for incoming data or
> not).
> 
> >
> >
> >>
> >> What exactly to you want to do?
> >>
> >> >
> >
> > I'm creating a distributed memory system that will maintain data
> > consistent chunks of memory on connected nodes.  The system has control
> > messages as well as the memory hunks to transport across the wire.
> >
> > What I want to  wind up with is a one-to-one correspondence between
> > memory hunks (that are variably sized) and "transport handles".  Along
> > with another handle for control msgs.   At least, I think I want
> > that.  :-)
> >
> > Right now I'm trying to shoe-horn in both control msgs and data "msgs"
> > into the same transport abstraction.  This is proving difficult with the
> > dearth of documentation for RDMA.
> >
> > I'm using RDMA-CM as well as (obviously) the IB verbs.
> >
> > I suppose I could move back to a socket interface for the control
> > messages and reserve RDMA for the big data movement....
> You can do it over the verbs, you don't really need to use socket for this.
> 
> >
> >
> > On a related note...  Is it possible to reuse a cma_id for different
> > memory regions by appropriately calling ibv_reg_mr() and
> > ibv_dereg_mr()?
> Sorry, but i don't really know the answer for this.
> You should contact Sean Hefty for questions about the cma ...
> 
> >
> > Thanks so much.
> >
> > -PWM
> 
> Dotan