[ofa-general] IB verbs client/server programming...

[Peter W. Morreale]

On Tue, 2008-07-15 at 09:39 +0300, Dotan Barak wrote:

> > In other words, what interfaces do I have to look at to implement the
> > "blocking" portion equivalent of a recv(2)?
> Do you want to block until the data will be received?

Yes.  Right now, all the examples I see involve a tight polling loop
waiting for, and acknowledging the events.   

 
> (if so: what is the meaning of this when the sender uses RDMA?)

For my purposes, RDMA is merely another transport. 


> 
> What exactly to you want to do?
> 
> >

I'm creating a distributed memory system that will maintain data
consistent chunks of memory on connected nodes.  The system has control
messages as well as the memory hunks to transport across the wire.  

What I want to  wind up with is a one-to-one correspondence between
memory hunks (that are variably sized) and "transport handles".  Along
with another handle for control msgs.   At least, I think I want
that.  :-)

Right now I'm trying to shoe-horn in both control msgs and data "msgs"
into the same transport abstraction.  This is proving difficult with the
dearth of documentation for RDMA.   

I'm using RDMA-CM as well as (obviously) the IB verbs.  

I suppose I could move back to a socket interface for the control
messages and reserve RDMA for the big data movement....  


On a related note...  Is it possible to reuse a cma_id for different
memory regions by appropriately calling ibv_reg_mr() and
ibv_dereg_mr()?    

Thanks so much.

-PWM