[ofa-general] correct buffer init for multiple receives

[Dotan Barak]

Hi.

Donour Sizemore wrote:
> Hi everybody.
>
> I'm having a bit of trouble setting up multiple receive buffers for 
> verbs. I'm using the ud pingpong example in ofed1.2 as an outline, but 
> that example posts the same buffer for all receives.
>
> I'm trying to do something like:
>
> -- 
>  for(i=0; i < IB_RXDEPTH; i++){
>     posix_memalign((void**)&(conn->bufs[i]),1024, (IB_MTU + 40));
>     memset(conn->bufs, 0, (IB_MTU+40));
>   }
which value are you using in the IB_MTU?
(the maximum supported IB MTU value is 4K)
>
>  conn->pd = ibv_alloc_pd(conn->context);
>  for(i=0; i < nbufs; i++)
>     conn->mr = ibv_reg_mr(conn->pd, (conn->bufs[i]), (IB_MTU+40), 
> IBV_ACCESS_LOCAL_WRITE);
EVERY memory registration gives you a Memory Region handle (with 
different lkey+rkey for this reason).
In this example you should have an array which will be filled with nbufs 
MR....

or

You can create one big buffer and handle the memory alignment when 
posting the WR yourself (if you wish).
> -- 
>
> Then I'm trying to do a bunch of ibv_post_recv()'s with each buf[i] as 
> the address in the ibv_sge.
No, because you will use the lkey of the last MR that you created with 
memory addresses of different buffers.
>
> Is this what I should be doing? It seems to be causing a big mess, 
> corrupting memory, and giving unrepeatable results.
When you post send/recv request for every memory buffer that you 
registered you need to use the appropriate lkey of this region.


I hope that this helped you...
Dotan